2.1. The Alvis Architecture: A Brief Overview

The EU-funded Alvis project runs for three years from 2004-2006, and is tasked with building a semantic peer-to-peer search engine.

The peer-to-peer (P2P) aspect of the Alvis system is one of the key areas in which it innovates. The metadata format used to describe the capabilities of individual peers is described in the document Alvis Task T3.1 - Peer-Description Metadata Format. Milestone M3.1 - Month 6 (June 2004) .

However, within each individual peer that makes up an Alvis network, a great deal of semantic work is done. Documents may be added to a peer's local database from a variety of sources, to become available for subsequent semantic searches. As documents enter the system, they pass through a pipeline of components that enhance them with various kinds of information. This document describes the format - or, rather, the family of formats - that such a document takes as it passes through the system.

It is important to understand that the communication described in this document is all within the context of single Alvis peer. Communication between peers expresses an entirely different set of concepts, and consequently uses an entirely different format, to be described elsewhere.

An Alvis peer that includes all the components described in this document is informally referred to as a ``fat peer'' because so much work is happening within the peer itself. This is in contrast with other candidate P2P architectures such as ``peer soup'', which the Alvis Architecture Group considered and rejected, in which each peer provides just one of the components described below, and the functionality of the components is exposed in the P2P protocol. The ``fat peer'' architecture achieves a simpler P2P protocol at the expense of each individual peer being more complex; whereas the ``peer soup'' architecture would have achieved smaller individual peers, but at the cost of a more complex protocol between them.

2.2. The Need for a Format for Enriched Documents

As documents are added to an Alvis peer, they pass between a series of components that perform various analyses and transformations, each enriching the document in different ways. Since the components are to be created by different partners, potentially using different programming languages running on different platforms, it is necessary to define a rigorous shared format in which the documents, together with their associated enrichments, can be expressed for interchange between components.

Task T3.2 (Semantic Document Metadata Framework) in Workpackage WP3 (Data Model and Standards) is to provide such a format. This document describes the format as it stands at month twelve (December 2004) of the Alvis project, and consitutes Milestone M3.2.

Since the Alvis proposal document was written, the consortium partners have adopted more explicitly descriptive terminology, so that what the WP3 part of the proposal referred to as ``semantic documents'' are now called ``enriched documents''. Accordingly, Task 3.2 is now more properly defined as the provision of a metadata format for enriched documents.

2.3. Principles for Designing the Format

In designing the Alvis metadata format for enriched document, we have been guided by four principles:

3.1. Acquisition Format

The first format - known as ``acquisition format'' - is a simple representation of a document broken into several semantically distinct parts (title, author, body of text, etc.), including information about the acquisition process. It does not seek to represent all the details of the original form of the document. (It may, however, carry with it a copy of the original document, as described below.)

The goal of the canonical document carried in acquisition format is to capture all and only the semantically significant content of the original document, and to put it in a single well-defined form that subsequent components of the pipeline can easily handle without needing to know anything about the vagaries of, for example, HTML. The task of comprehending the various input formats, then, is encapsulated neatly by the various Document Sources, and the other Alvis components need know nothing of them.

3.2. Linguistic Format

The second format - known as ``linguistic format'' - is an augmented version of acquisition format. That is, a linguistic document contains all the same XML elements and content as the acquisition document from which it was derived, but has additional elements describing the linguistic information extracted from the canonical document itself. This means that every acquisition document is also a linguistic document, but with zero linguistic information.

Linguistic documents are created by the Linguistic Analyser built in WP5 (Document Analysis and Normalization), and the linguistic document format is designed to capture the kind of information that WP5 generates.

Such information could be added to the document in one of two ways: either by embedding it within the core document, inserting additonal XML elements to elucidate the structure; or by placing the additional information alongside the core document, with pointers indicating what parts of the core document each annotation pertains to. This second approach is known as ``stand-off annotation'', and this is the approach taken by the linguistic format.

3.3. Relevance Format

The third format - ``relevance format'' is an augmented version of linguistic format. That is, a relevance document contains all the same XML elements and content as the linguistic document from which it was derived, but has additional elements describing the relevance information extracted from the linguistic document. This means that every relevance document is also a linguistic document,but with zero relevance information (and every acquisition document is a relevance document, but with zero linguistic or relevance information).

Relevance documents are created by the relevance package built in WP2 (Document Probability Model), and the relevance document format is designed to capture the kind of information that WP2 generates.

Document Sources such as the harvester should create enriched documents consisting only of of a acquisition section. The Linguistic Analyser should not alter the acquisition section of documents that pass through it in any way, only adding a linguistic section (or altering an existing linguistic section). The relevance package should not alter the acquisition or linguistic sections of documents that pass through it in any way, only adding a relevance section (or altering an existing relevance section).

5.1. Example Canonical Document

The following example canonical document serves as a motivating example for much of the discussion to follow.

<?xml version="1.0"?>
<!-- $Id: m3-2.html,v 1.1 2005-05-19 13:57:29 mike Exp $ -->
<canonicalDocument>
  <section title="Why Dinosaurs are Cool">
    Dinosaurs are much cooler than mammals because they are so much
    bigger.  T. rex could eat a tiger, easy.
    <section title="Saurischians">
      Saurischians include the sauropods (the biggest of all
      dinosaurs) and the theropods (the carnivorous dinosaurs).
    </section>
    <section title="Ornithischians">
      Ornithischians include Triceratops, Pachycephalosaurus,
      Stegosaurus, Ankylosaurus and Iguanodon.
    </section>
  </section>
  <section><!-- no title could be found for this section -->
    The coolest
    <ulink url="http://www.dinodata.net/">dinosaurs</ulink>
    of all were the sauropods.  They were way
    huge.  I mean, you may think it's a long way from the top of a
    giraffe to the bottom, but that's peanuts to a Brachiosaurus.
    The coolest sauropods of all were:
    <list>
      <item>Bruhathkayosaurus</item>
      <item>Amphicoelias fragillimus</item>
      <item>Brachiosaurus, which has the species:
        <list>
          <item>altithorax</item>
          <item>brancai</item>
          <item>?nougaredi</item>
        </list>
      </item>
      <item>
        <ulink url="http://www.snomnh.ou.edu/pdf/2000/00-27.pdf">
          Sauroposeidon
        </ulink>
      </item>
      <item>Migeod's mysterious M23 sauropod</item>
    </list>
  </section>
</canonicalDocument>

5.2. Intent of the Canonical Format

The intention of the <canonicalDocument> element is that it contains markup transformed from the original document where and only where that markup indicates document structure with some semantic significance. For example, if a Web harvester is preparing an HTML document that includes the following fragment:

Here is what <font face="Ariel">Holtz</font> says:
<blockquote>
  All those great coelurosaur fossils from Liaoning are a
  couple of lucky rolls of the taphonomic crap shoot.
</blockquote>
	

Future versions of this format may preserve questionable markup such as italics, which can have semantic significance in some fields (e.g. indicating the use of a formal scientific genus or species name in biology).

5.3. Elements Included in the Canonical Format

In accordance with the principles expounded above, and with the need for simplicity at this early stage in the Alvis project, we define canonical format initially to be capable of representing the following structural elements as well as plain text:

6.1. Example Enriched Document

The following example enriched document (actually a collection consisting of a single document) serves as a motivating example for much of the discussion to follow.

<?xml version="1.0" encoding="UTF-8"?>
<!-- $Id: m3-2.html,v 1.1 2005-05-19 13:57:29 mike Exp $ -->
<documentCollection>
  <documentRecord id="12345678">

    <!-- Information generated during the initial acquision of the
    document, whether by a web crawler, MS-Word converter, etc. -->
    <acquisition>

      <!-- Information, in the current WP7 format, to do with the
      acquisions process: acquision date, URLs where the document was
      found, expiry date, size, etc. -->
      <acquisitionData>
        <modifiedDate>2001-04-19</modifiedDate>
        <expiryDate>2004-11-06</expiryDate>
        <checkedDate>2004-10-06</checkedDate>
        <httpServer>WebSTAR/4.4(SSL) ID/72915</httpServer>
        <urls>
          <url>http://www.snomnh.ou.edu/pdf/2000/00-27.pdf</url>
        </urls>
      </acquisitionData>

      <!-- Original document represented as cleaned HTML, or text
      extracted from MSWord, PS, PDF, etc.  May be a binary format, with
      an attribute specifying base64 or quoted-printable encoding -->
      <originalDocument mimeType="text/plain" charSet="us-ascii">
        ...
      </originalDocument>

      <!-- Visible text from document, together with what internal
      structure we can express through canonical markup -->
      <canonicalDocument>
        <!-- As in previous example -->
      </canonicalDocument>

      <!-- Information, in the current WP7 format, that is _about_ the
      document rather than part of it: e.g., author, title, subject, DOI -->
      <metaData>
        <meta name="dc.author">Wedel, Mathew J.</meta>
        <meta name="dc.date">2000</meta>
        <meta name="dc.title">Sauroposeidon proteles, a new sauropod from
                the Early Cretaceous of Oklahoma</meta>
      </metaData>

      <!-- Link information from WP7 format. All URLs will contain an
      internal ID (not guaranteed to be unique across multiple crawler
      instances) -->
      <links>
        <outlinks> <!-- links to external pages -->
          <link type="a"> <!-- repeatable -->
            <anchorText>Text from this document</anchorText>
            <location documentId="...">URL</location>
          </link>
        </outlinks>
        <inlinks> <!-- links from external pages -->
          <link type="a"> <!-- repeatable -->
            <anchorText>PDF ( 1 MB)</anchorText>
            <location documentId="...">http://www.snomnh.ou.edu/publications/Articles/index.shtml</location>
          </link>
        </inlinks>
        <!-- Number of unique other hosts with links pointing to this page -->
        <inlinkHosts> ... </inlinkHosts>
      </links>

      <!-- Results of analysis done as part of the acquisition process,
      e.g. genre intuited from top-level domain name of the site from
      which a Web document was crawled -->
      <analysis>
        <!-- analysis also containes other analysed properties (mainly from
             the URL) with property name as tag and content as value -->
        <property name="topLevelDomain">edu</property>
        <property name="language">en</property>
        <property name="genre">article</property>
        <ranking scheme="..."> ... </ranking> <!-- repeatable -->
        <topic absoluteScore="150" relativeScore="570">
          <class>ALL</class>
        </topic>
        <topic absoluteScore="100" relativeScore="380">
          <class>CP</class>
          <terms>carnivorous plant[^\s]*, carnivor[^\s]*, </terms>
        </topic>
        <topic absoluteScore="50" relativeScore="190">
          <class>CP.Dionaea</class>
          <terms>flytrap[^\s]*, venus flytrap[^\s]*, </terms>
        </topic>
      </analysis>
    </acquisition>

    <!-- Annotations from WP5 -->
    <linguisticAnalysis>
      <!-- Details omitted: see Deliverable D5.1 -->
    </linguisticAnalysis >

    <!-- Relevance information added from WP2 -->
    <relevance>
      <scoreset type="ranking">
        <score topicId="1">8.36536</score>
        <score topicId="4">4.25395</score>
        <score topicId="19">0.44538</score>
        <score topicId="36">2.35349</score>
      </scoreset>
      <scoreset type="content">
        <score topicId="1">40.25395</score>
        <score topicId="4">2.947</score>
        <score topicId="17">0.44538</score>
        <score topicId="23">1.4629</score>
        <score topicId="36">2.35349</score>
      </scoreset>
    </relevance>
  </documentRecord>
</documentCollection>

6.2. Representing Multiple Documents in a Single Package

A <documentCollection> is merely a wrapper for several <documentRecord> objects, and has no information at all of its own: it's not a collection about some specific subject, or a collection of documents harvested from a particular place, or anything similar: it's just a simple, unstructured aggregate like a TAR archive or or ZIP file.

6.3. Document Identifiers and Identity

Each document in a complete Alvis network is identified by an opaque, unique identifier, represented by the id attribute on the <documentRecord> element. This identifier must remain constant as the record takes on its various forms (acquisition, linguistic, relevance) during its journey through the Alvis pipeline. This identifier may be subsequently used to specify records for deletion or update.

Because Document Sources are the first components to handle the documents that are passed through the Alvis pipeline, it falls to them to allocate the identifiers. However, the identifier identifiers the entire enriched document, in each of the forms that it takes, not just the <acquisition> section that is a Document Source's main responsibility. Accordingly, the id attribute is on the <documentRecord> element rather than <acquisition>.

Document Sources must choose identifiers that are, in the immortal words of RFC 1341 ``as unique as possible''. One possible mechanism for generating such globally unique IDs is for the peer maintainer to use an Internet domain-name that they own, and append a locally unique token. Another is simply to generate a long string of random bits. A third is to use an MD5 checksum of the document.

Using the MD5 checksum (or any checksum) has the property that two identical documents acquired at different times and by different Sources will have the same identifier, and will thus be, for Alvis purposes, ``the same record''. Is this property desirable or broken? It depend on what we mean by our notion of a record's identity.

When are two Alvis documents considered to be the same document? For identity purposes, an Alvis document is a specific sequence of bytes. This means that, given one document to compare with:

• If a document consisting of the same sequence of bytes is found at another URL (or is acquired by another process altogether) then that is deemed to be The Same Document.

• If the document is re-fetched from the same URL, and it includes a counter or a time indication, then it's a Different Document the second time (and each subsequent time).

• If another document is identical except that a typo has been fixed, or a tab character replaced by eight spaces, then it's A Different Document.

• If another document consists of exactly the same characters in the same order, but encoded in a different character set (e.g. UTF-8 vs. ISO-Latin-1) then it's A Different Document. (This is because Alvis considers documents to be identifiable as sequences of bytes rather than characters.)

6.4. The <acquisition> Section

<originalDocument mimeType="application/msword" charSet="utf-8"
	compression="gzip" encoding="base64">
H4sICDwAXEECA3RleHQuZG9jAO19C5xcRZlvdRKS4ZEQAoSAEdoYYRI6Qx6TyQOuy2QmIQkJGTLh
/ZAzMz2ZJjPdQ3fPhGFZFjG+EBBZBJZFRS4quMCPdV1EdL1cFlG5XNd1uVzkclcX0SuK3BhZVlwg
+6+vvjpVdU6d090TUMIv3flP+nSfU6fqq6+++l5V5wf/eMhPPv83R/2riLxOFBPFG7v3F5Ot7zLA
...
</originalDocument>

<meta name="dc.author">Wedel, Mathew J.</meta>
<meta name="dc.date">2000</meta>

6.5. The <linguisticAnalysis> Section

The <linguisticAnalysis> section follows the form described in Deliverable D5.1, Report on method and language for the production of the augmented document representations. That document both describes the format of this element and includes a DTD formally specifying it.

6.6. The <relevance> Section

The <relevance> section describes the results of WP2's document probability calculations as a sequence of <scoreset> subsections, each of which provides scores for a set of topics. Each <scoreset> has a type attribute specifying its applicability. Types likely to appear in relevance score-sets include:

Each <scoreset> element contains zero or more <score> elements, which in turn carry a topicId attribute and contain a a floating-point number measuring the score of the identified topic according to the document probability model. The form of the topicId attribute remains controversial: either an opaque numeric or symbolic identifier may be used, or a short phrase identifying the topic in a human-readable way.

In principle, this section of a relevance-format enriched document carries information analogous to that generated at harvesting time and encoded in <topic> elements within the <acquisition> section's <analysis> subsection. The Document Source and Document Probability software components use different approaches to trying to determine the topic of a document. For example, the subject-specific web crawler developed in WP7 is based on an an explicit topic definition (ontology), while the WP2 software's relevance figures are based on statistical modeling of documents in a collection.