Tei2html

The design of tei2html is based on a small set of guiding principles and design decisions. These are some assumptions, that I believe are reasonable to expect when digitizing text for research and preservation.

The guiding principles are:

The design decisions are:

As a result of these design decisions, a document can be neatly formatted with CSS with minimal rendering information 'polluting' the TEI master file. Note that to meet ePub restrictions on CSS, no inline (that is, in an attribute on an HTML element) CSS is generated.

Originally, TEI was defined in SGML. Since version P4, this TEI has moved on to XML. XML is a considerably simplified reincarnation of SGML. It does away with much of the complexity of SGML. However, TEI (and TEILite) was originally developed for SGML, and I started using TEI even before XML was conceived.

Although XML is simpler, and far more tools can deal with it, SGML is somewhat easier to type due to its more relaxed syntax rules. You don’t need to use quotes on all attribute values, nor need to provide close tags for all elements. For this reason, I normally produce my files in SGML. The conversion to XML is straightforward, and can (almost always) be fully automated. Since I am used to SGML, the examples in this document will be valid SGML snippets, but not necessarily well-formed XML.

Of course, you can work directly in XML instead of SGML, and the stylesheets are supposed to work with the P4 and P5 incarnations of TEI as well (although not all elements, even those of P3, are supported). Working in XML directly has a number of benefits as well:

TEI files are typically prepared from pre-existing sources, that is printed books, magazines or other textual works. In those cases, it may be desirable to not only to include the logical content of the work being transcribed, but also to describe the physical appearance of some aspects of the text. In that case, the @rend, @style and @rendition attributes can be used to record those aspects.

For many uses, a TEI file is not the most desirable format. For reading, it is far more convenient to have it available in some more presentation-oriented format. With tei2html, a TEI document can be converted to various output formats, which are:

These output formats place some restrictions on the structure of the TEI file, described in the following usage conventions. These usage conventions are not enforced by the DTD.

References to page numbers. In printed media, the page numbers produced in the output will differ from the source. To correctly replace page numbers in the source, each page number should be tagged as a reference (ref element) with @type=pageref. The tag should only enclose the page number itself, not any of the surrounding material. The transformation will replace the content of this tag with the actual number of the page the material referred to appears on. Almost always, the reference is not to the page itself, but to some element appearing on this page. For this reason, it is better not to link to the pb element, but to the element actually referred to. (In HTML output, the original page numbers will be shown in the margin and used in the text.)

Nesting of cross-references and anchors. In HTML, it is impossible to nest anchors and cross-references. As a results, certain elements in the TEI source should not be nested, as these will result in invalid nested anchors in HTML. For example, the ref element should not contain corr elements (as the latter generates an anchor for the automatically generated errata list). The proper way to resolve this is to place the ref element inside the corr element.

Nesting of paragraphs. In HTML, it is impossible to nest paragraphs. Usually, the transformation software will take this in account, and close (and re-open) open paragraphs in HTML when needed.

In a few other cases, the transformation may not result in entirely correct results. Always validate the result with tools such as tidy or epubcheck.

For XML files actually used by the XSLT stylesheets, any character encoding supported by XML/XSLT works. For SGML, the options are more limited.

Also, the pre-processing scripts in Perl can only deal with either pure 7-bit ASCII or the ISO 8859-1 character set. All characters outside those ranges are to be represented by character entities.

Use entities from the following sets:

For longer fragments in a non-Latin script, I normally use an ASCII based transliteration scheme, and supply tools (called patc) to convert these to Unicode. For documents completely a non-Latin script, it will probably be better to work with Unicode in a suitable editor (and using XML directly).

Most of the interaction on the command line is with small Perl scripts that start the actual XSLT processor, and some other scripts.

Before you can run the Perl scripts, you will need to make sure the paths to the various executables are configured correctly. Perl should be able to find the place where you’ve located your scripts (when running perl with the -S option.)

The various .jar files (for Saxon and epubcheck), and the SGML catalog files need to be located in a specific subdirectory of the directory where the perl scripts reside.

No easy window-based interface is yet available. I am working on a small HTML application to run the scripts interactively.

You can use the stylesheets with Oxygen, provided your sources are in XML format. Make sure you select an XSLT 3.0 processor to perform the transformation.

An XSLT 3.0 processor is required for tei2html, any XSLT 3.0 processor should work, however, I’ve developed these stylesheets with Saxon (using the freely available version Saxon HE).

You should download a reasonable recent version of Saxon-HE product from saxonica.com. I will take care tei2html will continue to work with the free versions of Saxon, no matter how tempting the additional features in the paid versions are (such as higher order functions, binary file handling, etc.)

Saxon-HE requires Java.

Make sure the java executables can be found on the path.

If you do not have Java, you can download it from http://java.com/en/

If you are planning to use the provided Perl scripts to glue things together, you will need a Perl interpreter.

For Windows, my advice is to download Strawberry Perl. Use either the 32 or 64 bits version, to match your system.

Note Upgrading Strawberry Perl does not work properly: please save-guard local installations in your site directory before upgrading, as the uninstaller will throw those files away.

Optional, only needed you want to use SGML as source format.

SX is an SGML to XML translator, and NSGML is an SGML validator, part of the SP package. Since XSLT only supports XML, you will need both those two tools to be able to work with SGML. They can be downloaded from James Clark website. For windows, get sp 1.3.4.

To enable SX and NSGML to understand your document types, you need to configure a catalog of DTDs (Which maps public DTDs to local resources containing their definitions). The scripts assume this is located in a file named dtd/CATALOG.

You will need to add the teilite.dtd to the CATALOG. This DTD can be found here: http://www.tei-c.org/Vault/P4/Lite/DTD/

A short explanation of Catalog files can be found in the SP documentation on James Clark’s website referenced above.

As an alternative to SX and NSGML, you can use osx and onsgml from the OpenJade Project. These are slightly more up-to-date. You will have to update the names in the main tei2html.pl script in that case.

To compress ePub files, you will need a zip utility. tei2html uses info-zip to handle the peculiar requirements of the ePub format. (See e.g., this blog entry on creating ePub files.)

Optional, only needed when you use mathematical formulas in TeX notation.

To convert mathematical formulas in TeX format to a format that can be included in static HTML pages, you will need to install the following:

Optional, only needed when you use the transcription schemes for non-Latin scripts I use in SGML.

Patc (pattern changer) is a small utility written in C to do multiple find-and-replace actions at once. You will need a C compiler to get it to work. It enables you to execute multiple find-replace actions in an efficient way. Mostly used to change the transliteration of non-Roman scripts I’ve used. If you don’t use that, you’ll not need it. (I’ve successfully compiled this on a variety of platforms, including Unix, and Windows; a solution file for Visual Studio 7.0 is included; contact me if you need a Windows binary. A make file for use with linux is also included.)

Optional, only needed if you want to check generated ePubs.

Epubcheck is a tool to validate epub books, it can be obtained from: https://github.com/IDPF/epubcheck. Note that tei2html doesn’t automatically generate correct ePubs: you can still do a lot of things that make ePubs non-conform, for example by including CSS3 constructs, or referring to resources in CSS and not including them in the ePub spine manually.

This tool also requires Java; the scripts assume you use epubcheck-3.0.1.jar, placed in the tools/lib subdirectory.

Optional, only needed if you want generated PDF output.

Note that Prince is a commercial product; a free version can be used for strictly private purposes, and downloaded from: https://www.princexml.com/. The free version does include a small icon on the first page to promote Prince. Shouldn’t be a big nuisance.

Optional, only needed if you want to browse through the documentation of the code.

You can clone XSLTdoc (at time of writing version 1.3.3) from GitHub, where you can also find the documentation.

To run tei2html from the command line, it will be practical to configure several environment variables, that is:

Most of my TEI files are in (old-school) SGML format with the '.tei' extension. This needs to be converted to XML, before XSLT wants to do anything with it.

Now that we have XML, we can apply the XSLT transformation on it. This will result in a single HTML output file. Here, again a few intermediate steps are needed.

The default configuration file is given below. Its actual value can be found in the modules/configuration.xsl.

This can also be found configuration.xsl.

The first element of the TEI header is the file description, <fileDesc>. This contains the most important meta-data in three main elements, the <titleStmt>, <publicationStmt>, and <sourceDesc>. Our template looks like this.

You might notice that the TEI header contains the title and author information twice. This is intentional, as one refers to the title of the file, and one to the title of the source. For the title given in the <titleStmt>, some TEI recommendations suggest appending "an electronic transcription" to the original title. I think that is unnecessary, and will use the original title as given on the title page, or perhaps normalized in some fashion. For personal names, I always give them in the natural order, that is, without a comma, and optionally I will supply the vital dates in parentheses, for example, <author>John Doe (1833-1901)</author>. You may add other elements allowed by TEI in this statement if needed.

In the publication statement, you can give some information on the publisher and the availability of the book. For typical Project Gutenberg publications, the information in the template is fine.

The following specific values can be used in the <publicationStmt> element to register the various ways the text is known at Project Gutenberg and elsewhere.

The <availability> element should include a short reference to the copyright status of the work. For most texts in Project Gutenberg, the following phrase will be appropriate: Not copyrighted in the United States. If you live elsewhere please check the laws of your country before downloading this ebook.

In the <sourceDesc>, the title and author information appears again. This time, you should use the exact title and author names as given on the title page of the original work used to prepare your Project Gutenberg text. If the title on the spine or cover of the book differs, this may be noted, but the title page is leading.

Sometimes it makes sense to also have a cover thumbnail which isn’t just a reduction of the (original) cover page. To do this, just specify (as a hack) cover-thumbnail(imagefile) in the rend attribute of the text element. Similarly, cover-image(imagefile) can be used to specify a cover image that is not part of the original book, for those who like to have both.

Simply drop an image file named favicon.jpg in the images directory, and tei2html will generate the metadata in the HTML to use it. Use a square image, upto 192 x 192 pixels.

When a page-break occurs between two divisions, it can be encoded in various was, e.g., before the closing tag:

Between the closing and opening tag:

After the opening tag:

Various guidelines for TEI propose different (conflicting) conventions for each option. The tei2html scripts all assume the first option (before the closing tag) This way, the generated tables of content, etc., refer to page-number that is current when the <div1> tag occurs.

It should be very well possible to write a small XSLT script to normalize this usage, avoiding complications in the other XSLT scripts.

See facsimile support

As mentioned, cross-references in printed books often reference to page numbers. The easiest way to encode this is to do something like this:

However, there are several problems with this approach:

For these reasons, tei2html specifies that references to page-numbers are tagged as:

That is, directly surrounding the page number, and the additional @type attribute set to pageref. This indicates to the rendering application that the content of the <ref> element needs to be replaced with the actual page number on which the details appear.

To prevent that the details do not appear on the indicated page, but on a previous or next page, you should replace the target with the non-<pb> element the details are really in (most likely a paragraph).

When creating new paged output, we will have to deal with yet another set of issues:

First, exact values for cross-references can only be established when final pagination is known. In some cases, this may result in changes in the page layout inducing changes in the page numbers itself ("see page 100" is wider than "see page 99"). Most current layout tools can deal with this type of issues reasonably, although in some extreme cases this still pops up.

Second, when generating an index, you want to avoid things like

but summarize them as

which is not supported by most current index generating tools, and is pretty hard to do as well. (tei2html does this when referring to pre-existing page-numbers, but this is not very useful for paged output.)

Currently, tei2html supports figures following the TEI P3-model, with some modifications.

A figure element describes a single image, and optional some heading, legend text and a description.

The way the image itself was left somewhat implementation dependent. Within the structure of SGML, it could be specified as an entity, which was hardly supported anywhere. tei2html worked around this by deriving the image file from the @id, using an additional attribute @url, or placing the image file name in a rendition ladder in the @rend attribute.

TEI P5 revised the figure model, introducing a graphic element, and allowing multiple graphics to be part of a single figure. This better matches with practices in books, but requires a revision of the code.

The new model allows multiple graphics in a figure and specifies the attribute @url as the way to specify the image.

The current code should remain functional for existing texts.

In addition to the TEI model, tei2html supports column definitions. These can be used to apply styling to all cells in a column. The following elements are defined:

column: a column definition, which can have the following attributes: @cols the number of columns this column definition applies to, by default only a single column; @rend the rendering that should be applied to each cell in the column matching the position of the column definition.

columnGroup: a group of column definitions, that can be repeated multiple times. A columnGroup should always have the attribute @repeat which indicates the number of times the column-group should be repeated.

Note that the @rend value align(decimal) triggers some special processing when generating HTML. Since HTML does not directly support alignment on the decimal separator, any column with align(decimal) will be split into two columns, with in the first the integer part of the numbers, and in the second the fractional part. CSS is used to make those parts align as expected.

Sometimes, rows are preceded by a tall brace. tei2html will automatically generate HTML to include an image of a tall brace when a cell spans more than one row and only includes a single brace character.

tei2html cannot directly process tables correctly. A number of pre-processing steps are required to correctly deal with the complexities tables introduce. These are handled in the xslt file normalize-table.xsl, which includes code to determine the correct column of each cell (taking into consideration spanned rows and columns) and the special handling of decimal alignment. The tei2html.pl perl script normally applies this transform to the XML before applying the main transform.

Nested tables are supported.

Notes, given in the element <note> are, by default, considered footnotes. That can be made explicit by adding @place="foot"

Footnotes appear at the bottom (or foot) of the page. Since HTML does not have the concept of a page, tei2html collects all footnotes at the end of the div0 or div1 element they appear in.

If this is undesired, you can also explicitly call for the footnotes to be rendered, by inserting a <divGen type="footnotes">, which will generate the footnotes in a generate section. If only the body of that section is desired, use <divGen type="footnotebody">.

Footnotes will be automatically numbered per division. In the text, a small superscript number will link to the actual footnote. This same number is rendered in front of the footnote, and will link back to the location in the text. A small up-arrow at the end of the footnote will also link back to the text.

Sometimes, multiple footnote references on a page refer to the same footnote. This can be encoded using the @sameAs attribute, linking it to the @id of the referenced footnote. This way, multiple references can be made to the same footnote (using the same number). The up-arrow after such a duplicated note will be followed lower-case letters, each linking back to an instance of the note reference in the text.

If you want to override the automatically assigned note number, you can provide an alternative marker with @rend="note-marker(*)". Note that this will not change the generated number of other notes, that is, the note will still be counted [TODO: fix this].

tei2html support various types of marginal notes. The way they are placed in the margin in the HTML output is defined by various CSS rules. To indicate a note is a marginal note, use the @place attribute. This can have the following values:

The top three options will place the note outside the text-block. The cut-in variants will place the note as a floating box inside the text-block of the main text.

Some care is need to prevent marginal notes to overlap each other. Simply don’t place them too close to each other (although that may not be an option when digitizing pre-existing books). It may help to widen the margin if needed.

Notes that appear inside a table can be marked with @place="table". In that case, the notes will be rendered directly below the table, and numbered with lower-case letters, starting anew for each table.

Notes in tables not marked as table notes will be treated as footnotes to the division the table appears in.

Text-critical works often include numerous notes linked to words or lines in the source text, indicating variant readings of the text in one or more editions being compared.

You can indicate apparatus notes with @place="apparatus". Notes marked thus will not be automatically rendered in the output, but require you request their rendering using <divGen type="apparatus">. At that location, all apparatus notes preceding this divGen will be rendered in a single paragraph (unless an apparatus notes contains multiple paragraphs itself).

All apparatus notes will be given the same reference symbol (by default a degree sign), which will be used to link the notes with their point of reference.

If you use multiple instances of <divGen type="apparatus">, multiple blocks of apparatus notes will be generated, each containing the notes before the block, but after the previously generated block.

TODO: support a better model for apparatus notes, as described in the TEI guidelines.

Use the suggested TEI notation: <formula id=f1 notation="TeX">$E = mc^2$</formula> (see TEI guidelines).

The Project Gutenberg prohibition on active content make the direct use of MathJax impossible, so we need to follow a more complicated way to include mathematical formulas in text intended for PG.

The process consists of three steps:

The resulting output files are placed in a folder formulas, and named according to the following naming scheme (inline|display)-<ID>.tex. The name is important, as the conversion script uses that to determine the correct MathJax option to use when converting the file to SVG, etc.

The script convertFormulas.pl will convert these files to files named (inline|display)-<ID>.svg. This script will use node.js with mathjax-node-cli to produce HTML, SVG, and MathML files from the exported formulas.

The following configuration options are available:

Simple rendering attribute values provide single keywords to provide a rendering hint. This type of usage is sufficient most of the time. The pre-defined rendering keywords are often element specific, and should be considered as hints only, that is, ignoring the rend attribute should not render the document illegible.

A slightly more powerful mechanism is provided by so-called 'rendering ladders', in which a number of key-value pairs are provided. These take the form key(value), and multiple can be present in a @rend attribute. They are either translated to CSS, or given a specific meaning, sometimes depending on the element they are applied to.

Sometimes, you want to exclude certain elements from the alignment (e.g., non-translated sections, headings, or figures). To do so, you have to include the element on both sides with the same @n attribute, and give one the following @rend attribute:

The other element should remain empty (as it will be skipped).

It is also possible to align texts kept in separate files. To do so, use the following @rend attribute:

When using the perl scripts that come with tei2html, the to-be-aligned sections will be included in an intermediate document in a pre-processing step (in include.xsl), which replaces align-with-document with align-with and places the to-be aligned section directly after the section it will be aligned with. By doing so, generated tables of contents, lists of issues in the colophon, and footnotes will be processed correctly. The tei2html xslt stylesheeets can also do this directly, but with limited support for those additional items.

Footnotes in aligned text will be placed at the bottom of the column they appear in. If footnotes are present in both aligned texts, both sides will have footnotes.

When aligning verses (or actually line-groups, encoded with the lg element), there is no need to number matching lines with a @n attribute. However, the number of items in the line-groups needs to match. Again to align two line-groups, on one of the line-groups add the following @rend attribute:

This will generate a table, in which all lines are placed in rows side-by-side. The code will recurse into nested lg elements, still using a single table.

The facsimile element is a top-level element, that describes a series of page images, and can stand independently of the transcribed text. This allows to specify just the metadata (in the Header) and the scans, to produce a digital facsimile.

The @facs attribute on pb elements can be used to point to scanned images of transcribed pages. This can be used to either link to some external source of page images (for example in the Internet Archive), or to link to an internal set of images (kept in a page-images subdirectory, for example).

Let’s call this "direct" facsimile links.

Alternatively, it can link to an element in the facsimile element, for example:

Let’s call this "indirect" facsimile links.

This later case also allows pointing to zones within a page, but that is currently out-of-scope for tei2html.

The two ways should not be combined. Currently, the code only allows the "direct" way of linking to page images.

For tei2html to support ePub 3.0 we need to do add some new information to our ePub files. In most cases, that information can easily be derived from the existing TEI files, so no changes to data-files will be required.

Also see ePub3 changes.

Many of the things to-do can be aligned with over-all support for HTML5/CSS3. Specific for ePub3 are:

The navigation document is now a valid XHTML page, wrapped in the <nav> element.

See this short article originally from threepress.

Will need to generate valid XHTML, which means:

SMIL files have a structure as given below, and must be placed in the same directory as the referenced HTML file.

The actual audio files are in a subdirectory called audio.

Currently, there is no support for SMIL types of overlay in TEI. We can solve this most conveniently by adding a rend attribute to a division, indicating the .smil file to use as an overlay.

And then parse the .smil file for the entries that need to be added to the OPF manifest (that is, the .smil file itself and the associated audio files). This leaves us with a way to obtain the required overlay metadata.

All this is handled in the file tei2opf.xml.

In the XSLT domain, the main naming convention uses dashes, so we use dashes for names of templates, modes, etc., except where such names directly reflect elements from the TEI domain that use camelCase.

In the TEI domain, the main naming convention is camelCase, so for names primarily used in a TEI context, we follow that.