# Pyrite Publisher

Rob Tillotson

rob@pyrite.org

## Contents

- 1 Introduction
- 2 Using Pyrite Publisher
- 3 Plugins and Chains
  - 3.1 E-Book Conversion Chains
  - 3.2 Advanced Plugin Management: Protocols and Priorities
- 4 Configuration and Options
- 5 Standard Plugins
  - 5.1 Protocols and Priorities
  - 5.2 `URLStream` - Input from a file or URL
  - 5.3 `PDBInput` - Input from a Doc or zTXT database
  - 5.4 `RawText` - Raw text parser
  - 5.5 `Text` - Text parser
  - 5.6 `TaggedText` - Tagged text parser
  - 5.7 `HTML` - HTML parser
  - 5.8 `BasicDoc` - Standard e-book assembler
  - 5.9 `TealDoc` - TealDoc-enhanced e-book assembler
  - 5.10 `RichReader` - RichReader-enhanced e-book assembler
  - 5.11 `DocOutput` - Output to a Doc-format database
  - 5.12 `zTXT` - Output to a zTXT-format database
  - 5.13 `TextOutput` - Output to a text file or the console
  - 5.14 `CopyDoc` - Copy from one e-book database format to another
- 6 Plugin Developers Reference
- 7 Protocols Reference
- 8 Low-Level Modules Reference
- About this document ...

# 1 Introduction

The Doc format is a simple text database format used by e-book readers on the Palm Computing Platform. Although there are a few proprietary e-book formats still in use, the Doc format is by far the most widely supported way to store long texts on the Palm platform, with over a dozen readers and editors supporting it.

In its simplest form, a Doc database is just compressed text, in a form designed for efficient storage and navigation on the palmtop. However, many newer readers support some form of "rich text" markup, either a subset of HTML or their own unique tagging schemes.

While all of these augmented readers can be an improvement over a traditional text-only reader, the proliferation of markup formats means that every one of them needs to have its own conversion program (which, often, is not available for any OS except Windows) to create Doc databases with the proper markup. It also means that the same Doc database might or might not be readable on a different reader, depending on what sort of markup is in it. Pyrite Publisher is intended to address these problems (to some extent) by providing a conversion framework with pluggable modules.

Pyrite Publisher separates the interpretation of its input from the rendering of that input into a specific markup language. During conversion, an input plugin is used to convert the input (usually text, HTML, or something similar) to a sequence of formatting events which are independent of any particular markup language; an output plugin converts those into the markup used by a particular Doc reader and creates the resulting Doc database.

# 2 Using Pyrite Publisher

Pyrite Publisher is a command-line program. A graphical user interface may be added in the future, but at present it can be run from the command line only. Normally, Pyrite Publisher's command line interface is called pyrpub, so the most basic way to use it is something like this:

      % pyrpub foo.txt

Pyrite Publisher will try to convert the named file in the best way it knows how, based on the type of data in the file. For example, if you give Pyrite Publisher a HTML file as input, it will automatically use its HTML parser. The result of the conversion will be a file named foo.pdb which you can install on your handheld.

# 3 Plugins and Chains

Most of Pyrite Publisher is built out of a set of plugins, which are assembled into chains to convert an input file to a Palm database. Each plugin has an input and an output; at the beginning of the chain the first plugin's input is the file you specify on the command line, and at the end of the chain the last plugin's output is the database you install on your handheld. In between, each plugin analyzes and modifies the data before passing it to the next plugin in the chain.

When you run Pyrite Publisher, it tries to construct a chain of plugins that can successfully convert the input file to a database. The plugins supplied with Pyrite Publisher are designed to be interchangeable so you can get different results simply by using different plugins.

You can choose plugins to use by using the -P command-line option. Pyrite Publisher will attempt to use the plugins you name, filling in the rest of the chain with whatever defaults are appropriate (unless the requested combination is impossible). You do not need to specify the whole chain, and order doesn't matter. For example, to use the tagged-text parser and create a zTXT instead of a regular Doc database, you could do the following:

      % pyrpub -P TaggedText,zTXT foo.txt

Each plugin may have its own command-line options to control its behavior. The help text (viewable with the -h option) shows the available options for all plugins, and you can list all of the installed plugins with `pyrpub -l`.

## 3.1 E-Book Conversion Chains

In most cases, the process of converting an input file to an e-book database is divided into four parts, each handled by one plugin:

- Retrieving the input file (`URLStream` or `PDBInput`)
- Parsing the input: reflowing text, finding bookmarks, analyzing markup, etc. (`RawText`, `Text`, `TaggedText`, and `HTML`)
- Applying markup and other reader-specific features (`BasicDoc`, `TealDoc`, and `RichReader`)
- Writing to the database (`DocOutput` and `zTXT`)

This process allows for support of multiple document formats without a lot of code duplication. A single parser plugin can format the same input for many different e-book readers, given appropriate choices for the following plugins.

## 3.2 Advanced Plugin Management: Protocols and Priorities

Internally, Pyrite Publisher links plugins into chains by the use of named protocols. A protocol is simply a specification for how two plugins talk to one another. For example, the doc-assembler protocol describes how a parser plugin like `HTML` talks to a markup plugin like `RichReader`. Every plugin knows what input protocols it supports, and what output protocol it will use when given a particular type of input, and Pyrite Publisher uses this information to decide what chains of plugins are possible.

The only exception to this is at the beginning of the chain. The first plugin in the chain is responsible from getting input from outside of Pyrite Publisher, from a file or URL or some other location. Its ``output protocol'' is actually the MIME type of the data.

For example, if you tell Pyrite Publisher to convert the file foo.html (and you have the standard set of plugins), a typical chain would go something like this:

1. Since there are only two input plugins available, `URLStream` and `PDBInput`, and this is not a `.pdb` or `.prc` file, the `URLStream` plugin is activated to read the input file.
2. The input file contains HTML, so the `URLStream` plugin declares that it will use the text/html or application/octet-stream protocol to feed the data to the next plugin.
3. The `HTML` plugin can handle the text/html protocol, so it seems like a good choice to be the next one in the stream. Its output follows the doc-assembler protocol.
4. There are several plugins that support the doc-assembler protocol; the default is `BasicDoc`. The output of `BasicDoc` follows the doc-database protocol.
5. The plugin that handles the doc-database protocol by default is `DocOutput`. It declares no output, so the chain ends here.

Some protocols are implemented by multiple plugins. In the standard set of plugins, for example, there are at least three that handle text/plain and three that handle doc-assembler. This means there are many possible ways to build a chain that converts a particular file into some kind of output. Pyrite Publisher chooses which one of these possible chains to use based on a priority system that lets each plugin declare how good its handling of a particular protocol is.

For example, among the standard set of plugins plain text files can be handled by the `Text`, `RawText`, `TaggedText`, and `HTML` plugins. Of those, the most generally useful choice is `Text`, so it declares a higher priority for the text/plain protocol than the others.

If you name one or more plugins on the command line, Pyrite Publisher will choose a chain that includes them and use priorities to resolve any ambiguities that still exist. You can adjust the priorities of individual plugins using the configuration file, described below.

# 4 Configuration and Options

Each Pyrite Publisher plugin can have its own set of configurable options, called properties. The command `pyrpub -list-properties` will display a list of all plugin properties.

At startup, Pyrite Publisher reads a configuration file called .pyrpub in your home directory. This configuration file can contain two types of statements: set statements which set a plugin property, and priority statements which control plugin priorities.

A set statement looks like this:

- `set` plugin.property = value ;

The semicolon at the end of the line is required. String values don't need to be quoted unless they contain non-alphanumeric characters. For boolean properties, the value should be either ``1'' or ``0''.

The priority statement comes in three forms:

- `priority` plugin input-protocol output-protocol number ;
- `priority` plugin input-protocol number
- `priority` plugin number

The first form sets a plugin's priority for a specific pair of input and output protocols; if that combination is invalid there will be no effect. The second form sets the plugin's priority for a specific input protocol and all possible outputs. The third form sets the plugin's priority for all possible combinations of input and output.

The second form is probably the most useful. If you use TealDoc all the time, for example, the following statement in your .pyrpubrc file might be helpful:

      priority TealDoc ``doc-assembler'' 100

# 5 Standard Plugins

The plugins described in this section are provided with Pyrite Publisher. If any of them are missing, it is likely that your installation is broken.

Unless otherwise specified, all plugin properties can be specified on the command line, by translating underscores to dashes. For example, a property called some_property will have a corresponding command line option called --some-property.

## 5.1 Protocols and Priorities

The standard plugins use the following protocols:

Protocol | Priorities | Description
--- | --- | ---
<textbf>application/octet-stream</textbf> | `RawText` 0 | MIME type for arbitrary data input
<textbf>text/html</textbf> | `HTML` 0 | MIME type for HTML input
<textbf>text/plain</textbf> | `Text` 0, `HTML` -10, `TaggedText` -10 | MIME type for text input
<textbf>doc-assembler</textbf> | `BasicDoc` 10, `RichReader` 0, `TealDoc` 0 | Markup generator
<textbf>doc-output</textbf> | `DocOutput` 0, `TextOutput` -1000, `zTXT` -1 | General e-book database output
<textbf>doc-only-output</textbf> | `DocOutput` 0, `TextOutput` -1000 | Doc-format database output

<textbf>DOC:raw</textbf> | `CopyDoc` 0 | Special protocol for e-book metadata passing

## 5.2 `URLStream` - Input from a file or URL

The `URLStream` plugin is the standard input handler. It allows Pyrite Publisher to retrieve input from local files or from remote locations specified by http or ftp URLs. If the file or remote URL ends in `.gz` it will be un-gzipped automatically. This plugin also determines the MIME type of the input, which in turn determines what the next plugin in the chain will be.

The `URLStream` plugin has no properties or command-line options.

## 5.3 `PDBInput` - Input from a Doc or zTXT database

The `PDBInput` plugin handles input from files or URLs ending in `.pdb` or `.prc`. Unlike `URLStream` it does not automatically un-gzip. If the input is a Doc or zTXT database it will be decompressed and treated as if it was a text file. In addition, metadata such as the document title and bookmarks may be passed along if the next plugin in the chain is compatible. If the input is not a Doc or zTXT database, or if it isn't a Palm database at all, it won't be converted at all.

The `PDBInput` plugin has no properties or command-line options.

## 5.4 `RawText` - Raw text parser

The `RawText` plugin takes raw text input and passes it along to an assembler plugin without any additional processing whatsoever.

The `RawText` plugin has no properties or command-line options.

## 5.5 `Text` - Text parser

The `Text` plugin takes text input and re-flows it to be more readable in a handheld document reader. Generally this consists of joining paragraphs into long lines so that they will be wrapped by the reader on the handheld. In addition, this plugin can automatically add bookmarks based on a regular expression search.

The `Text` plugin has the following properties:

bookmark_regexps
  Regular expressions to search for and bookmark. May be specified multiple times on in the configuration file or on the command line to search for more than one regular expression. (Command line option -r)
indented_paragraphs
  (boolean) If true, checks for indentation to determine the beginning of paragraphs, instead of looking for blank lines separating them. (Command line option --indented-paragraphs)

## 5.6 `TaggedText` - Tagged text parser

The `TaggedText` plugin does the same sort of paragraph reformatting as the `Text` plugin, but it also interprets special markup tags embedded in the text to set bookmarks, add annotations and headers, and the like. This is intended primarily to make it easier to produce e-books from plain text files, by marking interesting bits without requiring a full markup language like HTML.

The tags supported by this plugin are as follows:

TITLE title
  Set the document title.
BOOKMARK label
  Set a bookmark.
BM label
  Same as BOOKMARK.
Hn text
  Insert a heading; ``n'' is a digit from 1 to 6 specifying the heading level.
ANNOTATION label
  Insert an annotation. The annotation text follows this tag, up to an /ANNOTATION tag.
PRE
  Insert preformatted text. The text following this tag up to a /PRE tag is copied verbatim without reflowing paragraphs.
HR
  Insert a horizontal rule.

By default, all tags must appear on a line by themselves, prefixed by a single period.

The `TaggedText` plugin has the following properties:

bookmark_regexps
  Regular expressions to search for and bookmark. May be specified multiple times on in the configuration file or on the command line to search for more than one regular expression. (Command line option -r)
case_fold_tags
  (boolean) If true, markup tags can be lowercase or mixed-case.
tag_prefix
  The prefix which appears before each tag (the default is a single period).
tag_regexp
  A regular expression which matches a tag and its argument. It must contain two named groups, tag and arg, which match the tag and its argument respectively. A string-substitution variable named prefix will be replaced by the value of the tag_prefix property.

## 5.7 `HTML` - HTML parser

The `HTML` plugin parses HTML input and creates markup events which the next plugin in the chain can use to display it appropriately on the handheld. (If the next plugin ignores markup, the result will be to simply strip HTML tags from the input.) It can also produce footnotes showing link targets and set bookmarks at HTML headers and anchors.

The `HTML` plugin has the following properties:

annotate_links
  (boolean) If true, links in the input HTML will be marked in the output text, and the link target will be added to the output document as an annotation.
bookmark_anchors
  (boolean) If true, anchors with a name attribute will be bookmarked.
bookmark_headers
  The value of this property is a string which can contain one or more of the digits 1 through 6. The corresponding HTML header levels will be bookmarked.
footnote_links
  (boolean) If true, links in the input HTML will be marked in the output text, and the link targets will be listed at the end of the output text.

## 5.8 `BasicDoc` - Standard e-book assembler

The `BasicDoc` plugin creates a plain-text e-book, ignoring any markup the previous plugin asks for.

The `BasicDoc` plugin has the following properties:

footnote_marker_format
  The format of footnote markers in the output text. This property should contain a single `%s` which will be replaced by the footnote number.

## 5.9 `TealDoc` - TealDoc-enhanced e-book assembler

The `TealDoc` plugin creates a Doc e-book with markup that is viewable in the TealDoc application. (The resulting document can be read in a normal Doc reader, but the markup will be visible as tags embedded in the text.)

The `TealDoc` plugin has the following properties:

footnote_marker_format
  The format of footnote markers in the output text. This property should contain a single `%s` which will be replaced by the footnote number.
hN_style
  The style of headings; N is a digit from 1 to 6 specifying a heading level. A style consists of one or more words, separated by spaces, which may include: ``large'' or ``bold'' to specify font size, ``underline'' or ``invert'' to specify font decoration, and ``center'' or ``right'' to specify justification.

## 5.10 `RichReader` - RichReader-enhanced e-book assembler

The `RichReader` plugin creates a Doc e-book with markup that is viewable in the RichReader application. (The resulting document can be read in a normal Doc reader, but the markup will be visible as ``garbage'' characters embedded in the text.)

The `RichReader` plugin has the following properties:

footnote_marker_format
  The format of footnote markers in the output text. This property should contain a single `%s` which will be replaced by the footnote number.
hN_style
  The style of headings; N is a digit from 1 to 6 specifying a heading level. A style consists of one or more words, separated by spaces, which may include: ``normal'', ``small'', or ``large'' to specify font size; one or more of ``bold'', ``italic'', ``fixed'', ``underline'', and ``dotted'' to specify font decoration; and ``left'', ``right'', or ``center'' to specify justification. Note that these properties cannot be set from the command line, only in the configuration file.

## 5.11 `DocOutput` - Output to a Doc-format database

The `DocOutput` plugin creates a Doc-format e-book database, usable by any Doc-compatible application.

The `DocOutput` plugin has the following properties:

title
  The document title. (Command line option -t)
backup
  (boolean) If true, set the backup bit of the document database. (Command line option -b)
category
  The category number of the document. (Note: This is only useful if you know the category name to number mapping of your document reader.) (Command line option -c)
creator
  The creator ID of the document database. The default is ```REAd`''. (Command line option -C)
type
  The type ID of the document database. The default is ```TEXt`''. (Command line option -T)
output_directory
  Directory to place document databases into. (Command line option -d)
output_filename
  Full name (including directory) of the output database. Note that this will produce unwanted results if you are converting more than one document at one time. (Command line option -o)

## 5.12 `zTXT` - Output to a zTXT-format database

The `zTXT` plugin creates a zTXT-format e-book database. The zTXT format is currently supported by the Weasel reader, and provides annotations and better compression than the Doc format.

The `zTXT` plugin has the following properties:

title
  The document title. (Command line option -t)
output_directory
  Directory to place document databases into. (Command line option -d)
output_filename
  Full name (including directory) of the output database. Note that this will produce unwanted results if you are converting more than one document at one time. (Command line option -o)
random_access
  (boolean) If true the document is compressed in ``random access'' mode which provides better efficiency and allows larger documents to be read, at the cost of a few percent of compression. The default is for this property to be true, so specifying it on the command line will turn it off.

## 5.13 `TextOutput` - Output to a text file or the console

The `TextOutput` plugin sends output to the console or to a text file, instead of putting it in a database, to facilitate conversion of e-books back to plain text.

The `TextOutput` plugin has the following properties:

output_filename
  Full name (including directory) of the output database. Note that this will produce unwanted results if you are converting more than one document at one time. (Command line option -o)
dump_annotations
  (boolean) If true, output all annotation records (if present) after the text.
dump_bookmarks
  (boolean) If true, output all bookmarks (if present) after the text.
dump_appinfo
  (boolean) If true, output a string representation of the appinfo block (if present) after the text.
dump_all
  (boolean) If true, dump all annotations, bookmarks, and appinfo block after the text.

## 5.14 `CopyDoc` - Copy from one e-book database format to another

The `CopyDoc` plugin works with the `PDBInput` plugin to copy a document directly to an output plugin without any parsing or other processing. It is intended to allow one document format to be converted to another; for example, the following command will convert an existing e-book to zTXT format, preserving bookmarks and annotations if possible:

      % pyrpub -P CopyDoc,zTXT foo.pdb

# 6 Plugin Developers Reference

This section will be filled in later.

# 7 Protocols Reference

This section will be filled in later.

# 8 Low-Level Modules Reference

This section will be filled in later.

# About this document ...

Pyrite Publisher

This document was generated using the LaTeX2`HTML` translator.

LaTeX2`HTML` is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2`HTML` to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.