
                       The APT project design document


-----------------------------------------------------------------------------
                                      
                              Manoj Srivastava

                            <srivasta@debian.org>

                                Version 3.2.0

-----------------------------------------------------------------------------

Copyright © 1997 Manoj Srivastava

License Notice

    APT, including this document, is free software; you may redistribute it
    and/or modify it under the terms of the GNU General Public License as
    published by the Free Software Foundation; either version 2, or (at your
    option) any later version.

    This is distributed in the hope that it will be useful, but without any
    warranty; without even the implied warranty of merchantability or fitness
    for a particular purpose. See the GNU General Public License for more
    details.

    You should have received a copy of the GNU General Public License with
    your Debian system, in /usr/share/common-licenses/GPL, or with the 
    debiandoc-sgml source package as the file COPYING. If not, write to the
    Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

Abstract

This document is an overview of the specifications and design goals of the
APT project. It also attempts to give a broad description of the
implementation as well.

-----------------------------------------------------------------------------

Table of Contents

1. Introduction
2. Requirements
3. Procedural description
4. Modules and interfaces
5. Data flow and conversions analysis.

Chapter 1. Introduction

    APT is supposed to be a replacement for dselect, and not a replacement
    for dpkg. However, since addition functionality has been required for
    APT, and given the fact that this is very closely related to dpkg, it is
    not unreasonable to expect that additional functionality in the
    underlying dpkg would also be requested.

    Deity/dselect are the first introduction that people have to Debian, and
    unfortunately this first impression contributes greatly to the public
    perception of the distribution. It is imperative that this be a showcase
    for Debian, rather than frighten novices away (which has been an
    accusation often levelled at the current system)

Chapter 2. Requirements

     1. APT should be a replacement for dselect. Therefore it should have all
        the functionality that dselect has currently. This is the primary
        means of interaction between the user and the package management
        system, and it should be able to handle all tasks involved in
        installing, upgrading, and routine management without having the
        users take recourse to the underlying management system.

     2. It should be easier to use and less confusing for novice users. The
        primary stimulus for the creation of APT was the perceived
        intractability, complexity, and non-intuitive behavior of the
        existing user interface, and as such, human factors must be a primary
        mandate of APT.

     3. It should be able to group packages more flexibly, and possibly allow
        operations based on a group. One should be able to select, or
        deselect, a coherent group of related packages simultaneously,
        allowing one to add, remove, or upgrade functionality to a machine as
        one step.

     4. This would allow APT to handle standard installations, namely, one
        could then install a set of packages to enable a machine to fulfill
        specific tasks. Define a few standard installations, and which
        packages are included therein. The packages should be internally
        consistent.

     5. Make use of a keywords field in package headers; provide a standard
        list of keywords for people to use. This could be the underpinning to
        allow the previous two requirements to work (though the developers
        are not constrained to implement the previous requirements using
        keywords)

     6. Use dependencies, conflicts, and reverse dependencies to properly
        order packages for installation and removal. This has been a
        complaint in the past that the installation methods do not really
        understand dependencies, causing the upgrade process to break, or
        allowing the removal of packages that left the system in an untenable
        state by breaking the dependencies on packages that were dependent on
        the package being removed. A special emphasis is placed on handling
        pre-dependencies correctly; the target of a predependency has to be
        fully configured before attempting to install the pre-dependent
        package. Also, configure immediately requests mentioned below should
        be handled.

     7. Handle replacement of a package providing a virtual package with
        another (for example, it has been very difficult replacing sendmail
        with smail, or vice versa), making sure that the dependencies are
        still satisfied.

     8. Handle source lists for updates from multiple sources. APT should
        also be able to handle diverse methods of acquiring new packages;
        local filesystem, mountable CD-ROM drives, HTTP accessible
        repositories are some of the methods that come to mind. Also, the
        source lists can be separated into categories, such as main, contrib,
        non-local, non-free, my-very-own, etc. APT should be set up to
        retrieve the Packages files from these multiple source lists, as well
        as retrieving the packages themselves.

     9. Handle base of source and acquire all Packages files underneath.
        (possibly select based on architecture), this should be a simple
        extension of the previous requirement.

    10. Handle remote installation (to be implemented maybe in a future
        version, it still needs to be designed). This would ease the burden
        of maintaining multiple Debian machines on a site. In the authors
        opinion this is a killer difference for the distribution, though it
        may be too hard a problem to be implemented with the initial version
        of APT. However, some thought must be given to this to enable APT to
        retain hooks for future functionality, or at least to refrain from
        methods that may preclude remote activity. It is desirable that
        adding remote installation not require a redesign of APT from the
        ground up.

    11. Be scalable. Dselect worked a lot better with 400 packages, but at
        last count the number of packages was around twelve hundred and
        climbing. This also requires APT to pay attention to the needs of
        small machines which are low on memory (though this requirement shall
        diminish as we move towards bigger machines, it would still be nice
        if Debian worked on all old machines where Linux itself would work).

    12. Handle install immediately requests. Some packages, like watchdog,
        are required to be working for the stability of the machine itself.
        There are others which may be required for the correct functioning of
        a production machine, or which are mission critical applications. APT
        should, in these cases, upgrade the packages with minimal downtime;
        allowing these packages to be one of potentially hundreds of packages
        being upgraded concurrently may not satisfy the requirements of the
        package or the site. (Watchdog, for example, if not restarted
        quickly, may cause the machine to reboot in the midst of
        installation, which may cause havoc on the machine)

Chapter 3. Procedural description

    Set Options

        This process handles setting of user or site options, and
        configuration of all aspects of APT. It allows the user to set the
        location and order of package sources, allowing them to set up source
        list details, like http site locations, passwords, etc. Display
        options may also be set.

    Updates

        Build a list of available packages, using source lists or a base
        location and trawling for Packages files (needs to be aware of
        architecture). This may involve finding and retrieving Packages
        files, storing them locally for efficiency, and parsing the data for
        later use. This would entail contacting various underlying access
        modules (http, cdrom mounts, etc) Use a backing store for speed. This
        may also require downloading the actual package files locally for
        speed.

    Local status

        Build up a list of packages already installed. This requires reading
        and writing the local?? status file. For remote installation, this
        should probably use similar mechanisms as the Packages file retrieval
        does. Use the backing store for speed. One should consider multiple
        backing stores, one for each machine.

    Relationship determination

        Determine forward and reverse dependencies. All known dependency
        fields should be acted upon, since it is fairly cheap to do so.
        Update the backing store with this information.

    Selection

        Present the data to the user. Look at Behan Webster's documentation
        for the user interface procedures. (Note: In the authors opinion
        deletions and reverse dependencies should also be presented to the
        user, in a strictly symmetric fashion; this may make it easier to
        prevent a package being removed that breaks dependencies)

    Ordering of package installations and configuration

        Build a list of events. Simple topological sorting gives order of
        packages in dependency order. At certain points in this ordering,
        predependencies/immediate configure directives cause a break in
        normal ordering. We need to insert the uninstall/purge directive in
        the stream (default: as early as possible).

    Action

        Take the order of installations and removals and build up a stream of
        events to send to the packaging system (dpkg). Execute the list of
        events if successful. Do not partially install packages and leave
        system in broken state. Go to The Selection step as needed.

Chapter 4. Modules and interfaces

    The user interface module

        Look at Behan Webster's documentation.

    Widget set

        Related closely to above Could some one present design decisions of
        the widget set here?

    pdate Module

        Distinct versions of the same package are recorded separately, but if
        multiple Packages files contain the same version of a package, then
        only the first one is recorded. For this reason, the least expensive
        update source should be listed first (local file system is better
        than a remote http site)

        This module should interact with the user interface module to set and
        change configuration parameters for the modules listed below. It
        needs to record that information in an on disk data file, to be read
        on future invocations.

         1. HTTP methods

         2. mount and file traversal module(s)?

         3. Other methods ???
   
    Status file parser/generator

        The status file records the current state of the system, listing the
        packages installed, etc. The status file is also one method of
        communicating with dpkg, since it is perfectly permissible for the
        user to use APT to request packages be updated, put others on hold,
        mark other for removal, etc, and then run dpkg -BORGiE on a file
        system.

    Package file parser/generator

        Related to above. Handle multiple Packages files, from different
        sources. Each package contains a link back to the packages file
        structure that contains details about the origin of the data.

    Dependency module
          o dependency/conflict determination and linking

          o reverse dependency generator. Maybe merged with above

    Package ordering Module

        Create an ordering of the actions to be taken.

    Event generator

        module to interact with dpkg

Chapter 5. Data flow and conversions analysis.

                                                              _____________
                                                           __\|http modules|
                                                          /  /|____________|
                                        _ ____________   /     ________________
                                        |    update   | /     |mount/local file|
            |==========================>|   module    |/_____\|  traversals    |
            |                           |_____________|      /|________________|
            |                             ^        ^
            |                             |        |               ______________
      ______|_______    _ _____ ______    |   _____v________      \|            |
     |Configuration |   |configuration|   |   |Packages Files|  ===|Status file |
     |  module      |<=>|    data     |   |   |______________| /  /|____________|
     |______________|   |_____________|   |        ^          /
             ^                            |        |         /
             |                            | _______v_______|/_
             |                            | |              |    ________________
             |                            | |              |/_\|   Dependency  |
             |                            | |backing store |\ /|     Module    |
             |                            | |______________|  _|_______________|
             |                             \       ^          /|       ^
             |                              \      |         /         |
             |                              _\|____v_______|/__    ____v_______
             |_____________________________\| User interaction|    |    dpkg   |
                                           /|_________________|<==>  Invoker  |
                                                                   |___________|

    dpkg also interacts with status and available files.

    The backing store and the associated data structures are the core of APT.
    All modules essentially revolve around the backing store, feeding it
    data, adding and manipulating links and relationships between data in the
    backing store, allowing the user to interact with and modify the data in
    the backing store, and finally writing it out as the status file and
    possibly issuing directives to dpkg.

    The other focal point for APT is the user interface.

