# Internals

The internals chapter describes and analyzes most of the inner workings of Borg.

Borg uses a low-level, key-value store, the Repository, and implements a more complex data structure on top of it, which is made up of the manifest, archives, items and data Chunks.

Each repository can hold multiple archives, which represent individual backups that contain a full archive of the files specified when the backup was performed.

Deduplication is performed globally across all data in the repository (multiple backups and even multiple hosts), both on data and file metadata, using Chunks created by the chunker using a content-defined chunking algorithm - the Gear rolling hash of FastCDC (“fastcdc” chunker, the default) or Buzhash (“buzhash” and “buzhash64” chunker) - or a simpler fixed block size algorithm (“fixed” chunker).

To perform the repository-wide deduplication, a hash of each chunk is checked against the chunks index, which is a hash table of all chunks that already exist.

Layers in Borg. At the very top, commands are implemented, using a data access layer provided by the Archive and Item classes. The “key” object provides both compression and authenticated encryption used by the data access layer. The “key” object represents the sole trust boundary in Borg. The lowest layer is the repository accessed via class Repository. Repository uses `borgstore` internally.

Internals contents

- Security
  - Cryptography in Borg
    - Attack model
    - Structural Authentication
    - Passphrase notes
    - Encryption
      - AEAD modes
      - Authenticated modes
      - Unencrypted modes
      - Legacy modes
    - Offline key security
      - Multiple borg keys
    - Implementations used
  - Remote RPC protocol security
  - Using OpenSSL
  - Compression and Encryption
  - Fingerprinting
    - Stored chunk sizes
      - buzhash and buzhash64 chunker
      - fixed chunker
    - Secret key usage against fingerprinting
    - Stored chunk proximity
- Data structures and file formats
  - Repository
    - Keys
    - Repository objects
    - Repo object metadata
    - Compaction
  - The object graph
    - The manifest
      - Feature flags
    - Archives
    - Items
    - Item digests
    - Chunks
      - “fixed” chunker
      - “fastcdc” chunker
      - “buzhash64” chunker
      - “buzhash” chunker
      - “toeplitz-aes” chunker
      - “rabin-aes” chunker
      - “goldilocks-aes” chunker
  - The files cache
  - The chunks index
  - Indexes / Caches memory usage
  - HashIndex
  - Encryption
    - AEAD modes
    - AEAD usage limits
    - Modes without encryption
    - Legacy modes
    - All modes
  - Key files
  - Compression
  - Lock files (fslocking)
  - Locks (storelocking)
  - Breaking the locks
  - Checksumming data structures
    - Lower layer — file_integrity
    - Upper layer
  - HardLinkManager and the hlid concept
- Chunkers
  - Choosing a chunker
    - What the choice affects
    - The chunk-size fingerprinting threat
    - The chunkers at a glance
    - Recommendations
  - Technical background
    - Model and the broken class
    - The UHF-then-PRF construction
    - The three universal hashes
    - Performance and limits
    - References
- Pack files
  - Pack File Format
    - Per-blob layout
    - Pack ID
    - Namespace
  - Pack Index Entry
  - Index Namespace
  - Recovery Path
  - Repository Version and Feature Flags
- All about JSON: How to develop frontends
  - Dealing with non-unicode byte sequences and JSON limitations
  - Logging
  - Prompts
  - Passphrase prompts
  - Standard output
    - Archive formats
    - File listings
    - Archive Differencing
    - Archive Analysis
  - Message IDs