Package com.mckoi.database

## Class FixedSizeDataStore

- java.lang.Object
- - com.mckoi.database.FixedSizeDataStore

    public final class FixedSizeDataStore
    extends java.lang.Object

A file format that allows for the very quick retreival of data that is stored within it. To allow for such quick reference of information in the file, we must make stipulations about how the data is stored.

1. Each data element in the table must be a fixed length. This could be thought of like a 'sector' of a disk drive. Or in a table, each row may be of the fixed size.
2. We keep track of deleted rows via a linked list of sectors.

The header of the data store is as follows:

       0 4 (int)  : MAGIC        - used to identify file type.
       4 4 (int)  : version      - version of the file store.
       8 4 (int)  : sector_size  - the size of each sector in the store.
      12 8 (long) : delete_head  - the head sector of the delete list.
      20 8 (long) : sectors_used - number of sectors being used (not deleted).
      28 1 (byte) : open         - set to 1 when file opened, 0 when closed.
      29 4 (int)  : sector_start - offset where sector information starts.
      33 ...  63                 - reserved
      64 ... 191                 - reserved buffer for misc. state data.
     192 ... sector_start        - reserved

Each sector contains a 5 byte header. This header includes a byte that contains either USED or DELETED, and a int pointer to the next chained sector. The int pointer is used to either represent a pointer to the next sector in the chain of USED sectors, with -1 indicating the end. Or, if the sector is DELETED, it points to the next deleted sector in the chain.

Author:
  Tobias Downer

### Constructor Summary

Constructors
`FixedSizeDataStore​(java.io.File data_file, int sector_size, boolean cache_access, DebugLogger logger)` Constructs the data store.
`FixedSizeDataStore​(java.io.File data_file, int sector_size, DebugLogger logger)`

### Method Summary

All Methods Instance Methods Concrete Methods
`int` `addSector​(byte[] buf)` Adds a new sector into the store.
`int` `addSector​(byte[] buf, int offset, int length)` Adds a new sector into the store.
`int` `calculateSectorSpan​(int length)` Calculates the number of sectors the given length of bytes will span.
`boolean` `clearDeletedSectors()` Cleans up so all deleted sectors are completely removed from the store.
`void` `close()` Closes the data store.
`void` `copyTo​(java.io.File path)` Copies the entire contents of this store to a destination directory.
`void` `delete()` Deletes the data store from the file system.
`void` `deleteAcross​(int sector_head)` Deletes a set of sectors that have been chained together.
`void` `deleteAllSectors()` Deletes all sectors in the entire store.
`void` `deleteSector​(int sector)` Deletes a sector from the store.
`boolean` `exists()` Returns true if the file for this store exists.
`void` `fix​(UserTerminal terminal)` Attempts to repair this data store to a correct state.
`int` `getLengthOfLastOutputStream()` Returns the number of bytes that were written out by the last closed output stream returned by 'getSectorOutputStream'.
`byte[]` `getSector​(int sector)` Gets the contents of the sector at the given index.
`byte[]` `getSector​(int sector, byte[] buf)` Gets the contents of the sector at the given index.
`byte[]` `getSector​(int sector, byte[] buf, int offset, int length)` Gets the contents of the sector at the given index.
`int[]` `getSectorAsIntArray​(int sector, int[] buf)` Gets the contents of the sector at the given index as an int[] array.
`int[]` `getSectorChain​(int sector_head)` Traverses a sector chain and returns an array of all sectors that are part of the chain.
`int[]` `getSectorChain​(int sector_head, int length)` Traverses a sector chain and returns an array of all sectors that are part of the chain.
`java.io.InputStream` `getSectorInputStream​(int sector_head)` Returns an InputStream implementation that is used to read a stream of information from the store.
`int` `getSectorOfLastOutputStream()` Returns the first sector the OutputStream returned by 'getSectorOutputStream' wrote to.
`java.io.OutputStream` `getSectorOutputStream()` Returns an OutputStream implementation that is used to write a stream of information into this data store.
`int` `getSectorSize()` Returns the number of bytes that the user may store in a sector.
`int` `getSectorUseCount()` Returns the number of sectors in the store that are being used (as opposed to being deleted).
`void` `hardSynch()` Performs a hard synchronization of this store.
`boolean` `isClosed()` Returns true if the store is closed.
`boolean` `isReadOnly()` Returns true if the store has been opened in read only mode.
`boolean` `isSectorDeleted​(int sector)` Returns true if the sector number is flagged as deleted.
`void` `lock()` Locks the store by some process so that we may not reclaim deleted sectors.
`boolean` `open​(boolean read_only)` Opens the data store.
`int` `overwriteSector​(int sector, byte[] buf)` Writes the contents of a sector into the store overwritting any other information that may be stored there.
`int` `overwriteSector​(int sector, byte[] buf, int offset, int length)` Writes the contents of a sector into the store overwritting any other information that may be stored there.
`int` `rawSectorCount()` Returns the total number of sectors that are currently available (includes used and deleted sectors).
`int` `readAcross​(int sector_head, byte[] buf, int offset, int length)` Reads information across a chain of sectors and fills the byte[] array buffer.
`void` `readReservedBuffer​(byte[] info, int offset, int length)` Reads from the buffer reserve into the given byte array.
`void` `repair()` Repairs the consistancy of the store.
`java.lang.String` `statusString()` Returns a string that contains diagnostic information.
`void` `synch()` Synchronizes the memory store with the file header.
`long` `totalSize()` Returns the size of the data store file.
`void` `unlock()` Unlocks the store.
`void` `wipeLastOutputStream()` Wipes the SectorOutputStream from this object.
`int` `writeAcross​(byte[] buf, int offset, int length)` Writes a byte[] array of data across as many sectors as it takes to store the data.
`void` `writeReservedBuffer​(byte[] info, int offset, int length)`
`void` `writeReservedBuffer​(byte[] info, int offset, int length, int res_offset)` Every data store has a 128 byte buffer that can be used to store state information.

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Constructor Detail

#### FixedSizeDataStore

    public FixedSizeDataStore​(java.io.File data_file,
                              int sector_size,
                              boolean cache_access,
                              DebugLogger logger)

Constructs the data store. If 'sector_size' <= 0 then we determine sector size when the file is opened. If cached_access is true then all access to the store is through a cache which will greatly improve the performance of read dominated access.

#### FixedSizeDataStore

    public FixedSizeDataStore​(java.io.File data_file,
                              int sector_size,
                              DebugLogger logger)

### Method Detail

#### totalSize

    public long totalSize()

Returns the size of the data store file. This is the total number of bytes stored in the data store.

#### writeReservedBuffer

    public void writeReservedBuffer​(byte[] info,
                                    int offset,
                                    int length,
                                    int res_offset)
                             throws java.io.IOException

Every data store has a 128 byte buffer that can be used to store state information. The buffer starts at offset 64 of the file until offset 192. This method writes data to that offset.

Throws:
  `java.io.IOException`

#### writeReservedBuffer

    public void writeReservedBuffer​(byte[] info,
                                    int offset,
                                    int length)
                             throws java.io.IOException

Throws:
  `java.io.IOException`

#### readReservedBuffer

    public void readReservedBuffer​(byte[] info,
                                   int offset,
                                   int length)
                            throws java.io.IOException

Reads from the buffer reserve into the given byte array.

Throws:
  `java.io.IOException`

#### synch

    public void synch()
               throws java.io.IOException

Synchronizes the memory store with the file header. This writes information into the header. This should be called periodically. Synch does nothing for a read only store.

Throws:
  `java.io.IOException`

#### hardSynch

    public void hardSynch()
                   throws java.io.IOException

Performs a hard synchronization of this store. This will force the OS to synchronize the contents of the data store. hardSynch does nothing for a read only store.

Throws:
  `java.io.IOException`

#### isReadOnly

    public boolean isReadOnly()

Returns true if the store has been opened in read only mode.

#### open

    public boolean open​(boolean read_only)
                 throws java.io.IOException

Opens the data store. The data store can be opened in 'read only' mode. Returns 'true' if the open procedure should repair itself (dirty open) or false if the file was cleanly closed down.

It is not possible to open a damaged store in read only mode.

Parameters:
  `read_only` - if true, then the database is opened in read only mode, otherwise it is opened in read/write mode.
Throws:
  `java.io.IOException`

#### close

    public void close()
               throws java.io.IOException

Closes the data store.

Throws:
  `java.io.IOException`

#### isClosed

    public boolean isClosed()

Returns true if the store is closed.

#### delete

    public void delete()

Deletes the data store from the file system.

#### exists

    public boolean exists()
                   throws java.io.IOException

Returns true if the file for this store exists.

Throws:
  `java.io.IOException`

#### getSectorSize

    public int getSectorSize()

Returns the number of bytes that the user may store in a sector. The actual sector space in the file may be slightly larger.

#### getSectorUseCount

    public int getSectorUseCount()

Returns the number of sectors in the store that are being used (as opposed to being deleted).

#### rawSectorCount

    public int rawSectorCount()
                       throws java.io.IOException

Returns the total number of sectors that are currently available (includes used and deleted sectors).

Throws:
  `java.io.IOException`

#### lock

    public void lock()

Locks the store by some process so that we may not reclaim deleted sectors. The purpose of this is in the situation where we have a slow thread accessing information from the store, and a seperate thread is still able to modifying (delete and add) to the store.

#### unlock

    public void unlock()

Unlocks the store.

#### isSectorDeleted

    public boolean isSectorDeleted​(int sector)
                            throws java.io.IOException

Returns true if the sector number is flagged as deleted. If returns false then the sector is being used.

Throws:
  `java.io.IOException`

#### getSector

    public byte[] getSector​(int sector,
                            byte[] buf,
                            int offset,
                            int length)
                     throws java.io.IOException

Gets the contents of the sector at the given index.

Throws:
  `java.io.IOException`

#### getSector

    public byte[] getSector​(int sector,
                            byte[] buf)
                     throws java.io.IOException

Gets the contents of the sector at the given index.

Throws:
  `java.io.IOException`

#### getSector

    public byte[] getSector​(int sector)
                     throws java.io.IOException

Gets the contents of the sector at the given index.

Throws:
  `java.io.IOException`

#### getSectorAsIntArray

    public int[] getSectorAsIntArray​(int sector,
                                     int[] buf)
                              throws java.io.IOException

Gets the contents of the sector at the given index as an int[] array. The array size is /4 of the sector size. If the sector size is not divisible by 4 then the last 1-3 bytes are truncated.

Throws:
  `java.io.IOException`

#### readAcross

    public int readAcross​(int sector_head,
                          byte[] buf,
                          int offset,
                          int length)
                   throws java.io.IOException

Reads information across a chain of sectors and fills the byte[] array buffer. Returns the number of bytes that were read (should always be equal to 'length').

Throws:
  `java.io.IOException`

#### getSectorChain

    public int[] getSectorChain​(int sector_head,
                                int length)
                         throws java.io.IOException

Traverses a sector chain and returns an array of all sectors that are part of the chain. Useful for diagnostic, repair and statistical operations.

Throws:
  `java.io.IOException`

#### getSectorChain

    public int[] getSectorChain​(int sector_head)
                         throws java.io.IOException

Traverses a sector chain and returns an array of all sectors that are part of the chain. Useful for diagnostic, repair and statistical operations.

Throws:
  `java.io.IOException`

#### deleteSector

    public void deleteSector​(int sector)
                      throws java.io.IOException

Deletes a sector from the store. The sector is only marked as deleted, however, and the contents may still be accessed via the 'getSector' methods. If the store is add locked, then it is guarenteed that no deleted sectors will be overwritten until the add lock is taken from the table.

Throws an IO error if the sector is marked as deleted.

Throws:
  `java.io.IOException`

#### deleteAcross

    public void deleteAcross​(int sector_head)
                      throws java.io.IOException

Deletes a set of sectors that have been chained together. This should be used to delete data added via the 'write' method. However, it can be used to delete data added via the 'addSector'

Throws:
  `java.io.IOException`

#### deleteAllSectors

    public void deleteAllSectors()
                          throws java.io.IOException

Deletes all sectors in the entire store. Use with care.

Throws:
  `java.io.IOException`

#### overwriteSector

    public int overwriteSector​(int sector,
                               byte[] buf,
                               int offset,
                               int length)
                        throws java.io.IOException

Writes the contents of a sector into the store overwritting any other information that may be stored there. This is used as a rough data editting command.

Throws:
  `java.io.IOException`

#### overwriteSector

    public int overwriteSector​(int sector,
                               byte[] buf)
                        throws java.io.IOException

Writes the contents of a sector into the store overwritting any other information that may be stored there. This is used as a rough data editting command.

Throws:
  `java.io.IOException`

#### addSector

    public int addSector​(byte[] buf,
                         int offset,
                         int length)
                  throws java.io.IOException

Adds a new sector into the store. It finds a suitable sector to store the information and returns the sector number. If lock_count > 0 then we do not reclaim deleted sectors, otherwise we do.

Throws:
  `java.io.IOException`

#### addSector

    public int addSector​(byte[] buf)
                  throws java.io.IOException

Adds a new sector into the store. It finds a suitable sector to store the information and returns the sector number. If lock_count > 0 then we do not reclaim deleted sectors, otherwise we do.

Throws:
  `java.io.IOException`

#### calculateSectorSpan

    public int calculateSectorSpan​(int length)

Calculates the number of sectors the given length of bytes will span.

#### writeAcross

    public int writeAcross​(byte[] buf,
                           int offset,
                           int length)
                    throws java.io.IOException

Writes a byte[] array of data across as many sectors as it takes to store the data. Returns the index to the first sector that contains the start of the data.

Throws:
  `java.io.IOException`

#### getSectorOutputStream

    public java.io.OutputStream getSectorOutputStream()
                                               throws java.io.IOException

Returns an OutputStream implementation that is used to write a stream of information into this data store. As data is written into the stream, the data is flushed into this store at the next available sector. When the stream is closed, the entire contents of the stream will be contained within the store. A call to 'getSectorOfLastOutputStream' can be used to return an index that is used to reference this stream of information in the store.

NOTE: While an output stream returned by this method is not closed, it is unsafe to use any methods in the FixedSizeDataStore object.

Throws:
  `java.io.IOException`

#### getSectorOfLastOutputStream

    public int getSectorOfLastOutputStream()

Returns the first sector the OutputStream returned by 'getSectorOutputStream' wrote to. This is the start of the chain.

#### getLengthOfLastOutputStream

    public int getLengthOfLastOutputStream()

Returns the number of bytes that were written out by the last closed output stream returned by 'getSectorOutputStream'.

#### wipeLastOutputStream

    public void wipeLastOutputStream()

Wipes the SectorOutputStream from this object. This should be closed after the stream is closed.

#### getSectorInputStream

    public java.io.InputStream getSectorInputStream​(int sector_head)
                                             throws java.io.IOException

Returns an InputStream implementation that is used to read a stream of information from the store. This input stream will iterate through the sector chain given.

NOTE: Using this InputStream, an end of stream identifier is never produced. When the last sector in the chain is reached, the input stream will first read padding whitespace, then it will either loop to the start of the last sector, or move to another undefined sector. You must not rely on this stream reaching an EOF.

Throws:
  `java.io.IOException`

#### copyTo

    public void copyTo​(java.io.File path)
                throws java.io.IOException

Copies the entire contents of this store to a destination directory. This can only be called when the data store is open. It makes an exact copy of the file.

The purpose of this method is so we can make a copy of the data in this store while the store is open and 'live'.

We assume synchronization on this object.

Parameters:
  `path` - the directory to copy this file to.
Throws:
  `java.io.IOException`

#### fix

    public void fix​(UserTerminal terminal)
             throws java.io.IOException

Attempts to repair this data store to a correct state. The UserTerminal object can be used to ask the user questions and to output information on the progress of the repair.

The store must have been opened before this method is called.

Throws:
  `java.io.IOException`

#### clearDeletedSectors

    public boolean clearDeletedSectors()
                                throws java.io.IOException

Cleans up so all deleted sectors are completely removed from the store. This has the effect of reducing the size of the file by the size of every deleted sector.

It is extremely important that nothing can be read/written from the file while this is happening. And certainly, we can not have any locks on this store.

Returns true if the layout of the sectors changed (so we can fix indices that point to sectors).

Throws:
  `java.io.IOException`

#### repair

    public void repair()
                throws java.io.IOException

Repairs the consistancy of the store. This is an expensive operation that runs through every sector and determines if it's deleted or used. If it's deleted it is added into the deleted linked list.

Repair assumes we can at least get past the 'open' method. This method does not change the order of the sectors in the store. However it may change the order in which deleted sectors are reclaimed.

In a perfect world, this should never need to be called. However, it's a good idea to call this every so often because we are assured that the delete linked list and 'used_sector_count' variables will be correct when the method returns.

It is not possible to repair a store that's been opened in read only mode.

Throws:
  `java.io.IOException`

#### statusString

    public java.lang.String statusString()
                                  throws java.io.IOException

Returns a string that contains diagnostic information.

Throws:
  `java.io.IOException`