Interface Database

All Superinterfaces:
AutoCloseable, CauseCloseable, Closeable, Flushable
public interface Database extends CauseCloseable, Flushable
Primary database interface, containing a collection of transactional indexes. Call open to obtain a Database instance. Examples:

Open a non-durable database, limited to a max size of 100MB: 

var config = new DatabaseConfig().maxCacheSize(100_000_000);
Database db = Database.open(config);
Index data = db.openIndex("mydata");

Open a regular database, with a fixed cache size, and a weak durability mode for the best transactional commit performance. 

var config = new DatabaseConfig()
   .baseFilePath("/var/lib/tupl/myapp")
   .cacheSize(100_000_000)
   .durabilityMode(DurabilityMode.NO_FLUSH);

Database db = Database.open(config);
Index data = db.openIndex("mydata");

The following files are created by the above example:

  • /var/lib/tupl/myapp.db – primary database file
  • /var/lib/tupl/myapp.lock – lock file to ensure that at most one process can have the database open
  • /var/lib/tupl/myapp.redo.0 – first transaction redo log file

New redo log files are created by checkpoints, which also delete the old files. When snapshots are in progress, one or more numbered temporary files are created. For example: /var/lib/tupl/myapp.temp.123.

See Also:

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    applyCachePrimer(InputStream in)
    Prime the cache, from a set encoded earlier.
    Snapshot
    beginSnapshot()
    Support for capturing a snapshot (hot backup) of the database, while still allowing concurrent modifications.
    default long
    capacityLimit()
    Returns the current capacity limit, rounded down by page size.
    default void
    capacityLimit(long bytes)
    Set a soft capacity limit for the database, to prevent filling up the storage device.
    default void
    capacityLimitOverride(long bytes)
    Set capacity limits for the current thread, allowing it to perform tasks which can free up space.
    void
    checkpoint()
    Durably sync and checkpoint all changes to the database.
    default void
    close()
    Closes the database, ensuring durability of committed transactions.
    void
    close(Throwable cause)
    Closes the database after an unexpected failure.
    Lock
    commitLock()
    Returns the checkpoint commit lock, which can be held to prevent checkpoints from capturing a safe commit point.
    boolean
    compactFile(CompactionObserver observer, double target)
    Compacts the database by shrinking the database file.
    static Database
    connect(SocketAddress addr, SSLContext context, long... tokens)
    Establish a remote connection to a database which is running a server.
    void
    createCachePrimer(OutputStream out)
    Writes a cache priming set into the given stream, which can then be used later to prime the cache.
    CustomHandler
    customWriter(String name)
    Returns a handler instance suitable for writing custom redo and undo operations.
    Runnable
    deleteIndex(Index index)
    Fully closes and deletes the given index, but does not immediately reclaim the pages it occupied.
    static Database
    destroy(DatabaseConfig config)
    Delete the contents of an existing database, and replace it with an empty one.
    boolean
    failover()
    If the database instance is currently acting as a leader, attempt to give up leadership and become a replica.
    Index
    findIndex(byte[] name)
    Returns the given named index, returning null if not found.
    default Index
    findIndex(String name)
    Returns the given named index, returning null if not found.
    default <R> Table<R>
    findTable(Class<R> type)
    Convenience method which returns a Table that uses the index named by the row type itself.
    void
    flush()
    Flushes all committed transactions, but not durably.
    default Index
    indexById(byte[] id)
    Returns an index by its identifier, returning null if not found.
    Index
    indexById(long id)
    Returns an index by its identifier, returning null if not found.
    View
    indexRegistryById()
    Returns an unmodifiable View which maps all available index identifiers to names.
    View
    indexRegistryByName()
    Returns an unmodifiable View which maps all available index names to identifiers.
    boolean
    isClosed()
    Returns true if database was explicitly closed, or if it was closed due to a panic.
    boolean
    isLeader()
    Returns true if the database instance is currently the leader.
    Server
    newServer()
    Returns an object for enabling remote access into this database.
    Sorter
    newSorter()
    Returns a new Sorter instance.
    Index
    newTemporaryIndex()
    Creates a new unnamed temporary index.
    default Transaction
    newTransaction()
    Returns a new Transaction with the default durability mode.
    Transaction
    newTransaction(DurabilityMode durabilityMode)
    Returns a new Transaction with the given durability mode.
    static Database
    open(DatabaseConfig config)
    Open a database, creating it if necessary.
    Index
    openIndex(byte[] name)
    Returns the given named index, creating it if necessary.
    default Index
    openIndex(String name)
    Returns the given named index, creating it if necessary.
    default <J> Table<J>
    openJoinTable(Class<J> joinType, String spec)
    Convenience method which joins tables together, opening them if necessary.
    default <R> Table<R>
    openTable(Class<R> type)
    Convenience method which returns a Table that uses the index named by the row type itself.
    long
    preallocate(long bytes)
    Preallocates pages for immediate use.
    PrepareHandler
    prepareWriter(String name)
    Returns a handler instance suitable for preparing transactions.
    static Database
    rebuild(DatabaseConfig oldConfig, DatabaseConfig newConfig, int numThreads)
    Open an existing database and copy all the data into a new database, which can have a different configuration.
    void
    renameIndex(Index index, byte[] newName)
    Renames the given index to the one given.
    default void
    renameIndex(Index index, String newName)
    Renames the given index to the one given.
    static Database
    restoreFromSnapshot(DatabaseConfig config, InputStream in)
    Restore from a snapshot, into the data files defined by the given configuration.
    void
    resumeCheckpoints()
    Resume automatic checkpoints after having been temporarily suspended.
    void
    shutdown()
    Cleanly closes the database, ensuring durability of all modifications.
    DatabaseStats
    stats()
    Returns a collection of database statistics.
    void
    suspendCheckpoints()
    Temporarily suspend automatic checkpoints and wait for any in-progress checkpoint to complete.
    void
    sync()
    Durably flushes all committed transactions.
    void
    uponLeader(Runnable acquired, Runnable lost)
    Registers the given task to start in a separate thread when the database instance has become the leader.
    boolean
    verify(VerificationObserver observer, int numThreads)
    Verifies the integrity of the database and all indexes.

  • Method Details

    • open

      static Database open(DatabaseConfig config) throws IOException
      Open a database, creating it if necessary.
      Throws:
      IOException

    • destroy

      static Database destroy(DatabaseConfig config) throws IOException
      Delete the contents of an existing database, and replace it with an empty one. When using a raw block device for the data file, this method must be used to format it. When database is replicated, calling destroy only affects the local replica.
      Throws:
      IOException

    • connect

      static Database connect(SocketAddress addr, SSLContext context, long... tokens) throws IOException
      Establish a remote connection to a database which is running a server.
      Parameters:
      context - optionally pass a context to open a secure connection
      Throws:
      IllegalArgumentException - if not given one or two tokens
      IOException

    • rebuild

      static Database rebuild(DatabaseConfig oldConfig, DatabaseConfig newConfig, int numThreads) throws IOException
      Open an existing database and copy all the data into a new database, which can have a different configuration.
      Parameters:
      numThreads - pass 0 for default, or if negative, the actual number will be (-numThreads * availableProcessors).
      Returns:
      the newly built database
      Throws:
      IllegalStateException - if the new database already exists or if either is replicated
      IOException

    • openIndex

      Index openIndex(byte[] name) throws IOException
      Returns the given named index, creating it if necessary.
      Returns:
      shared Index instance
      Throws:
      IOException

    • openIndex

      default Index openIndex(String name) throws IOException
      Returns the given named index, creating it if necessary. Name is UTF-8 encoded.
      Returns:
      shared Index instance
      Throws:
      IOException

    • findIndex

      Index findIndex(byte[] name) throws IOException
      Returns the given named index, returning null if not found.
      Returns:
      shared Index instance; null if not found
      Throws:
      IOException

    • findIndex

      default Index findIndex(String name) throws IOException
      Returns the given named index, returning null if not found. Name is UTF-8 encoded.
      Returns:
      shared Index instance; null if not found
      Throws:
      IOException

    • indexById

      Index indexById(long id) throws IOException
      Returns an index by its identifier, returning null if not found.
      Returns:
      shared Index instance
      Throws:
      IllegalArgumentException - if id is reserved
      IOException

    • indexById

      default Index indexById(byte[] id) throws IOException
      Returns an index by its identifier, returning null if not found.
      Parameters:
      id - big-endian encoded long integer
      Returns:
      shared Index instance
      Throws:
      IllegalArgumentException - if id is malformed or reserved
      IOException

    • openTable

      default <R> Table<R> openTable(Class<R> type) throws IOException
      Convenience method which returns a Table that uses the index named by the row type itself.
      Returns:
      shared Table instance
      Throws:
      IOException
      See Also:

    • findTable

      default <R> Table<R> findTable(Class<R> type) throws IOException
      Convenience method which returns a Table that uses the index named by the row type itself.
      Returns:
      shared Table instance; null if not found
      Throws:
      IOException
      See Also:

    • openJoinTable

      default <J> Table<J> openJoinTable(Class<J> joinType, String spec) throws IOException
      Convenience method which joins tables together, opening them if necessary.
      Parameters:
      spec - join specification
      Throws:
      NullPointerException - if any parameters are null
      IllegalArgumentException - if join type or specification is malformed
      IOException
      See Also:

    • renameIndex

      void renameIndex(Index index, byte[] newName) throws IOException
      Renames the given index to the one given.
      Parameters:
      index - non-null open index
      newName - new non-null name
      Throws:
      ClosedIndexException - if index reference is closed
      IllegalStateException - if name is already in use by another index
      IllegalStateException - if index belongs to another database instance
      IOException

    • renameIndex

      default void renameIndex(Index index, String newName) throws IOException
      Renames the given index to the one given. Name is UTF-8 encoded.
      Parameters:
      index - non-null open index
      newName - new non-null name
      Throws:
      ClosedIndexException - if index reference is closed
      IllegalStateException - if name is already in use by another index
      IllegalStateException - if index belongs to another database instance
      IOException

    • deleteIndex

      Runnable deleteIndex(Index index) throws IOException
      Fully closes and deletes the given index, but does not immediately reclaim the pages it occupied. Run the returned task in any thread to reclaim the pages.

      Once deleted, accesing the index causes exceptions to be thrown. A new index by the original name can be created, which will be assigned a different unique identifier. Any transactions still referring to the old index will not affect the new index.

      If the deletion task is never started or it doesn't finish normally, it will resume when the database is re-opened. All resumed deletions are completed in serial order by a background thread.

      Parameters:
      index - non-null open index
      Returns:
      non-null task to call for reclaiming the pages used by the deleted index
      Throws:
      ClosedIndexException - if index reference is closed
      IllegalStateException - if index belongs to another database instance
      IOException
      See Also:

    • newTemporaryIndex

      Index newTemporaryIndex() throws IOException
      Creates a new unnamed temporary index. Temporary indexes never get written to the redo log, and they are deleted when the database is re-opened. Temporary indexes should be explicitly deleted when no longer needed, rather than waiting until the database is re-opened.
      Throws:
      IOException

    • indexRegistryByName

      View indexRegistryByName() throws IOException
      Returns an unmodifiable View which maps all available index names to identifiers. Identifiers are long integers, big-endian encoded.
      Throws:
      IOException

    • indexRegistryById

      View indexRegistryById() throws IOException
      Returns an unmodifiable View which maps all available index identifiers to names. Identifiers are long integers, big-endian encoded.
      Throws:
      IOException

    • newTransaction

      default Transaction newTransaction()
      Returns a new Transaction with the default durability mode.

    • newTransaction

      Transaction newTransaction(DurabilityMode durabilityMode)
      Returns a new Transaction with the given durability mode. If null, the default is used.

    • customWriter

      CustomHandler customWriter(String name) throws IOException
      Returns a handler instance suitable for writing custom redo and undo operations. A corresponding recovery instance must have been provided when the database was opened, via the customHandlers config method.
      Returns:
      new writer instance
      Throws:
      IllegalStateException - if no recovery instance by the given name is installed
      IOException

    • prepareWriter

      PrepareHandler prepareWriter(String name) throws IOException
      Returns a handler instance suitable for preparing transactions. A corresponding recovery instance must have been provided when the database was opened, via the prepareHandlers config method.
      Returns:
      new writer instance
      Throws:
      IllegalStateException - if no recovery instance by the given name is installed
      IOException

    • newSorter

      Sorter newSorter()
      Returns a new Sorter instance. The standard algorithm is a parallel external mergesort, which attempts to use all available processors. All external storage is maintained in the database itself, in the form of temporary indexes.

    • preallocate

      long preallocate(long bytes) throws IOException
      Preallocates pages for immediate use. The actual amount allocated varies, depending on the amount of free pages already available.
      Returns:
      actual amount allocated
      Throws:
      IOException

    • capacityLimit

      default void capacityLimit(long bytes)
      Set a soft capacity limit for the database, to prevent filling up the storage device. When the limit is reached, writes might fail with a DatabaseFullException. No explicit limit is defined by default, and the option is ignored by non-durable databases. The limit is checked only when the database attempts to grow, and so it can be set smaller than the current database size.

      Even with a capacity limit, the database file can still slowly grow in size. Explicit overrides and critical operations can allocate space which can only be reclaimed by compaction.

      Parameters:
      bytes - maximum capacity, in bytes; pass -1 for no limit

    • capacityLimit

      default long capacityLimit()
      Returns the current capacity limit, rounded down by page size.
      Returns:
      maximum capacity, in bytes; is -1 if no limit

    • capacityLimitOverride

      default void capacityLimitOverride(long bytes)
      Set capacity limits for the current thread, allowing it to perform tasks which can free up space. While doing so, it might require additional temporary storage.
      Parameters:
      bytes - maximum capacity, in bytes; pass -1 for no limit; pass 0 to remove override

    • beginSnapshot

      Snapshot beginSnapshot() throws IOException
      Support for capturing a snapshot (hot backup) of the database, while still allowing concurrent modifications. The snapshot contains all data up to the last checkpoint. Call the checkpoint method immediately before to ensure that an up-to-date snapshot is captured.

      To restore from a snapshot, store it in the primary data file, which is the base file with a ".db" extension. Make sure no redo log files exist and then open the database. Alternatively, call restoreFromSnapshot, which also supports restoring into separate data files.

      During the snapshot, temporary files are created to hold pre-modified copies of pages. If the snapshot destination stream blocks for too long, these files keep growing. File growth rate increases too if the database is being heavily modified. In the worst case, the temporary files can become larger than the primary database files.

      Returns:
      a snapshot control object, which must be closed when no longer needed
      Throws:
      IOException

    • restoreFromSnapshot

      static Database restoreFromSnapshot(DatabaseConfig config, InputStream in) throws IOException
      Restore from a snapshot, into the data files defined by the given configuration. All existing data and redo log files at the snapshot destination are deleted before the restore begins.
      Parameters:
      in - snapshot source; does not require extra buffering; auto-closed
      Throws:
      IOException

    • createCachePrimer

      void createCachePrimer(OutputStream out) throws IOException
      Writes a cache priming set into the given stream, which can then be used later to prime the cache.
      Parameters:
      out - cache priming destination; buffering is recommended; not auto-closed
      Throws:
      IOException
      See Also:

    • applyCachePrimer

      void applyCachePrimer(InputStream in) throws IOException
      Prime the cache, from a set encoded earlier.
      Parameters:
      in - caching priming source; buffering is recommended; auto-closed
      Throws:
      IOException
      See Also:

    • newServer

      Server newServer() throws IOException
      Returns an object for enabling remote access into this database. As long as the server is still open, the JVM won't exit. Closing the database will also close the server.

      If the database is configured with replication, remote access is already enabled, and so a server doesn't need to be created.

      Throws:
      IOException
      See Also:

    • stats

      DatabaseStats stats()
      Returns a collection of database statistics.

    • flush

      void flush() throws IOException
      Flushes all committed transactions, but not durably. Transactions committed with no-flush effectively become no-sync durable.

      When the database is replicated, the no-flush mode is identical to the no-sync mode. Calling this method on a replicated database has no effect.

      Specified by:
      flush in interface Flushable
      Throws:
      IOException

    • sync

      void sync() throws IOException
      Durably flushes all committed transactions. Transactions committed with no-flush and no-sync effectively become sync durable.
      Throws:
      IOException

    • checkpoint

      void checkpoint() throws IOException
      Durably sync and checkpoint all changes to the database. In addition to ensuring that all committed transactions are durable, checkpointing ensures that non-transactional modifications are durable. Checkpoints are performed automatically by a background thread, at a configurable rate.
      Throws:
      IOException

    • suspendCheckpoints

      void suspendCheckpoints()
      Temporarily suspend automatic checkpoints and wait for any in-progress checkpoint to complete. Suspend may be invoked multiple times, but each must be paired with a resume call to enable automatic checkpoints again.
      Throws:
      IllegalStateException - if suspended more than 231 times

    • resumeCheckpoints

      void resumeCheckpoints()
      Resume automatic checkpoints after having been temporarily suspended.
      Throws:
      IllegalStateException - if resumed more than suspended

    • commitLock

      Lock commitLock()
      Returns the checkpoint commit lock, which can be held to prevent checkpoints from capturing a safe commit point. By holding the commit lock, multiple modifications can be made to the database atomically, even without using a transaction.

      The commit lock should only ever be held briefly, because it prevents checkpoints from starting. Holding the commit lock can stall other threads trying to make modifications, if a checkpoint is trying to start. In addition, a thread holding the commit lock must not attempt to issue a checkpoint, because deadlock is possible.

    • compactFile

      boolean compactFile(CompactionObserver observer, double target) throws IOException
      Compacts the database by shrinking the database file. The compaction target is the desired file utilization, and it controls how much compaction should be performed. A target of 0.0 performs no compaction, and a value of 1.0 attempts to compact as much as possible.

      If the compaction target cannot be met, the entire operation aborts. If the database is being concurrently modified, large compaction targets will likely never succeed. Although compacting by smaller amounts is more likely to succeed, the entire database must still be scanned. A minimum target of 0.5 is recommended for the compaction to be worth the effort.

      Compaction requires some amount of free space for page movement, and so some free space might still linger following a massive compaction. More iterations are required to fully complete such a compaction. The first iteration might actually cause the file to grow slightly. This can be prevented by doing a less massive compaction first.

      Parameters:
      observer - optional observer; pass null for default
      target - database file compaction target [0.0, 1.0]
      Returns:
      false if file compaction aborted
      Throws:
      IllegalArgumentException - if compaction target is out of bounds
      IllegalStateException - if compaction is already in progress
      IOException

    • verify

      boolean verify(VerificationObserver observer, int numThreads) throws IOException
      Verifies the integrity of the database and all indexes. Using multiple threads speeds up verification, even though some nodes might be visited multiple times.
      Parameters:
      observer - optional observer; pass null for default
      numThreads - pass 0 for default, or if negative, the actual number will be (-numThreads * availableProcessors).
      Returns:
      true if verification passed
      Throws:
      IOException

    • isLeader

      boolean isLeader()
      Returns true if the database instance is currently the leader.

    • uponLeader

      void uponLeader(Runnable acquired, Runnable lost)
      Registers the given task to start in a separate thread when the database instance has become the leader. If already the leader, the task is started immediately. The task is started at most once per registration, and a second task is called when leadership is lost. It too is called in a separate thread, and at most once per registration. Note that if leadership is quickly lost, the second task might run before the first task.
      Parameters:
      acquired - called when leadership is acquired (can be null)
      lost - called when leadership is lost (can be null)

    • failover

      boolean failover() throws IOException
      If the database instance is currently acting as a leader, attempt to give up leadership and become a replica. If the database is a replica, or if failover is successful, true is returned. When false is returned, the database is likely still the leader, either because the database isn't replicated, or because no replicas exist to failover to.
      Throws:
      IOException

    • close

      default void close() throws IOException
      Closes the database, ensuring durability of committed transactions. No checkpoint is performed by this method, and so non-transactional modifications can be lost.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException
      See Also:

    • close

      void close(Throwable cause) throws IOException
      Closes the database after an unexpected failure. No checkpoint is performed by this method, and so non-transactional modifications can be lost.
      Specified by:
      close in interface CauseCloseable
      Parameters:
      cause - if non-null, delivers a panic event and future database accesses will rethrow the cause
      Throws:
      IOException
      See Also:

    • isClosed

      boolean isClosed()
      Returns true if database was explicitly closed, or if it was closed due to a panic.

    • shutdown

      void shutdown() throws IOException
      Cleanly closes the database, ensuring durability of all modifications. A checkpoint is issued first, and so a quick recovery is performed when the database is re-opened. As a side effect of shutting down, all extraneous files are deleted.
      Throws:
      IOException