Module com.renomad.minum
Package com.renomad.minum.database

Class DbEngine2<T extends DbData<?>>

java.lang.Object
com.renomad.minum.database.AbstractDb<T>
com.renomad.minum.database.DbEngine2<T>
Type Parameters:
T - the type of data we'll be persisting (must extend from DbData)
public final class DbEngine2<T extends DbData<?>> extends AbstractDb<T>
a memory-based disk-persisted database class.

Engine 2 is a database engine that improves on the performance from the first database provided by Minum. It does this by using different strategies for disk persistence.

The mental model of the previous Minum database has been an in-memory data structure in which every change is eventually written to its own file on disk for persistence. Data changes affect just their relevant files. The benefit of this approach is extreme simplicity. It requires very little code, relying as it does on the operating system's file capabilities.

However, there are two performance problems with this approach. First is when the data changes are arriving at a high rate. In that situation, the in-memory portion keeps up to date, but the disk portion may lag by minutes. The second problem is start-up time. When the database starts, it reads files into memory. The database can read about 6,000 files a second in the best case. If there are a million data items, it would take about 160 seconds to load it into memory, which is far too long.

The new approach to disk persistence is to append each change to a file. Append-only file changes can be very fast. These append files are eventually consolidated into files partitioned by their index - data with indexes between 1 and 1000 go into one file, between 1001 and 2000 go into another, and so on.

Startup is magnitudes faster by this approach. What took the previous database 160 seconds to load requires only 2 seconds. Writes to disk are also faster. What would have taken several minutes to write should only take a few seconds now.

This new approach uses a different file structure than the previous. If it is desired to use the new engine on existing data, it is possible to convert the old data format to the new. Construct an instance of the new engine, pointing at the same name as the previous, and it will convert the data. If the previous call looked like this:

Db photoDb = context.getDb("photos", Photograph.EMPTY);

Then converting to the new database is just replacing it with the following line. Please, backup your database before this change.

DbEngine2 photoDb = context.getDb2("photos", Photograph.EMPTY);

Once the new engine starts up, it will notice the old file structure and convert it over. The methods and behaviors are mostly the same between the old and new engines, so the update should be straightforward.

(By the way, it *is* possible to convert back to the old file structure, by starting the database the old way again. Just be aware that each time the files are converted, it takes longer than normal to start the database)

However, something to note is that using the old database is still fine in many cases, particularly for prototypes or systems which do not contain large amounts of data. If your system is working fine, there is no need to change things.

  • Constructor Details

    • DbEngine2

      public DbEngine2(Path dbDirectory, Context context, T instance)
      Constructs an in-memory disk-persisted database. Loading of data from disk happens at the first invocation of any command changing or requesting data, such as write(DbData), delete(DbData), or values(). See the private method loadData() for details.
      Parameters:
      dbDirectory - this uniquely names your database, and also sets the directory name for this data. The expected use case is to name this after the data in question. For example, "users", or "accounts".
      context - used to provide important state data to several components
      instance - an instance of the DbData object relevant for use in this database. Note that each database (that is, each instance of this class), focuses on just one data, which must be an implementation of DbData.

  • Method Details

    • write

      public T write(T newData)
      Write data to the database. Use an index of 0 to store new data, and a positive non-zero value to update data.

      Example of adding new data to the database: 

               final var newSalt = StringUtils.generateSecureRandomString(10);
         final var hashedPassword = CryptoUtils.createPasswordHash(newPassword, newSalt);
         final var newUser = new User(0L, newUsername, hashedPassword, newSalt);
         userDb.write(newUser);

      Example of updating data: 

              // write the updated salted password to the database
        final var updatedUser = new User(
                user().getIndex(),
                user().getUsername(),
                hashedPassword,
                newSalt);
        userDb.write(updatedUser);
      Specified by:
      write in class AbstractDb<T extends DbData<?>>
      Parameters:
      newData - the data we are writing
      Returns:
      the data with its new index assigned.
      Throws:
      DbException - if there is a failure to write

    • delete

      public void delete(T dataToDelete)
      Delete data

      Example:

           userDb.delete(user);
      Specified by:
      delete in class AbstractDb<T extends DbData<?>>
      Parameters:
      dataToDelete - the data we are serializing and writing
      Throws:
      DbException - if there is a failure to delete

    • loadData

      public void loadData()
      This is what loads the data from disk the first time someone needs it. Because it is locked, only one thread can enter at a time. The first one in will load the data, and the second will encounter a branch which skips loading.
      Specified by:
      loadData in class AbstractDb<T extends DbData<?>>

    • values

      public Collection<T> values()
      This method provides read capability for the values of a database.
      The returned collection is a read-only view over the data, through Collections.unmodifiableCollection(Collection)

      Example:

      boolean doesUserAlreadyExist(String username) {
    return userDb.values().stream().anyMatch(x -> x.getUsername().equals(username));
}
      Specified by:
      values in class AbstractDb<T extends DbData<?>>

    • registerIndex

      public boolean registerIndex(String indexName, Function<T,String> keyObtainingFunction)
      Description copied from class: AbstractDb
      Register an index in the database for higher performance data access.

      This command should be run immediately after database declaration, or more specifically, before any data is loaded from disk. Otherwise, it would be possible to skip indexing that data.


      Example: 
               final var myDatabase = context.getDb("photos", Photograph.EMPTY);
         myDatabase.registerIndex("url", photo -> photo.getUrl());
      Overrides:
      registerIndex in class AbstractDb<T extends DbData<?>>
      Parameters:
      indexName - a string used to distinguish this index. This string will be used again when requesting data in a method like AbstractDb.getIndexedData(java.lang.String, java.lang.String) or AbstractDb.findExactlyOne(java.lang.String, java.lang.String)
      keyObtainingFunction - a function which obtains data from the data in this database, used to partition the data into groups (potentially up to a 1-to-1 correspondence between id and object)
      Returns:
      true if the registration succeeded

    • getIndexedData

      public Collection<T> getIndexedData(String indexName, String key)
      Description copied from class: AbstractDb
      Given the name of a registered index (see AbstractDb.registerIndex(String, Function)), use the key to find the collection of data that matches it.
      Overrides:
      getIndexedData in class AbstractDb<T extends DbData<?>>
      Parameters:
      indexName - the name of an index
      key - a string value that matches a partition calculated from the partition function provided to AbstractDb.registerIndex(String, Function)
      Returns:
      a collection of data, an empty collection if nothing found

    • flush

      public void flush()
      This command calls DatabaseAppender.flush(), which will force any in-memory-buffered data to be written to disk. This is not commonly necessary to call for business purposes, but tests may require it if you want to be absolutely sure the data is written to disk at a particular moment.

    • stop

      public void stop()
      This is here to match the contract of Db but all it does is tell the interior file writer to write its data to disk.
      Specified by:
      stop in class AbstractDb<T extends DbData<?>>

    • stop

      public void stop(int count, int sleepTime)
      No real difference to stop() but here to have a similar contract to Db
      Specified by:
      stop in class AbstractDb<T extends DbData<?>>
      Parameters:
      count - number of loops before we are done waiting for a clean close and instead crash the instance closed.
      sleepTime - how long to wait, in milliseconds, for each iteration of the waiting loop.