Skip to main content

ParallelLoader

ParallelLoader Objects​

class ParallelLoader(ParallelQuery.ParallelQuery)

Parallel and Batch Loader for ApertureDB

This takes a dataset (which is a collection of homogeneous objects) or a derived class, and optimally inserts them into database by splitting them into batches, and passing the batches to multiple workers.

ingest​

def ingest(generator,
           batchsize: int = 1,
           numthreads: int = 4,
           stats: bool = False) -> None

Method to ingest data into the database

Arguments:

  • generator type - The list of data, or a class derived from Subscriptable to be ingested.
  • batchsize int, optional - The size of batch to be used. Defaults to 1.
  • numthreads int, optional - Number of workers to create. Defaults to 4.
  • stats bool, optional - If stats need to be presented, realtime. Defaults to False.