primeqa.ir.dense.dpr_top.dataloader.distloader_base.MultiFileLoader

class primeqa.ir.dense.dpr_top.dataloader.distloader_base.MultiFileLoader(hypers: primeqa.ir.dense.dpr_top.torch_util.hypers_base.HypersBase, per_gpu_batch_size: int, train_dir: str, *, checkpoint_info=None, files_per_dataloader=1, uneven_batches=False)

Bases: object

handles the multi-file splitting across processes and the checkpointing

Methods

all_batches
batch_dict
display_batch
get_checkpoint_info
get_dataloader
quick_test
reset