primeqa.ir.dense.dpr_top.util.line_corpus.read_lines

primeqa.ir.dense.dpr_top.util.line_corpus.read_lines(input_files, limit=0, report_every=100000, *, errors=None, shuffled_files=None)

This takes a list (or single) input files and iterates over the lines in them :param input_files: Directory name or list of file names :param limit: maximum number of lines to read :param report_every: log info after this many lines :return: