primeqa.ir.dense.dpr_top.dpr.dpr_util.tokenize_queries

primeqa.ir.dense.dpr_top.dpr.dpr_util.tokenize_queries(tokenizer: Union[transformers.models.rag.tokenization_rag.RagTokenizer, transformers.models.dpr.tokenization_dpr.DPRQuestionEncoderTokenizer, transformers.models.dpr.tokenization_dpr_fast.DPRQuestionEncoderTokenizerFast], queries: Union[List[str], List[Tuple[str, str]]], *, max_length: int)

Parameters

• tokenizer – a RagTokenizer (the question_encoder will be used) or a DPRQuestionEncoderTokenizer

• queries – list of string queries or list of title/text pairs

• max_length –

Returns