primeqa.mrc.data_models.data_collator.FiDDataCollator

class primeqa.mrc.data_models.data_collator.FiDDataCollator(tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, model: Optional[Any] = None, padding: Union[bool, str, transformers.file_utils.PaddingStrategy] = True, max_length: Optional[int] = None, pad_to_multiple_of: Optional[int] = None, label_pad_token_id: int = - 100, return_tensors: str = 'pt')

Bases: object

Modified from DataCollatorForSeq2Seq Do not pad the features for FID The padding has been done in the preprocessor

Methods

Attributes

label_pad_token_id
max_length
model
pad_to_multiple_of
padding
return_tensors
tokenizer

__call__(features, return_tensors=None)

Call self as a function.