primeqa.mrc.metrics.nq_f1.nq_f1.NQF1#

class primeqa.mrc.metrics.nq_f1.nq_f1.NQF1(config_name: Optional[str] = None, keep_in_memory: bool = False, cache_dir: Optional[str] = None, num_process: int = 1, process_id: int = 0, seed: Optional[int] = None, experiment_id: Optional[str] = None, max_concurrent_cache_files: int = 10000, timeout: Union[int, float] = 100, **kwargs)#

Bases: datasets.metric.Metric

The F1 score is the harmonic mean of the precision and recall. It can be computed with: F1 = 2 * (precision * recall) / (precision + recall).

Parameters

predictions – Predicted labels.
references – Ground truth labels.

Returns: metrics dict comprising:

LONG ANSWER R@P TABLE.

SHORT ANSWER R@P TABLE.

Methods

`add`	Add one prediction and reference for the metric's stack.
`add_batch`	Add a batch of predictions and references for the metric's stack.
`compute`	Compute the metrics.
`download_and_prepare`	Downloads and prepares dataset for reading.

Attributes

`citation`
`codebase_urls`
`description`
`experiment_id`
`features`
`format`
`homepage`
`info`	`datasets.MetricInfo` object containing all the metadata in the metric.
`inputs_description`
`license`
`name`
`reference_urls`
`streamable`

add(*, prediction=None, reference=None, **kwargs)#

Add one prediction and reference for the metric’s stack.

Parameters

prediction (list/array/tensor, optional) – Predictions.
reference (list/array/tensor, optional) – References.

add_batch(*, predictions=None, references=None, **kwargs)#

Add a batch of predictions and references for the metric’s stack.

Parameters

predictions (list/array/tensor, optional) – Predictions.
references (list/array/tensor, optional) – References.

compute(*, predictions=None, references=None, **kwargs) → Optional[dict]#

Compute the metrics.

Usage of positional arguments is not allowed to prevent mistakes.

Parameters

predictions (list/array/tensor, optional) – Predictions.
references (list/array/tensor, optional) – References.
**kwargs (optional) – Keyword arguments that will be forwarded to the metrics _compute() method (see details in the docstring).

Returns

dict or None

Dictionary with the metrics if this metric is run on the main process (process_id == 0).
None if the metric is not run on the main process (process_id != 0).

download_and_prepare(download_config: Optional[datasets.utils.file_utils.DownloadConfig] = None, dl_manager: Optional[datasets.utils.download_manager.DownloadManager] = None)#

Downloads and prepares dataset for reading.

Parameters

download_config (DownloadConfig, optional) – Specific download configuration parameters.
dl_manager (DownloadManager, optional) – Specific download manager to use.

property info#: datasets.MetricInfo object containing all the metadata in the metric.