primeqa.mrc.metrics.nq_f1.nq_f1.NQF1#

class primeqa.mrc.metrics.nq_f1.nq_f1.NQF1(config_name: Optional[str] = None, keep_in_memory: bool = False, cache_dir: Optional[str] = None, num_process: int = 1, process_id: int = 0, seed: Optional[int] = None, experiment_id: Optional[str] = None, max_concurrent_cache_files: int = 10000, timeout: Union[int, float] = 100, **kwargs)#

Bases: datasets.metric.Metric

The F1 score is the harmonic mean of the precision and recall. It can be computed with: F1 = 2 * (precision * recall) / (precision + recall).

Parameters
  • predictions – Predicted labels.

  • references – Ground truth labels.

Returns: metrics dict comprising:

  • LONG ANSWER R@P TABLE.

  • SHORT ANSWER R@P TABLE.

Methods

add

Add one prediction and reference for the metric's stack.

add_batch

Add a batch of predictions and references for the metric's stack.

compute

Compute the metrics.

download_and_prepare

Downloads and prepares dataset for reading.

Attributes

citation

codebase_urls

description

experiment_id

features

format

homepage

info

datasets.MetricInfo object containing all the metadata in the metric.

inputs_description

license

name

reference_urls

streamable

add(*, prediction=None, reference=None, **kwargs)#

Add one prediction and reference for the metric’s stack.

Parameters
  • prediction (list/array/tensor, optional) – Predictions.

  • reference (list/array/tensor, optional) – References.

add_batch(*, predictions=None, references=None, **kwargs)#

Add a batch of predictions and references for the metric’s stack.

Parameters
  • predictions (list/array/tensor, optional) – Predictions.

  • references (list/array/tensor, optional) – References.

compute(*, predictions=None, references=None, **kwargs) Optional[dict]#

Compute the metrics.

Usage of positional arguments is not allowed to prevent mistakes.

Parameters
  • predictions (list/array/tensor, optional) – Predictions.

  • references (list/array/tensor, optional) – References.

  • **kwargs (optional) – Keyword arguments that will be forwarded to the metrics _compute() method (see details in the docstring).

Returns

dict or None

  • Dictionary with the metrics if this metric is run on the main process (process_id == 0).

  • None if the metric is not run on the main process (process_id != 0).

download_and_prepare(download_config: Optional[datasets.utils.file_utils.DownloadConfig] = None, dl_manager: Optional[datasets.utils.download_manager.DownloadManager] = None)#

Downloads and prepares dataset for reading.

Parameters
  • download_config (DownloadConfig, optional) – Specific download configuration parameters.

  • dl_manager (DownloadManager, optional) – Specific download manager to use.

property info#

datasets.MetricInfo object containing all the metadata in the metric.