primeqa.mrc.metrics.nq_f1.nq_eval.score_short_answer#

primeqa.mrc.metrics.nq_f1.nq_eval.score_short_answer(gold_label_list, pred_label, short_non_null_threshold=None)#

Scores a short answer as correct or not.

  1. First decide if there is a gold short answer with SHORT_NO_NULL_THRESHOLD.

  2. The prediction will get a match if: a. There is a gold short answer. b. The prediction span set match exactly with one of the non-null gold

    short answer span set.

Parameters

  • gold_label_list – A list of NQLabel.

  • pred_label – A single NQLabel.

  • short_non_null_threshold – Min number of non null annotations required before considering the question as having a non null answer. Optional.

Returns

gold_has_answer, pred_has_answer, is_correct, score