eval_utils

Functions

gold_has_long_answer	Gets vote from multi-annotators for judging if there is a long answer. :param gold_label_list: list of gold labels for judging :param long_non_null_threshold: Require this many non-null long answer annotations to count gold as containing a long answer. Defaults to 2 like the original paper.
gold_has_short_answer	Gets vote from multi-annotators for judging if there is a short answer. :param gold_label_list: list of gold labels :param short_non_null_threshold: Require this many non-null short answer annotations to count gold as containing a short answer. Defaults to 2 like the original paper.
is_null_span_list	Returns true iff all spans in span_list are null or span_list is empty.
nonnull_span_equal	Given two spans, return if they are equal.
parse_json_as_predictions
parse_json_as_topk_predictions
read_annotation	Read annotations with real multiple processes.
read_annotation_from_one_split	Read annotation from one split of file.
read_prediction_json_from_file	Read the prediction json with scores.
span_set_equal	Make the spans are completely equal besides null spans.

Classes

NQLabel
NQSpan	A class for handling token and byte spans.

Exceptions

InconsistentSpanError