The prime repository for state-of-the-art Multilingual and Multimedia Question Answering research and development.

PrimeQA is a public open source repository that enables researchers and developers to train state-of-the-art models for question answering (QA). By using PrimeQA, a researcher can replicate the experiments outlined in a paper published in the latest NLP conference while also enjoying the capability to download pre-trained models (from an online repository) and run them on their own custom data. PrimeQA is built on top of the Transformers toolkit and uses datasets and models that are directly downloadable.

The models within PrimeQA supports End-to-end Question Answering. PrimeQA answers questions via

Information Retrieval: Retrieving documents and passages using both traditional (e.g. BM25) and neural (e.g. ColBERT) models
Multilingual Machine Reading Comprehension: Extract and/ or generate answers given the source document or passage.
Multilingual Question Generation: Supports generation of questions for effective domain adaptation over tables and multilingual text.

Some examples of models (applicable on benchmark datasets) supported are :

Traditional IR with BM25 Pyserini
Neural IR with ColBERT, DPR (collaboration with Stanford NLP IR led by Chris Potts & Matei Zaharia). Replicating the experiments that Dr. Decr (Li et. al, 2022) performed to reach the top of the XOR TyDI leaderboard.
Machine Reading Comprehension with XLM-R: to replicate the experiments to get to the top of the TyDI leaderboard similar to the performance of the IBM GAAMA system. Coming soon: code to replicate GAAMA’s performance on Natural Questions.
Multimedia QA over news & movies: coming soon! to replicate the experiments run over multi-hop QA over images, text over variety of domains. Collaboration with UIUC Blender lab.

🏅 Top of the Leaderboard#

PrimeQA is at the top of several leaderboards: XOR-TyDi, TyDiQA-main, OTT-QA and HybridQA.

XOR-TyDi #

TyDiQA-main #

OTT-QA #

HybridQA #

Getting Started#

Installation

New to PrimeQA? Check out the getting started guide.

Tutorials

Try out PrimeQA components via these tutorials for building your own search applications.

Development

Saw a typo in the documentation? Want to improve existing functionalities? The contributing guidelines will guide you through the process of improving PrimeQA.

Testing

Our goal is that every module and package should have a thorough set of unit tests. These tests should exercise the full functionality as well as its robustness to erroneous or unexpected input argument.

Learn more#

Section	Description
Quick tour: Entry Points for PrimeQA	Different entry points for PrimeQA: Information Retrieval, Reading Comprehension, TableQA and Question Generation
Tutorials: Jupyter Notebooks	Notebooks to get started on QA tasks
Examples: Applying PrimeQA on various QA tasks	Example scripts for fine-tuning PrimeQA models on a range of QA tasks
Model sharing and uploading	Upload and share your fine-tuned models with the community
Pull Request	PrimeQA Pull Request
Generate Documentation	How Documentation works
Orchestrator Service REST Microservice	Proof-of-concept code for PrimeQA Orchestrator microservice
Tooling UI	Demo UI