News

Facebook AI's KILT Benchmark Aligns 11 Data Sets to a Single Source

• By John K. Waters
• 09/23/2020

Facebook AI has published a unified benchmark to help artificial intelligence (AI) researchers build models that are better able to leverage real-world knowledge, the group announced this week.

KILT (Knowledge Intensive Language Tasks) unifies its 11 data sets in a single format and grounds them in a single preprocessed collection of the entire Wikipedia corpus. In other words, all the data sets in KILT are aligned with a snapshot of the entire contents of Wikipedia, which serves as a single knowledge source. Mapping all data sets to a single source makes research work in this area much more convenient, and also enables more accurate and balanced evaluation across different models, Facebook says.

"When evaluating how models perform on knowledge-based tasks," the Facebook AI blog explains," it's important to consider not just the particular output but also the specific information used to produce it. The KILT benchmark includes provenance information, or the mapping of the correct knowledge that can solve the task. For several tasks, we make the provenance annotation more comprehensive with an annotation campaign. Together, the output and provenance allow researchers to assess a model's accuracy and its ability to justify a model prediction.

The KILT benchmark is described in a paper ("KILT: a Benchmark for Knowledge Intensive Language Tasks"), which can be downloaded here.

The KILT benchmark consists of 11 datasets spanning 5 distinct tasks (fact-checking, open-domain question answering, slot filling, entity linking, and dialog generation), and includes the test set for all datasets considered, the paper's authors explained.

"An important aim of KILT is cover many different ways of seeking knowledge," they wrote. "For this reason, we select tasks that provide a variety of ways to formulate both the input query (e.g., a claim to verify, a text chunk to annotate, a structured query, a natural question or a conversation) and the expected output (e.g., discrete, extractive, or abstractive)."

Facebook AI has release The KILT Library, an open-source library with multi-framework connectors to most of the retrieval baselines. "We will continue adding baselines and pre-trained models to the library, as well as logic to interchange and experiment with different modular components," the researchers wrote.

The KILT Library is available on GitHub.

"The goal [of the KILT research] is to catalyze and facilitate research towards general and explainable models equipped with task-agnostic representations of knowledge," the researchers concluded. "[W]e plan to explore multi-task learning to exploit synergies between KILT tasks and datasets in the future, and to develop general approaches for representing largescale textual knowledge sources that are useful for multiple downstream tasks."