News

Google Open Sources Dialog Datasets for AI Assistant

• By John K. Waters
• 09/16/2019

Google has released its Coached Conversational Preference Elicitation (CCPE) and Taskmaster-1 English dialog datasets to open source. The two collections of pairs of people engaged in spoken conversations are now available to developers of AI assistants as training material for modeling natural language.

The CCPE dataset comprises 502 dialogs with 12,000 "annotated utterances" between a user and an assistant discussing movie preferences in natural language.

The Taskmaster-1 dataset comprises 13,215 English dialogs, including 7,708 written and 5,507 spoken. As its name implies, this dataset is based on several tasks, including ordering pizza, creating auto repair appointments, setting up rides for hire, ordering movie tickets, ordering coffee drinks, and making restaurant reservations.

Both datasets were generated using the "Wizard-of-Oz" (WOZ) method, in which the users believed they were interacting with an autonomous system that was actually being operated by a person -- a human being behind the scenes pulling levers and flipping switches. Also called "Wizard of Oz testing" and "Wizard of Oz prototyping, it's a type of design methodology that's often used to evaluate the usability of speech and natural language systems.

Google AI researchers are sharing these datasets in an effort to address what they see as a fundamental problem in this space, what Google Research scientists Bill Byrne and Filip Radlinski describe as "the lack of quality training data that accurately reflects the way people express their needs and preferences to a digital assistant."

Despite ongoing efforts in many quarters, Byrne and Radlinski said in a blog post, "the conversations we might observe with today's digital assistants don't reach the level of dialog complexity we need to model human-level understanding."

"It is our hope that these datasets will be useful to the research community for experimentation and analysis in both dialog systems and conversational recommendation," they said.

The CCPE dialogs were generated in sessions involving live assistants asking questions "designed to minimize the bias in the terminology" used by the other person (the "user") to convey his or her preferences, and to obtain these preferences in natural language, the researchers said. Each dialog is annotated with "entity mentions, preferences expressed about entities, descriptions of entities provided, and other statements of entities."

"Among the insights into this dataset, we find that the ways in which people describe their preferences are amazingly rich" the researchers said. "This dataset is the first to characterize that richness at scale. We also find that preferences do not always match the way digital assistants, or for that matter recommendation sites, characterize options. To put it another way, the filters on your favorite movie Web site or service probably don't match the language you would use in describing the sorts of movies that you like when seeking a recommendation from a person."

The Taskmaster-1 datasets were generated using two methodologies. In addition to the WOZ method, a one-person, written technique was used "to increase the corpus size and speaker diversity."

"For written dialogs," the researchers explained, "we engaged people to create the full conversation themselves based on scenarios outlined for each task, thereby playing roles of both the user and assistant. So, while the spoken dialogs more closely reflect conversational language, written dialogs are both appropriately rich and complex, yet are cheaper and easier to collect."

This dataset also uses a simple annotation schema that provides "sufficient grounding for the data, while making it easy for workers to apply labels to the dialog consistently," they said.

A detailed description of Taskmaster-1 and links to a full description of the data, methodology, and analyses, as well as sample conversations and user instructions is available here. Additional details about the CCPE dataset are available here. A useful CCPE case study can be downloaded here.

Google doubled down on its AI strategy last year, when it rebranded its Google Research Group to Google AI.