News

Researchers Unveil Mosaic Machine Learning Algorithm to Find Similar Images

The Mosaic system can retrieve similar images based on several comparison dimensions. For example, given an image of a modern American glass sculpture, the Mosaic system can find similar based on culture (Austrian, Chinese, German, etc.) or based on media (painting, ceramic, textile, etc.)

• By Pure AI Editors
• 08/19/2020

Researchers from Microsoft and MIT have developed a new system called Mosaic for finding similar images. Given a source image, standard techniques that find similar images rely solely on some form of pixel similarity.

The Mosaic system can retrieve similar images based on several comparison dimensions. For example, given an image of a modern American glass sculpture, the Mosaic system can find similar based on culture (Austrian, Chinese, German, etc.) or based on media (painting, ceramic, textile, etc.) See Figure 1.

Data scientists, developers and others can experiment with the Mosaic system themselves. The demo allows users to select a query image from a library of artwork and then find matching images by culture or by media. See Figure 2.

Mark Hamilton, one of the members of Mosaic team from Microsoft, commented, "Every time I use the algorithm, I find surprises. The first surprise of the project was the answer to a question that we couldn't imagine. In particular, we asked it the outlandish question: 'What is the closest musical instrument to this Double Face Banyan?' "

The Microsoft and MIT collaborators have published a research paper that explains the technical details of the Mosaic algorithm titled, "Conditional Image Retrieval," by M. Hamilton, S. Fu, W. Freeman, and M. Lu. The paper is available in PDF format at https://arxiv.org/abs/2007.07177.

Hamilton noted, "Going forward, we hope this work inspires others to think about how tools from information retrieval can help other fields like the arts, humanities, social science, and medicine."