Article by Koninklijke Nederlandse Akademie van Wetenschappen / the Dutch National Expertise Centre and Research Data Repository.
The MuseIT project aims to co-design, develop, and co-evaluate a multisensory, user-centered platform for enriched engagement with cultural assets. Our project partners work with a variety of modalities (sound, video, haptic, three dimensional) and various cultural heritage assets (architecture, music, visual media), as well as work on media creation through co-design, all of which result in large amounts of data and associated metadata.
Metadata is essentially ‘data about data’ - it can provide context to the different datasets and assets used throughout the project, and is incredibly important for collaborative efforts.
While all project partners deal with data to some extent, for KNAW-DANS, their work is all about the data. DANS is an institute and collection of data repositories based in The Netherlands and is part of the Dutch Academy of Sciences (KNAW). In addition to their project role in the research data management, DANS is also able to lend their expertise on Dataverse to the MuseIT project as a data referencing system that allows the different project partners and work packages to collaborate across domains. The Dataverse Project is an open source web application that allows users to share and preserve research data through self-sustained Dataverse “installations”. Within the MuseIT project, there are a few planned installations, the first of which is a collaborative effort between ShareMusic & X-Systems: http://database.sharemusic.se
So what is the benefit of maintaining your research data in a repository with detailed metadata? DANS R&D engineer Vyacheslav (Slava) Tykhonov is the force behind the MuseIT x Dataverse implementations, and below has laid out a way that MuseIT can harness the structure provided with Dataverse and large language models (LLMs) to help link modalities and provide enriched outputs. LLMs are a type of AI model that is trained on text-based datasets to respond to prompts, and can be useful for finding connections between concepts much faster than researchers alone.
From Slava: “Let’s imagine you have a photo or drawing of famous paintings. You’re registering them as a dataset in Dataverse and providing a detailed description of all details such as the author, date, style, location, form of modality, etc. You link each field to a well-known ontology, for example, linking the title and author to a metadata standard (such as Dublin Core), and then to the corresponding Wikidata entry.”
“To describe multimodalities, ShareMusic created its own metadata schema and linked every field to the corresponding entry from the Wikidata database. For example, the media modality property (P12548) is used to indicate how people can access the dataset (Listen; Feel; Experience). Since LLM models typically include Wikidata data in their training sets, and the description of the property is recognized and understood by the LLM, this provides an opportunity to use those property descriptions as ‘context’ together with field values. For example, it can help translate values into various languages or link them to existing controlled vocabularies.”
“Moreover, when you use an LLM to generate descriptions or translations of cultural heritage artifacts, the LLM can use the structured data from Wikidata (such as the painter's biography, the painting's history, etc.) to produce more accurate and informative outputs. These outputs can then be incorporated into a knowledge graph that connects different forms of data (text, images, etc.) about art.”
Various project partners from the MuseIT project recently delivered a presentation on this topic at the DARIAH Annual Event in Lisbon, Portugal (June 19, 2024). The talk, titled, ‘Guarding accessibility - AI supported ontology engineering in the context of the MuseIT repository design,’ discussed the new ways that the MuseIT project will use to describe accessibility facets of artefacts in the MuseIT repository. One of the key messages from this talk was the need for a human touch, so to speak. While LLMs and similar tools like generative AI could benefit research and society in general, double checking outputs for mistakes, false connections, and so-called ‘hallucinations’ are necessary. The abstract of the presentation and the slides are available via Zenodo: https://doi.org/10.5281/zenodo.12533424.
The next steps for this aspect of the project are to 1) build and fill more dataverse installations with other project partners and 2) bring in to MuseIT the metadata for Cultural Heritage artifacts from Europeana and linking them in the shared knowledge graph.
Comments