Austrian newspaper “Der Standard” published an article about DAD (in German), focusing on our problems with algorithmic bias and our critical stance towards digital curation in general.
DAD’s Arthur Flexer presented the paper “Computational filling of curatorial gaps in a fine arts exhibition” at the International Conference on Computational Creativity. The conference was supposed to be held in Mexico City, but due to Covid was shifted to a virtual enviroment resembling the ancient town of Tenochtitlan, which preceded what is now the historic center of Mexico City.
Flexer A.: Computational filling of curatorial gaps in a fine arts exhibition, in Proceedings of the Twelfth International Conference on Computational Creativity (ICCC ’21), pp. 2-5, 2021.
The art blogger eSeL published a photo story about the opening night of our project presentation.
Our final PRESENTATION SPACE called „Dust and Data. Artificial Intelligence im Museum“ will start on Tuesday, the 8th of June 2021.
The opening of this 3 month exhibition can be attended online at the 8th of June, 19.00 in a ZOOM lifestream (in German only). Please register at the Volkskundemuseum Wien to get the link. You can also join in person from 19.30-21.00 directly at the museum (Laudongasse 15–19, 1080 Vienna).
We are very much looking forward to presenting our results and ongoing work on the art of curating in the age of Artificial Intelligence. We hope to see many of you at the opening or during the run-time of the exhibition.
Our exhibition at the Volkskundemuseum Wien has been announced at their website: “DUST AND DATA, Artificial Intelligence im Museum” (Mi, 09.06.2021 – So, 29.08.2021). This will be a final point for DAD but at the same time a starting point for further endeavors based on our achievements and works-in-progress presented in this exhibition.
Die Museen sind jetzt alle digital. Auch museale Inhalte sind vielfach digital geworden – als Kopien und Scans der Originale oder auch als originär digitale Objekte. Die meisten Sammlungen sind durch ihre digitale Erschließung heute viel zugänglicher, gleichzeitig müssen Kurator*innen und Ausstellungsbesucher*innen mit einer Flut von Inhalten inner- und außerhalb der Ausstellungen umgehen. Auch die Kommunikation zwischen Museen und dem nunmehr vernetzten Publikum wird weitestgehend zu einem digitalen Unterfangen – nicht nur in Zeiten der Pandemie.
Dust and Data ist ein künstlerisches Forschungsprojekt, dass sich nicht mit der Digitalisierung an sich, sondern mit den darauf aufbauenden Möglichkeiten, Chancen und Gefahren der Artificial Intelligence beschäftigt. Anhand einzelner Museen und Sammlungen beschreiben wir neue Möglichkeiten des Ausstellens und des Betrachtens. AI getriebene Maschinen, die nicht nur die Bedeutung von Kunstwerken und Sammlungsobjekten verstehen lernen, sondern mitunter auch selbst kreativ werden sind dabei unsere Begleiter und Guides. Neue Zusammenhänge, neue Analyseergebnisse, neue Narrative entstehen dabei im Rahmen eines Zusammentreffens von Kurator*innen, Publikum und Maschinen. Wir zeigen dabei auch eine gänzliche neue Art des musealen Arbeitens und unsere (teils offenen) Fragen an die Beteiligung der neuen Technologien.
Das Volkskundemuseum Wien wird bei diesem Projekt zur Gastgeberin für neue Einblicke in die Sammlungen und Ausstellungsräume der Glyptothek, der Akademie der bildenden Künste, des Belvederes und thematisiert dabei auch die eigene Sammlung.
Kuratierung: Niko Wahl, Arthur Flexer, Irina Koerdt, Alexander Martos, Sanja Utech
Di, 8.6.2021, 19.00 Uhr
Our ongoing cooperation with the Belvedere now has its own entry at their website. See our previous blog posts concerning results for this DAD case study [1,2,3,4], as well as the following publication:
Flexer A.: Discovering X Degrees of Keyword Separation in a Fine Arts Collection, in Proceedings of the 37th International Conference on Machine Learning, Machine Learning for Media Discovery Workshop, Vienna, Austria, PMLR 108, 2020.
DAD´s Arthur Flexer gave a virtual lecture on our plans for the DAD exhibition opening early summer at the Austrian Museum of Folk Life and Folk Art. The lecture was given at the Austrian Research Institute for Artificial Intelligence (OFAI), where DAD was also located during the first nine month. This last CRITICAL SPACE was conducted to get feedback about our plans to build exhibits documenting our engagement with the collections of the Glyptothek of the Academy of Fine Arts Vienna, the Volkskundemuseum Wien and the Belvedere. Results for these three case studies differ according to the level of interaction between curators and machine: from using natural language processing tools for research in museum databases to a symbiotic interaction between curators and algorithms to robots visiting museums autonomously.
The ensuing conversation centered around ways to include and document aspects of algorithmic bias and societal stereotypes existing in natural language models. We also discussed our approaches to turn digital findings into analog exhibits and how using a robot to explore the Glyptothek aligns with the public’s (mis)conception that AI is predominantly about building machines not software.
Our initial intention was to review and audit the current development of all our approaches after 18 months of the project‘s time span in a (semi-public) symposium-esque format. Due to COVID-19 this CONCLUSIVE SPACE was a somewhat smaller affair in the form of a one day workshop (29.1.2021) including the whole DAD team with all COVID precautions taken.
We planned the remaining half year of the project which will be used for a final documentation of the assembled approaches, of the models, code and curatorial developments. This CONSTRUCTIVE SPACE will also finalize hands-on displays and physical proof of concepts for use in the final PRESENTATION SPACE.
This PRESENTATION SPACE will be a comprehensive exhibition documenting and presenting all achievements and works-in-progress at the Austrian Museum of Folk Life and Folk Art. Please watch this space for a making-of and behind-the-scenes documentation of this process, as well as announcement of the exhibition which will open beginning of summer.
DAD’s Arthur Flexer presented our work on analysing the semantic meaning of works of art at the International online conference The Art Museum in the Digital Age of the Belvedere Research Center. This conference is concerned with the digital transformation of art museums, which seems even more relevant lately because of COVID-19 related lockdowns and closures.
Arthur presented our (somewhat radical) approach to analyse text about artworks rather than the usual route of analysing images of the artworks. We chose this semantic driven approach because a lot of information about an artwork cannot be found in the artwork itself. Think e.g. of subjecting the “Mona Lisa” to an automatic visual analysis. Computational results will tell you that it is a picture of a young woman, in front of a landscape, and (if your algorithm is really good) is sort of smiling. This information of course totally misses the significance of the painting for (Western) art history, its immense relevance and the many connotations it has. All of this rather is a societal construct and result of centuries of discourse and reception history (for more on this see our previous blogpost). Our semantic driven approach  towards the collection of the Belvedere enables us to discover X degrees of keyword separation between works of art.
This is achieved by using the technique of word embedding [Mikolov et al 2013], which encodes semantic similarities between words by modelling the context to their neighboring words in a large training text corpus. This was used to embed keywords of Belvedere´s online fine arts collection and obtain pathways through the resulting semantic space.
The above result starts with a painting having keywords ’Clouds’, ’Mountain’, ’Meadow’ from which we transit to ’Mountain’, ’Lake’, ’Alps’ and ’Austria’, next to a painting tagged ’Fog’, then one with ’Rocky coast’ and finally with ’Clouds’, ’Rocky Coast’, ’Sea’. Our pathway therefore smoothly transits from a mountain setting to a lake in the mountains to the sea.
We also presented one very concrete solution for a room in Belvedere’s permanent exhibition. It is a room about “Viennese portraiture in the Biedermeier period”, assembling the “greatest portrait painters” from this period. In the above picture you can see four blue frames which indicate empty slots which we like to fill using our algorithm with the respective neighboring artworks as input.
The keywords for these neighboring artworks however are purely descriptive, e.g. ‘headgear’, ‘necklace’, ‘bonnet’, ‘eye contact’, probably not doing the semantic content of the artworks full justice. We believe that one underlying topic of the Biedermeier room is ‘gender’, with all but one painting depicting females. We therefore add an additional algorithmic constraint by requiring all suggested artworks to respects both the requirement of being part of a pathway and having a ‘gender’ related keyword. Since ‘gender’ is not a keyword in the Belvedere taxonomy, we use word embedding to obtain Belvedere keywords with high similarity to the topic of ‘gender’. This translation step yields keywords like: ‘femaleness’, ‘religion’, ‘islam’, ‘equality’, ‘motherhood’ or ‘headscarf’. It is obvious that these keywords point to a stereotypical discourse of gender, quickly derailing towards topics of religion and a compulsion to wear headscarfs or women being predominantly seen in their role as mothers.
This is also why we termed the use of word embedding in this context world embedding: it confronts the very rigid taxonomy of the Belvedere keywords (based on Iconclass, a classification system for cultural content) with everyday language as represented in the textual training data of the word embedding. It thereby recontextualizes or even “resocializes” taxonomic art histories via natural language processing since it uncovers biases and prejudice in our use of language and (re-?) introduces them to the world of fine arts.
The above picture shows three paintings from the Biedermeier room plus four additional paintings (with red frames) which our algorithm suggests. The second painting from the left is suggested because its keyword ‘femaleness’ is a gender keyword and its keyword ‘necklace’ makes it similar to the keywords of the first painting (‘earrings’, ‘pearl necklace’) and the one in the middle (‘brooch’, ‘bracelet’). The 5th painting from the left is suggested because ‘headscarf’ is a gender keyword and ‘eye contact’ and ‘earring’ make it similar to both the painting in the middle (‘brooch’, ‘bracelet’, ‘eye contact’) and the painting on the far right (‘eye contact’, ‘bonnet’).
In the ensuing discussion with the conference’s audience Arthur Flexer advocated that our semantic apprach is more helpful for building a curatorial narrative than a purely aesthetic procedure. It allows to answer the question about curatorial gaps between artworks shown in an exhibition. What works of art exist in the holdings of the museum that fit the curatorial narrative but did not succeed in becoming part of the exhibition?
He also tried to make clear that by using such a machine learning tool like word embedding, curating becomes a joint endeavor of man and machine, where curatorial decisions have to be formulated as input and constraints to the algorithm. But even a simple curatorial Google search already is an interaction of man and machine, with algorithms (oblique to the curator) nevertheless to a certain extent shaping their curatorial enterprise by showing specific selections of information only. It was also discussed that such a man/machine approach is able to uncover algorithmic biases in the methods used, as e.g. stereotypical representations of societal discourse in word embedding.
Looking towards future extensions of our work it can be said that of course we could analyse longer (art historic) texts about artworks with the same methodology thereby gaining much richer semantic context then by relying on simple keywords only. Another possible extension is to embed semantic and visual information simultaneously which could yield curatorial solutions that respect semantic and viusal constraints at the same time [Frome et al 2013].