2nd Liminal Space at the ]a[ Research Day

On 12.11.2020 DAD’s Niko Wahl presented our intermediate results on the third research day of the Academy of Fine Arts, Vienna. Due to COVID-19, this event was an online ZOOM meeting. The goal of the ]a[ research days is to give an overview of all ongoing research projects at the academy including discussions with all participating colleagues.

Niko Wahl gave a short introduction of our project and an overview of DAD’ s collaborations with three different museums, where we work with an archive of an ethnological journal, a fine arts gallery and the statues in the Academy’s Glyptothek.

Since our work with the Austrian Museum of Folk Life and Folk Art and with the Belvedere, Vienna, has already been documented in previous blogposts [1][2], lets turn to the presentation of plaster casts at the Academy‘s Glyptothek, which we explored with Dusty, an off-the-shelf household robot.

Many people associate Artificial Intelligence (AI) with the development of ever more powerful and dextrous robots, along with horror scenarios of these machines taking over the planet. In reality robots are a small part of AI which is rather dominated by machine learning software solutions powering your Internet search engine, the natural language interface to your mobile phone, online music, movie and product recommendations and many other everyday technologies.

On the other hand, many people already own robots with limited forms of AI, for instance vacuum cleaning robots. What if we confront such a household robot with a – supposedly obsolete – museum collection of historic plaster copies of famous statues, whose very physis seems to be made of dust.

The robot takes its own route through the museum space. Following its built-in algorithms it perpetually finds new ways through the collection. It seemingly decides for itself in what order to visit the museum objects, all the time metaphorically internalizing the objects of art while inhaling their dust.

Other visitors are free to follow the robot on its path through the museum space engaging with its exhibition narrative. They might benefit form surprising relationships between objects of art established by the often creative course of the robot. Smart last generation vacuum cleaning robots are able to share their sensory experiences with others of their kind. These shared experiences usually are measurements of objects and how to avoid them when traversing a room. But what if this cloud communication, usually not accessible to us, deals with objects of art instead of everyday items? Will meeting David or the Pieta change the robots’ discourse? What if the robot meets a portrait of itself?


4th Critical Space at the Belvedere Research Center

On the 22nd of September 2020 the DAD team met with Christian Huemer and Johanna Aufreiter from the Belvedere Research Center to discuss our results concerning Belvedere’s online collection. One focus of the meeting was our engagement with the room on “Viennese Portraiture in the Biedermeier Period” in Belvedere’s permanent exhibition.

Applying our algorithm to find pathways of semantic meaning [Flexer 2020] between works of art, we are able to suggest additional works for the liminal spaces between individiual positions in the curatorial narrative, opening up new sub-narratives for the room. Based on a word embedding [Mikolov et al 2013] of the keywords associated with the paintings, our algorithm suggests works of art which follow a pathway between the respective semantic meanings. Moreover we are able to further constrain our liminal curation by requiring all art works to fit an additional overall topic chosen by a human curator, again translated to the language of Belvedere’s keyword system via word embedding. As an example see a “Gender” constraint applied to the Biedermeier room.

A conceivable outcome is a revision of the Biedermeier room achieved via a joint curation of human and machine. This, as well as other approaches towards the Belvedere collection, will be the center of further exchange between DAD and the Belvedere.

All depicted paintings in this blog post by Belvedere, Vienna, Austria (CC BY-SA 4.0).

Activities Spaces

1st Liminal Space at the Machine Learning for Media Discovery Workshop

DAD’s Arthur Flexer presented our work on discovering semantic pathways through Belvedere’s fine arts collection at the “Machine Learning for Media Discovery Workshop” (18th of July 2020) of the “International Conference on Machine Learning”. The conference was supposed to happen in Vienna, Austria, but due to COVID-19 went fully virtual. You can see Arthur present his poster in a dedicated Zoom room below.

You can read about the results in our previous blog post, read the respective scientific paper and look at the poster.

While a virtual workshop is not able to replace the experience and liveliness of a physical scientific meeting, it still allowed us to get an increasing degree of public exposure for our work in progress, which is the purpose of our Liminal Spaces.

Citation: Flexer A.: Discovering X Degrees of Keyword Separation in a Fine Arts Collection, in Proceedings of the 37th International Conference on Machine Learning, Machine Learning for Media Discovery Workshop, Vienna, Austria, PMLR 108, 2020.


3rd Critical Space on discovering semantic pathways through a fine arts collection

DAD´s Arthur Flexer gave a semi-virtual lecture on “Discovering X Degrees of Keyword Separation in a Fine Arts Collection” at the Austrian Research Institute for Artificial Intelligence (OFAI, 24.6.2020). The presented work is inspired by the project ‘X Degrees of Separation‘ by ‘Google Arts and Culture’, which explores the “hidden paths through culture” by analyzing visual features of artworks to find pathways between any two artifacts through a chain of artworks. In his work, Arthur Flexer is more interested in finding pathways of the semantic meaning of works of art rather than just their visual features. Therefore he used word embedding [Mikolov et al 2013], which encodes semantic similarities between words by modelling the context to their neighboring words in a large training text corpus. This is used to embed keywords of Belvedere´s online fine arts collection and obtain pathways through the resulting semantic space.

Keywords from left to right: [‘Resurrection’, ‘Christ’], [‘Christ’], [‘Death’, ‘Skeleton’], [‘Vulture’], [‘Angel’, ‘Air’, ‘Martyrdom’, ‘Suffering’, ‘Failure’, ‘Death’, ‘Andreas’, ‘Multiple Layer Room’]. All images by Belvedere, Vienna, Austria (CC BY-SA 4.0).

The above exemplary result starts with a sculpture with keywords ‘Resurrection’ and ‘Christ’ where the painting in the end position has keywords around the topic of ‘Death’ and ‘Martyrdom’. The second artwork in the pathway is a relief showing ‘Christ’, while the third is a painting tagged with ‘Death’ and ‘Skeleton’, hence already semantically closer to the topics of ‘Martyrdom’, ‘Suffering’ and ‘Death’ of the end artwork. In fourth position is an etching with the only keyword ‘Vulture’, which is semantically close to ‘Angel’, ‘Air’ and ‘Death’ of the ending artwork.

In the ensuing discussion of results it was found remarkable how machine learning via word embedding replicates existing biases and prejudice in the society. In the above query with the word “Homosexuality” the most similar word out of 22 million terms in the word embedding model is “Paedophilia”, one of the worst prejudice against homosexual people. The word embedding model has been trained on the Wikipedia and Common Crawl corpus [Mikolov et al 2018], which helps explaining the replication of very common and persisting prejudice in our society.

OFAI´s Brigitte Krenn found it interesting how the very reglemented and almost scientific language in Belvedere’s keywords (stemming from the Iconclass project) is contrasted with everyday language via usage of word embedding. As can be seen above, the most similar keywords to “Homosexuality” are “Rape”, “Religion”, “Violence” and “Islam” (all translated from German). This is of course a direct result of the biases inherent to the word embedding model. DAD’s Alexander Martos called this phenomenon “re-socialising of arts via natural language processing” or rather “re-a-socialising” since it uncovers asocial societal tendencies and (re-?) introduces them to the world of fine arts.


“GO CURATOR” mock-up presented during 2nd Critical Space

During our 2nd Critical Space with guest expert Bob Sturm (KTH Royal Institute of Technology, Stockholm, Sweden) we presented and discussed a first mock-up of our “GO CURATOR” idea.

Here you can see DAD team members setting up the physical model.

GO CURATOR analyses text describing paintings in museum collections. Topic modelling is used to represent the semantic content in these texts, thereby targeting the semantic meaning of the paintings themselves.

Result is a probabilistic distribution across topics for every painting. E.g. in the painting below, the topics “food”, “act”, “human” and “object” are present to an equal extent of 25%.

Curators or museum visitors can change the exhibition interactively by adjusting which topics should be present to what extent. GO CURATOR then automatically adjusts the choice of paintings and their exact hanging in the museum room.


2nd Critical Space with Bob Sturm on frontiers of artificial creativity

During the research visit of Bob Sturm we will discuss the frontiers of artificial creativity and its criticism in the context of DUST AND DATA. Bob Sturm will also give a public lecture about his work on using machine learning to compose Irish folk music. His talk will also feature live accordion playing.

“Folk the Algorithms” – Bob Sturm, KTH Royal Institute of Technology, Stockholm, Sweden

In this talk/musical performance, I will recount how a bit of Saturday morning humor turned into an ERC Consolidator Grant four years later. It’s a story of an engineer with an artistic bent meeting a machine learning algorithm through a blog. One part of the story involves the naive misappropriation of music data without consideration of its provenance and significance. Another part involves the serious contemplation of such transgressions, and then endeavors taken to redress them. A variety of interesting perspectives and questions have arisen out of this story, which will be subject to study in the project, Music at the Frontiers of Artificial Creativity and Criticism (MUSAiC, ERC-2019-COG No. 864189).

Time: Wednesday, 26th of February 2020, 6:30 p.m. sharp

Location: Oesterreichisches Forschungsinstitut fuer Artificial Intelligence, OFAI
Freyung 6, Stiege 6, Tuer 7, 1010 Wien


1st Critical Space @OFAI

DAD’s first “Critical Space” took place at OFAI on 16th of December 2019. The full day event included the complete DAD team and Fritz von Sunderhaar.