3rd Critical Space on discovering semantic pathways through a fine arts collection

DAD´s Arthur Flexer gave a semi-virtual lecture on “Discovering X Degrees of Keyword Separation in a Fine Arts Collection” at the Austrian Research Institute for Artificial Intelligence (OFAI, 24.6.2020). The presented work is inspired by the project ‘X Degrees of Separation‘ by ‘Google Arts and Culture’, which explores the “hidden paths through culture” by analyzing visual features of artworks to find pathways between any two artifacts through a chain of artworks. In his work, Arthur Flexer is more interested in finding pathways of the semantic meaning of works of art rather than just their visual features. Therefore he used word embedding [Mikolov et al 2013], which encodes semantic similarities between words by modelling the context to their neighboring words in a large training text corpus. This is used to embed keywords of Belvedere´s online fine arts collection and obtain pathways through the resulting semantic space.

Keywords from left to right: [‘Resurrection’, ‘Christ’], [‘Christ’], [‘Death’, ‘Skeleton’], [‘Vulture’], [‘Angel’, ‘Air’, ‘Martyrdom’, ‘Suffering’, ‘Failure’, ‘Death’, ‘Andreas’, ‘Multiple Layer Room’]. All images by Belvedere, Vienna, Austria (CC BY-SA 4.0).

The above exemplary result starts with a sculpture with keywords ‘Resurrection’ and ‘Christ’ where the painting in the end position has keywords around the topic of ‘Death’ and ‘Martyrdom’. The second artwork in the pathway is a relief showing ‘Christ’, while the third is a painting tagged with ‘Death’ and ‘Skeleton’, hence already semantically closer to the topics of ‘Martyrdom’, ‘Suffering’ and ‘Death’ of the end artwork. In fourth position is an etching with the only keyword ‘Vulture’, which is semantically close to ‘Angel’, ‘Air’ and ‘Death’ of the ending artwork.

In the ensuing discussion of results it was found remarkable how machine learning via word embedding replicates existing biases and prejudice in the society. In the above query with the word “Homosexuality” the most similar word out of 22 million terms in the word embedding model is “Paedophilia”, one of the worst prejudice against homosexual people. The word embedding model has been trained on the Wikipedia and Common Crawl corpus [Mikolov et al 2018], which helps explaining the replication of very common and persisting prejudice in our society.

OFAI´s Brigitte Krenn found it interesting how the very reglemented and almost scientific language in Belvedere’s keywords (stemming from the Iconclass project) is contrasted with everyday language via usage of word embedding. As can be seen above, the most similar keywords to “Homosexuality” are “Rape”, “Religion”, “Violence” and “Islam” (all translated from German). This is of course a direct result of the biases inherent to the word embedding model. DAD’s Alexander Martos called this phenomenon “re-socialising of arts via natural language processing” or rather “re-a-socialising” since it uncovers asocial societal tendencies and (re-?) introduces them to the world of fine arts.


“GO CURATOR” mock-up presented during 2nd Critical Space

During our 2nd Critical Space with guest expert Bob Sturm (KTH Royal Institute of Technology, Stockholm, Sweden) we presented and discussed a first mock-up of our “GO CURATOR” idea.

Here you can see DAD team members setting up the physical model.

GO CURATOR analyses text describing paintings in museum collections. Topic modelling is used to represent the semantic content in these texts, thereby targeting the semantic meaning of the paintings themselves.

Result is a probabilistic distribution across topics for every painting. E.g. in the painting below, the topics “food”, “act”, “human” and “object” are present to an equal extent of 25%.

Curators or museum visitors can change the exhibition interactively by adjusting which topics should be present to what extent. GO CURATOR then automatically adjusts the choice of paintings and their exact hanging in the museum room.


2nd Critical Space with Bob Sturm on frontiers of artificial creativity

During the research visit of Bob Sturm we will discuss the frontiers of artificial creativity and its criticism in the context of DUST AND DATA. Bob Sturm will also give a public lecture about his work on using machine learning to compose Irish folk music. His talk will also feature live accordion playing.

“Folk the Algorithms” – Bob Sturm, KTH Royal Institute of Technology, Stockholm, Sweden

In this talk/musical performance, I will recount how a bit of Saturday morning humor turned into an ERC Consolidator Grant four years later. It’s a story of an engineer with an artistic bent meeting a machine learning algorithm through a blog. One part of the story involves the naive misappropriation of music data without consideration of its provenance and significance. Another part involves the serious contemplation of such transgressions, and then endeavors taken to redress them. A variety of interesting perspectives and questions have arisen out of this story, which will be subject to study in the project, Music at the Frontiers of Artificial Creativity and Criticism (MUSAiC, ERC-2019-COG No. 864189).

Time: Wednesday, 26th of February 2020, 6:30 p.m. sharp

Location: Oesterreichisches Forschungsinstitut fuer Artificial Intelligence, OFAI
Freyung 6, Stiege 6, Tuer 7, 1010 Wien


1st Critical Space @OFAI

DAD’s first “Critical Space” took place at OFAI on 16th of December 2019. The full day event included the complete DAD team and Fritz von Sunderhaar.