Fairness, Accountability and Transparency in ISMIR 2019

A post compiling most of the resources that discussed these topics at ISMIR 2019

Posted by Vinod Subramanian on January 13, 2020 · 9 mins read

Introduction

The tagline for this year’s ISMIR was ‘Across the Bridge’ with the intention to actively improve the diversity of the community across cultural backgrounds, scientific disciplines and professional affiliations. Either as a result of this initiative or as a happy coincidence there were a lot of papers, workshops and keynotes that talked about the ethical aspects that we need to be aware of in the MIR community. A key step to this awareness and sensitivity of the impact of MIR research is to be more inclusive and diverse so that people from different backgrounds can provide insight on this important aspect of research. In this post I will cover the papers and other related events that addressed the impact of MIR research through the lens of Fairness, Accountability and Transparency.

Papers

Ethical Dimensions of Music Information Retrieval Technology

Andre Holpzafel, Bob L. Sturm, Mark Coeckelbergh

This is not a paper from ISMIR 2019. It was published in the accompanying journal TISMIR. The paper provides motivations for why ethical aspects of MIR is an important field of study through hypothetical examples which are very easy to understand and fun to read. It also provides a theoretical framework to study ethics in MIR research. I am including it in this summary because it provides a good framework to appreciate the rest of the work that I mention in this article.

Data Usage in MIR: History & Future Recommendations

Wenqin Chen, Jessica Keast, Jordan Moody, Corinne Moriarty, Felicia Villalobos, Virtue Winter, Xueqi Zhang, Xuanqi Lyu, Elizabeth Freeman, Jessie Wang, Sherry Cai, Katherine M. Kinnard

In the MIR community we are starting to see a widening gap in the access to data and computing power between companies and academic labs with limited resources. This paper presents solutions on how to make access to resources more democratic. The fact that they address the legal and logistical concerns of implementing these solutions is valuable because we can start working on this immediately.

Can We Listen To It Together? Factors Influencing Reception of Music Recommendations and Post-recommendation Behavior

Jin Ha Lee, Liz Pritchard, Chris Hubbles

How does a recommendation system interact with a person? When are we more likely to accept a recommendation and when are we likely to reject it? It appears that the social aspect of recommendation is very important. This work finds that recommendations from a friend, more explainable recommendations or co-listening increase the likelihood of listening to a recommendation.

From an ethical point of view I was struck by a quote of one of the participants of the study as to why they may not listen to a recommendation: Honestly, a lot of reasons I won’t listen to something is outside the sphere of music. How would a music streaming service know I don’t want those recs because the fanbase is full of white supremacists or because the singer is a sexual predator? My view is that it is unethical for a streaming service to recommend songs that have white supremacist fanbases or artists who are sexual predators. So how do we ensure that our recommendation systems do not exhibit these unethical qualities?

Automatic Music Transcription and Ethnomusicology: A User Study

Andre Holzapfel, Emmanouil Benetos

We typically use objective metrics to evaluate our transcription systems but do these objective metrics directly translate into usefulness in the real world? This work examines the usefulness of automatic music transcription for ethnomusicologists. Does using an automatic system decrease the time required for transcription, decrease errors etc. In the case of the Sousta corpus this work shows that an automatic transcription does not aid ethnomusicologists in a significant way.

It could be argued that given some training the musicologists would be able to benefit from the automatic. However it seems unclear as to whose responsibility it is to administer this training. Is it the job of the creators of the algorithm? Is it the job of the people who are trying to use the algorithm?

Towards Interpretable Polyphonic Transcription with Invertible Neural Networks

Rainer Kelz, Gerhard Widmer

The goal of this work is to better understand the way in which deep learning models make predictions. The paper adapts invertible neural networks for the task of polyphonic transcription and demonstrates a few simple experiments to show that a person can intuitively understand the relationship between the input and output. As an example they were able to show that the a7 note does not have a good mapping between the input and output because the dataset does not have a high number of a7 notes to train on.

Keynotes, Tutorials and Workshops

In addition to these papers there were plenty of other resources from ISMIR 2019 that delve into the realm of ethics.

WiMIR Workshop

I was not able to attend this event so I cannot provide a first hand account of the conversations that occured or the nature of the event. However, if you look at the webpage for WiMIR there were a lot of projects talking about issues of bias in audio embeddings, the ethics of music generation systems, the ethics of collecting data for music recommendation systems etc.

Tutorial: Fairness, Accountability and Transparency in Music Information Research

This was a very interactive tutorial where audience members were able to discuss their ideas on ethical issues in MIR. Emilia Gomez started off by motivating the reasons for caring about ethical issues in Artificial Intelligence. She presented the 7 principles for trustworthy AI and concluded with questions we need to ask ourselves about how MIR systems impact society. She was followed by Andre Holzapfel who presented the work done in the TISMIR paper mentioned earlier in this article. Marcus Marion continued the tutorial after the break offering some insights from the legal system and how Artificial Intelligence can be biased to the detriment of a marginalized group. As researchers we cannot hide behind the fact that the data is biased, we need to design our system to account for data biases so that these biases are not reflected in our systems. Bob Sturm started his part of the tutorial with a controversial automated hiring system that seems to use visual information along with a questionnaire in order to select the best candidates for a job. The system does not explain its decisions it just gives the recruiter the “relevant” candidates. The general theme of his section is to ask the question “How do you know your system is working?” Is the system actually learning something meaningful or has it memorized some patterns that exist in the dataset that give it the appearance of good performance?

Music and Algorithmic Responsibilities in Practice

Henriette Cramer

This was my favorite keynote at ISMIR this year. It explores music as a cultural phenomenon and how that can feedback into deciding who gets to play music and be heard. This obviously runs into ethical issues such as discrimination based on gender, geography and community. Unfortunately the video for this talk is not uploaded yet as of this blog post but I am linking to Henriette Cramer’s personal website where she has posted some of her other talks in a similar space.

Conclusion

I believe that music technology is about improving the experience of music for composers, musicians and listeners. This means that through our research we have the power to influence which kind of music adapts to the digital age better. We have a responsibility as researchers not to misuse this power and consciously work towards creating a fair space for people from all backgrounds to participate in.