Understanding Racial and Gender Bias in AI and How to Avoid It in Your Designs and Design Education

How biases are present in our design processes and design tools

Sarah Pagliaccio
Adjunct Professor
Lesley University
College of Art and Design
Brandeis University

We are using artificial intelligence-enabled software every day when we post to social media, take pictures, and ask our phones for directions. But these apps are not designed to serve everyone equally. Institutional bias is present in the tools that we hold in our hands and place on our kitchen counters. Recent research on and disclosures by the tech giants have revealed that voice and facial-recognition apps are optimized for white, male voices and faces, respectively. The data used to “teach” the algorithms learned from the historical training data that women belong in the home and black men belong in jail. All of this leads to bias where we least want to see it—in courtrooms, in classrooms1, in elections, in our social-media feeds, in our digital assistants, and in our design tools. (Meanwhile, women drive 75-95% of purchasing decisions in the US and we are rapidly becoming a majority-minority nation.)

In this presentation, we will review some of the most egregious recent examples of AI-driven racism and sexism and take a look at some less well-known examples, including the changes to Twitter’s algorithm that favored white faces over black faces; the MSN robot editor that confused the faces of mixed-race celebrities; AI assistants that screen job applications and immigration applications; voice-recognition apps in cars that don’t understand women drivers, voice-activated apps that assist disabled veterans; and predictive-text software that could exacerbate hate speech. We will explore how these biases are present in our design processes and design tools, specifically those that use speech, image, and name generators.

Finally, we will review options for confronting these biases—like taking the implicit bias test, knowing the flaws in underlying data sources we rely on, expanding our user research to include diverse audiences, and using text sentiment analysis to remove our own bias from interview scripts, among other options—so we do not perpetuate gender and racial bias in our design solutions and design education.

Notes and References

  1. A group of 68 white, elementary-school teachers listened to and rated a group of white and black students for personality, quality of response to a prompt, and current and future academic abilities. The white teachers uniformly rated white students higher than black students; black, English-speaking students; and students with low physical attractiveness. The researchers concluded that some of these children’s academic failures might be based on their race and dialect rather than their actual performance. (Indicating that no one is immune from cultural biases, the duo who performed this research labeled the nonblack, English dialect that the white students spoke Standard English rather than White English.) See DeMeis, Debra Kanai, and Ralph R. Turner, “Effects of Students’ Race, Physical Attractiveness, and Dialect on Teachers’ Evaluations.” Contemporary Educational Psychology, Vol. 3, No. 1, January 1978.

This research was presented at the Design Incubation Colloquium 7.3: Florida Atlantic University on Saturday, April 10, 2021.

What Can Machine Learning Contribute to Empathy in Design? How to Build a Journey Map Using Big Data and Text Sentiment Analysis

Sarah Pagliaccio
Principal, User Experience Designer
Black Pepper

What can machine learning contribute to empathy in design? How to build a journey map using big data and text sentiment analysis.

Art and design are meant to reflect the world around us, show empathy for those we design for, and reflect the emotional state of our customers and target users. But how are we meant to empathize with situations that are unfamiliar or out of context? What happens when we over-empathize and project our own emotional states on our customers’ experiences? That’s where machine learning comes in. With enough input, we can use machine learning tools, specifically text sentiment analysis, to provide an objective score of our users’ emotional experiences. By feeding transcripts of customer interviews into a computer, we can remove our own subjectivity from our analysis and form a holistic picture of others’ needs and wants.

These sentiment scores can turn words into pictures, emotions into graphs, expanding our understanding of design goals and tasks.

Using Shakespeare’s A Midsummer Night’s Dream as a case study, we will talk through the emotional journey, i.e., the customer journey map, of major characters in the play using text sentiment analysis. A discussion of how these techniques can be applied to consumer application and website design will follow.

This research was presented at the Design Incubation Colloquium 5.1: DePaul University on October 27, 2018.

Guided Experiential Learning for Design Innovators

C.J. Yeh
Professor, Assistant Chair
Graphic Design

Fashion Institute of Technology

FIT is one of the pioneers in creative technology and design education. For this presentation, the founder of the Creative Technology program at FIT, C.J. Yeh, will introduce the most innovative design projects from FIT’s creative technology courses.

FIT’s Creative Technology curriculum has strong focuses on augment and virtual reality, user experience design, design thinking, and digital thinking. Digital thinking is probably the primary difference between Creative Technology and the other programs at FIT. For Creative Technology program at FIT, technology is more than just a tool, it is an arena in which the students learn to explore new possibilities in digital media and conceptualize new experiences and digital product innovations that has never been done before.

One of the most unique pedagogy from FIT’s Creative Technology and Design Program is called “Guided Experiential Learning.” It is a unique merger between the traditional studio classes and internship. Through its Guided Experiential Learning initiatives, FIT faculty and students have worked with major brands and international research institutions like the National Football League (NFL), Infor, and Fabrica–a highly regarded research center in Italy. For each Guided Experiential Learning project, FIT’s faculty design customized workshops, lectures, and training to maximize the learning for students, and, at the same time, ensure the collaborating brands/organizations receive the highest quality design products at the end of the process.

This presentation will share case studies, best practices, and insights on how Guided Experiential Learning has been adopted in higher education. Relevant pedagogies and teaching methodologies will be introduced, and a discussion regarding the challenges and opportunities particularly in its application and relevance to college-level design education.

This research was presented at the Design Incubation Colloquium 4.2: CAA 2018 Conference Los Angeles on February 24, 2018.