How could social media systems be designed as safe places that really work for the people who use them? What can art help us understand about machine learning data sets? Caroline Sinders of Convocation Design joins us this week to talk about her research-based art practice that’s trying to change perspectives about what exactly is going wrong on the Internet, and just how exciting it may be to fix it.
Caroline has worked at a designer and researcher in many tech and art world contexts. In this episode she talks about her recent article in Slate and her projects Feminist Data Set and TRK . Caroline also mentions Thomas Thwaites’s The Toaster Project.
Transcript
Ethan Zuckerman:
Hey, everybody. Welcome back to Reimagining the Internet. I’m Ethan Zuckerman. I’m here with Caroline Sinders, who’s a machine learning design researcher and artist. She’s the founder of Convocation Design and Research, which is an agency that focuses on designing for the public good and solving really difficult communication problems. In that context, Caroline’s done work for IBM Watson, Amnesty International, Wikimedia. She’s been a researcher and a fellow at half a dozen wonderful places. And that’s because she’s one of the world’s leading experts on online harassment. She’s a super deep thinker about data and feminism. And she’s someone that I really like to check in on around these questions of how social media systems could be made to actually work for humans who live in the world. Caroline, how are you?
Caroline Sinders:
I’m good. Thank you so much for having me.
Ethan Zuckerman:
I know the swamp behind you is somewhere from Louisiana. Where are you right now? Where are we reaching you?
Caroline Sinders:
Currently I’m in London, but virtually it looks like I’m in the Honey Island Swamp between Louisiana and Mississippi right off I-10.
Ethan Zuckerman:
So, I would love to jump in to some recent work that you’ve done with Vandinika Shukla. And this came out on a piece in slate recently about online harassment, particularly as it affects journalists, and particularly female journalists, journalists of color. This came in some ways out of some really terrible decisions that Slack made recently. Can you take us back to sort of Slack’s design choices?
Caroline Sinders:
For sure. So this piece came out of a residency I had with the Harvard Kennedy School called The Digital HKS Program. We were funded by The Open Technology Fund and we actually also worked with at Elise Vogel. And part of what we were interested in looking at is how design can play into safety and online harassment inside of platforms. And Slack is one that is just a space that just gets a lot of criticism and doesn’t necessarily like to respond or adapt to it. And within that, one of the things we started looking at were what were different safety mechanisms, like block buttons, for example. Slack does not have that. And we ended up doing the research report before this new feature Slack had rolled out and we ended up writing our slate piece around the time Slack rolled out this feature where pretty much anyone could message each other on Slack.
Caroline Sinders:
I think they called it ‘global messaging’. You could apparently opt out of it, but you would still see a message from someone, like in your email. So there’s still a way to sort of see that message. And it really sort of highlighted the lack of forethought or safety in terms of users. And it often makes me wonder, “Well, who were they thinking of when they designed this?” Slack tends to classically say that they are a work tool, that they are designing for teams. And on one hand, I want to hold them to that and say, “Yes, I understand you’re designing in a professional context, but you’re kind of refusing to acknowledge that Slack is used and personal settings, if you will.” And by ‘personal’ I mean, it’s used by alumni groups, which is not in an office or it’s used by groups of friends who want to talk with each other, or it’s used by activists to organize different projects.
Caroline Sinders:
It’s used for conferences to coordinate people discussing. So it’s used in like all of these really sort of blurry areas. It’s not just the office. And so I bring up the office because whenever we’ve pushed back to Slack, say, “Why don’t you have a black button?” They were like,”Oh, we feel like this cuts down professional communication in an office setting.” And so this global DM tool they had, I’m sure they thought in their minds, “This is going to be great for a company that’s like the size of IBM or Google, that’s above like 10,000 employees. And you just need to find someone, but maybe they’re not in your right team.” But that design in that context is one thing. But then when you apply it to, “Oh, so anyone, in any Slack, could be messaged.” And all of a sudden you can see how this could very quickly be weaponized, and be used to harm.
Ethan Zuckerman:
In some ways what’s so problematic about Slack is not that they made this category mistake, right? So their category mistake is they assume that they are workplace tool. As we know, harassment, particularly sexual harassment, never ever happens in the workplace. So you’d never need to deal with something like harassment in those contexts. But even if we give them that category error and we sort of say, “Slack, you think you’re a work tool, but you’re not purely a work tool. And therefore you have to think about harassment.” Why are technology companies so bad at anticipating harassing uses of their tools? This just happens again, and again, and again. It’s almost like harassers are endlessly creative, but rather than credit the harassers, in some ways, I just want to sort of blame the tool builders. Why are we so bad at anticipating misuses of our tools?
Caroline Sinders:
I feel like a recent book that’s come out, Silicon Values, by Jillian C. York, covers this also. Part of it, I think one could argue, is that a lot of designers and a lot of tech companies could still design from a very north American US focused positive space. And we see this really embedded in modern design culture, particularly in product design and for-profit product design, that we only create like positive use cases in our mind of these tools. We don’t necessarily take a security or threat modeling approach to design. And I’m thinking super broadly, but this is sort of what I’ve noticed, is like a fly on the wall and a lot of different corporate settings that I’ve been able to observe or lead conversations at or work in. And that, I think, is part of it.
Caroline Sinders:
And I think another part of it is… One of the things I wonder is are there enough experts inside of design teams that can also sort of speak to the experiences of facing online harassment? A lot of these teams are very white. They’re very north American. I mean, there’s so many things I’m trying to run through what all of them are. Some of it is, is there enough research or are they like regularly engaging with users? And I think in some cases they are, it’s just that hasn’t built up enough fluency inside of all the workers in these companies, right? So I’m a design researcher. I’m often working in the Internet Freedom and Human Rights Space. So for me, that’s the fluency I have, is I’m only coming at it from people that have been harmed.
Caroline Sinders:
So I can be a real bummer if you will, inside of certain design scenarios, when it comes to ideating, what’s a new thing to build? Because I’m always like “That’s going to harm someone. That’s going to do this. That’s a bad idea. Don’t do that.” And that’s kind of the opposite of how Americans, especially Silicon Valley designers kind of operate. We’re always operating from the ideal use case. And I’m operating from, “What could possibly go wrong? Let’s bring in threat modeling and security procedures sort of into this to affect design for us to design better.”
Caroline Sinders:
So I think part of the big issue is design culture and how we build things. I think another issue is we aren’t prioritizing human rights research and human rights education across all disciplines. I think another thing is we aren’t, or designers or design teams aren’t necessarily educating themselves on use cases, even going beyond personas of saying, “This is a use case, we need to be actively threat modeling against these use cases.” For example, requiring people to read Design Justice, and Silicon Values, and Mistrust, for example, would be helpful in an education for people onboarding or working in these companies. They also, I think, need more training, but more folding in actual threat modeling into their daily practices. I think that’s like the first layer.
Ethan Zuckerman:
I’m so glad you brought Jillian York. We’ve actually recorded an interview with her about Silicon Values that’s going to come out on the feed probably before our piece with you will come out. And the book is just so rich, but one of the things that I love about it is that it talks about this sort of value set that is affecting these tools. I feel like if we sat down to design new tools with the idea of “How do we make sure that we harm as few people as possible?” rather than “How do we get this up to a billion users as fast as we possibly can?” But I feel like everything in the economic system and the innovation system sort of mitigates against that. How are you as an artist, and a practitioner, and a writer, and a designer, instead of all the different things you are, how are you trying to get across that message that we need to think about harm as well as think about growth?
Caroline Sinders:
That’s where I really try to reach designers. I try to teach classes in Silicon Values. So I taught one on Ethics and Human Rights and how it affects design. A lot of the writing I do or research I do is directly on how things like online harassment manifests into a design layer. I had just published a policy paper, actually a few days ago, with the German Marshall Fund on dark patterns and design culture. So one of the things I think that’s really important to keep in mind in these conversations is it’s all of these intersecting problems, and design culture is just one of them. And I think in that space, that’s where I’m trying to sort of use my expertise as someone who’s trained as a designer, who’s worked in these areas, but also recognizing that product design and design like UX and UI is the layer or the thing that takes code and policy.
Caroline Sinders:
It makes it understandable and interactable for a user. So, for example, your security settings, you don’t have to be a security engineer to understand how to use them. You don’t have to open up a command line when even reporting something like privacy. You don’t have to be a lawyer to try to understand or decipher how to report something. And that being something that doesn’t mean that those two examples are great or the best design that they could be. But rather, design is taking something that is policy heavy, like the divisions of online harassment, like how a platform will try to understand that content and seeking the tech, where that content is going to go, and it’s turning it into an understandable interface and flow for a user to report something. And so, thinking about the politics of design in terms of these spaces is really important because design is what can obfuscate, or elevate these issues.
Caroline Sinders:
So, for me, that’s kind of the space I try to really occupy in. So, when I hear a policy recommendation, one of the things I want to try to do is translate that into something for a designer. So that can be an op-ed, like the space of advocacy, like my Digital HKS Report. That was really me trying to replicate studies I thought were happening inside of social networks. How are social media companies trying to understand online harassment against journalists? Let’s try to replicate one of those studies that will then manifest into design recommendations.
Caroline Sinders:
In other groups, after we published related things, which is so important, like Penn-America and the World Wide Web Foundation. And there really is a moment, I think, right here, for what I’m thinking of, is the seatbelt of online safety. What are things that we could start to standardize more? And again, a seatbelt is something that was designed,, was an open source and was then able to be sort of implemented across all cars. So how can we start to think of the security settings and security features in a similar way inside of software? [crosstalk 00:12:53]
Ethan Zuckerman:
The seatbelt’s a wonderful analogy, right? Because while we think of the seatbelt is this wonderful move towards safety culture, it was also sort of introduced as a way to fend off a much more expensive safety events, which was the airbag, which had also been invented, but was going to be so much more expensive to implement. So, design really is where the rubber meets the road in quite a literal sense, even including things like those questions of constraint. What can we afford to do? What can we not afford to do? I’m super fascinated where some of your recent research has gone. I would say, in some ways, sort of deeper into the heart of the machine, then the design layer, with things like your Feminist Dataset Project, where I understand that project as looking at gender biases throughout sort the lifestyle of a machine learning system. And you make this great analogy and introducing it to this project focused on toasters. Can you tell us about toasters and then tell us about The Feminist Dataset?
Caroline Sinders:
Yeah. So, pedagogically, The Feminist Dataset is inspired by The Critical Design Project, by Thomas Thwaites, it’s called The Toaster Project. So Thwaites builds a commercial toaster from scratch, from digging up iron ore out of his backyard to smelting it, to creating his own plastic. And at the very end, he creates this technically working toaster. Should you use it to cook? Don’t think that maybe you should. But it was… It’s really kind of misshapen and beautifully. And it is like a comic kind of on this sort of ubiquitous thing, a toaster that you can go into a corner shop and buy for 15 or $20. There are so many of them, and he was really interested in the concept of how do you take that ubiquitous hardware and really unpack it. And so I’m very much inspired by that process, but with software.
Caroline Sinders:
So how do you carefully consider and remake every angle and part of the machine learning pipeline from start to finish, but with intersectional feminist values? So it’s not just data collection, it’s not just the kind of data, it’s how do you train it? How do you clean it? How do you label it? What’s the act within that that would be feminist? What’s the platform you’re using? How would you create a feminist audit? And then also, what is the kind of algorithm we should use? And part of the point of the project is also trying to engage with failure in a way. My project will not be truly intersectionally feminist. I’m a cis white lady, for example. But also, it’s like, can software ever truly be feminist? Is kind of a question. And I’m interested in the trade-offs people make. And so for me, I’m the experiment, what’s the trade off I’m going to make? And I’m going to have to justify that.
Ethan Zuckerman:
I mean, it’s interesting, as we talk about machine learning, as we talk about AI, everyone wants to talk about harm. Is harm the right way to think about this? Do we think about women being mislabeled or misconstrued or stereotyped? I mean, should we talk about what are the harms that come out of a machine learning system that is not a feminist system, or is there a better paradigm than harm when we start interrogating things like machine learning systems?
Caroline Sinders:
I mean, I think harm is a good one. It’s also abroad. And so I think, whose definition of harm, who’s being harmed always has to be asked, right? Because if you were to give all the best intentions to a first-year design student, who is, let’s say, white from a small town, and you’re like, “Make sure it doesn’t harm.” They’re going to have a very, very different perspective than per se, like [inaudible 00:16:57] thinking of harm, right? So I think this is where having standards or examples is really helpful. And so the project itself, from this dataset, is intersectional, but it’s also very much trying to recognize that harm comes from all of these different angles. I think often when we talk about machine learning, we do focus on the data that goes in, the sort of connections, or inferences, or predictions, or simulations, and algorithms making any output.
Caroline Sinders:
And I think for me, I’m interested in those things, but I also think harm can manifest in so many more ways. And, for example, even if you’re making something equitably or rather if the output is equitable, what if the process of building it was not equitable? What if that’s the problem? And so then using the framework of intersectional feminism, I actually have to address that because at a certain point, I’m going to have to hire someone to help me, or hire a series of people, or have a series of people help me do the cleaning or the training. So I have to actually think through, well, what is an equitable work environment to engage with people and that would then be feminist? And I think that that’s also really important, because all the different steps and choices we make, all have trade-offs.
Caroline Sinders:
And if we’re just only focusing on the outputs of artificial intelligence and not the actual structure of building and making, and not all of these other invisible areas that feed into it, like, “Can we ever truly fix it?” Right? Or can we ever truly try to course correct or try to improve it? That’s why I think it’s been really important to see all, for the past few years, so many conversations around Mechanical Turk, for example, and recognizing that that is a part of this sort of invisible nature of machine learning. I think that there are other things to also sort of think about within that, but that’s what the project is really trying to unpack, is it’s like if I were building the toaster, every small screw I’m using, I really have to think about, “Where did this come from?” “Who made it?” “How did I get it?” “Where did I get it from?” And I’m doing that effectively with software.
Ethan Zuckerman:
So I’d love to drill in a little bit on that Mechanical Turk question, right? So, one of the big things that comes into play with machine learning systems is that what they’re really doing is they’re extrapolating from training data. So to use an example from the wonderful television show, Silicon Valley, we can look at a set of images and say, “Is this a hot dog?”, “Is this not a hot dog?” One of the more popular ways to do the ‘is a hot dog is not a hot dog’, is to hire people on Mechanical Turk and ask them to code images as ‘hot dog’, ‘not hot dog’. You dug into the ethics instead of the implications of this in a project for Mozilla. Can I get you to talk about that piece?
Caroline Sinders:
Yeah, sure. That’s TRK, technically responsible knowledge. I designed and researched that and then built it with [inaudible 00:20:05]. And then the graphic design from Rainbow Unicorn for a fellowship I had with The Mozilla Foundation. And I was really interested in what was like an accessible browser-based tool people could use in lieu of Mechanical Turk. I couldn’t find enough data on something like CrowdFlower to actually ensure that it had feminist values. During this project I became a Mechanical Turk myself, just to get a sense of what the interfaces felt like, what the experience was. And within this project, we started thinking a lot about just money and cost. And so this project I should highlight, it does a few things and it does a few things. One because it is a that I made for a fellowship.
Caroline Sinders:
And then two, this is one of the trade-offs that will have to be documented for The Eventual Feminist Dataset, a book that I will write at some point, which is… There are just things we can’t do for this. There’s an edge where the art project is, and a startup would begin, for example. And that’s just sort of the reality of the kind of internet we live on. So, Feminist Dataset, TRK is a part of Feminist Dataset, and it is an open source browser-based tool. Anyone can use the code. So if any of your students wants to use it or run an experiment, it’s on GitHub, please use it. You can upload datasets to be labeled and cleaned, or you can also be the one cleaning them. And if you’re uploading it for people, you have to actually provide a lot of context around it, like what you’re using or what the images are, why.
Caroline Sinders:
And almost like a warning label or a trigger warning. So people are notified as to what it is. And then, if you are sending up to sort of clean it, you’re given like a preview of in-text only of what the data sets as it is. And then you’re invited to reject it or accept it once you get into the interface. And then, it just asks for some information about you. And then that’s injected plain textile into the dataset, inspired by data sheets for datasets. And that’s like a part of research work I was doing a little bit around the same time was thinking of why don’t we have data ingredients? Why don’t we have standardized labels? That’s something that consumer reports is also working on now. I think it’s really important to make these things readable and legible. Then, when you first get on the website, the first thing you see actually is the wage calculator.
Caroline Sinders:
And that was one of the experiments I did as a fellow was thinking about, well, can I… If I first assumed like good intention for people who are hiring for microservice tasks, why are things under price? Beyond the regular reasons of service fees, etc. And one of the things we sort of realized really doing a lot of design research into the layout of something like mechanical Turk and CrowdFlower is that it’s not actually aggregating what an entire person’s day would be. Meaning it’s not actually saying, “Okay, you’re doing a hundred tasks. How long are these tasks?” It’s only going like task and price, but time is a really big important variable in terms of if something is priced equitably or not. For example, I told you, “Oh, Ethan, I need help with something. I can pay you 20 Euro.” It’s a really great deal if it’s five minutes. It’s a really bad deal if it’s two hours or [crosstalk 00:23:33]
Ethan Zuckerman:
And if I have to fly and join you in the first place. It’s probably an offer, but I turn down.
Caroline Sinders:
Right. So not including time. And the length of a task is actually really problematic. And so we wanted to create a visualizing interface to say like, “Okay, even if you’re pricing an image at like 10 cents per image, how long does it take?”, “How many things do people have to actually fill out?” And then we started timing ourselves to do stuff at this time. Also, when we were playing with the time element, the New York Times did an article and they even created a little experiments. We started using that to see how long it really takes to do these things. And then, we started doing all this math around what is a living wage? And I never thought I’d be doing this much math and an art project which… My mom is an accountant, so she laughs at this. But we wanted to create a provocation.
Caroline Sinders:
And this is where we were also realizing there’s some things we can’t do. So we decided to focus on the two biggest groups in Mechanical Turk. And they come from the United States and India. And then we decided to focus on the US because there’s just a higher cost of living there. And then we started looking at what is the state with the highest minimum wage? And ironically it’s Washington state, where Amazon is headquartered. So we’re like, “Great. We’re an art project. That’s pure poetry. Let’s try to figure out the living wage of someone in Seattle if they were a Mechanical Turker.” And so we started to look up what were recommendations around a living wage in Seattle. And we found that a few different newspapers were recommending actually something more like $16.
Caroline Sinders:
And then we started to work backwards of, “What does an equitable day look like for someone?” So, for example, you only are paid for what you’re working, right? So if you take a bathroom break, you’re not paid for that time. So we started… I took a best practice for working remotely, which was a 55 minute, like hour counts for a full hour. So we’re like, “Okay, we need a minus five minutes from every hour, right, to then start to figure out how many worked hours is someone working to cover all this pay.” We also took, I think, a 45-minute lunch break, and then another break. We looked at like how often McDonald’s allows people for breaks. And we brought McDonald’s often because when people describe gig work, they describe it as being better than flipping burgers, except McDonald’s gives you paid breaks.
Caroline Sinders:
And so we then realized that the actual workday that we needed to calculate for was six and a half hours, as opposed to eight or nine hours. So then, that means that the work hour has to be over $19 an hour to accommodate for this paid time off because people need paid breaks. So then we knew that that’s what we were aiming per an hour. So then we then tallied up the whole day, someone used to work a minimum of… Or the whole day needs to account to like a minimum of over like a hundred something dollars. And so then we realized, “Okay, the minimum a task is, is going to be a certain amount of sense.” But then we had to figure out if you will, how long is that task?
Caroline Sinders:
And so then this is where we were measuring ourselves. And so we decided on about 20 seconds per task, and each task has to be above 11 cents. So that would allow for someone to work a full day. If they get really tired, they will take longer. And so we came to this conclusion sort of by also practicing, like on me, I would try and see how many tasks can I do in an hour for this to sort of work?
Caroline Sinders:
And it’s still a pretty hard day if you’re working that much, but effectively that’s the minimum someone needs to be paid. But that’s not taking into account things like taxes, but that’s how we sort of got to this visualizer to be like, “Well, are you actually pricing equitably?” And again, this is where design is really important. The design of all these platforms, the interface really works against you if you are trading price equitably, because you can’t see how long does it take, how’s time affect this? And that was something we realized was just super important to take into account that people should be able to make a living wage. And even if you have them labeling one image, that’s one thing out of their day, right? And it has to contribute towards this.
Ethan Zuckerman:
Kind of wrapping up some of the themes we’ve been talking about here, how would you go about building a novel social media platform, consistent with feminist values? How do you sort of take these questions about automated systems, these questions about designing for the bad use cases, as well as the good ones? Speculate with us a little bit on sort of designing that sort of a system.
Caroline Sinders:
Great that you asked that, because that’s like one of the new projects I’m also starting, which is, Can you build a feminist social network and what would it look like? And I think one of the things I start with, again, a scale. I think a lot of the problems are scale. And so the provocation I give students when I lecture on this or I host workshops is instead of designing an encyclopedia, what if we’re designing a paper zine? What if something is allowed to organically decay? What if it’s not designed to last forever? Then how does that change our relationship to software? , for example, or how has that changed our relationship to permanence? How has that changed our relationship to communities? That doesn’t mean design things that don’t work. And that doesn’t mean design things that are not safe, but instead of imagining that everything we create is going to be the next Facebook, why does it have to be that way?
Caroline Sinders:
Why can’t it be something that exists for a moment in time that is quite beautiful or interesting. And then from there, the question starts to become, “Okay what is safety in this way, in this place?” “And how do you determine safety?” And I do think scale very much plays into this. I think it’s designing from the very beginning or thinking through features, people would want, how people would want to talk to each other, then how do you ensure or foster that kind of engagement, and how do you, again, offer different kinds of safety protocols? But then the big thing is, often, if you’re starting something… I recently did this as a workshop in France. The thing I would tell, I think I would tell students is like, “Well, you’re also responsible for people’s safety. If it’s your software and they’re coming something you’ve built, you are actually responsible for people.”
Caroline Sinders:
So you need to ensure that not only do you have a code of conduct that, you know how to implement a code of conduct online. Do you have a place for people to report? Do you have a system for looking through reporting? Have you thought of a ladder of consequences? Which is just another way to say, “What’s your policy for when someone does X versus Y? What are you going to do? How are you going to tell them? How have you determined that?” And you can just see like the students’ heads to sort of start to explode a little bit. And all these things are thoughtful and intentional. They’re things you have to do. They cannot be after thoughts. So if you’re designing a space for people, even if it’s for five or 10 people that you know, at some point conflict is going to happen or harassment, and conflict is super normal.
Caroline Sinders:
Another thing I tell people is community doesn’t mean friendship. Conflict is really normal. People are going to argue over really dumb things. I argue with my husband every day around, “What do we want for breakfast?” Totally normal, not a lot of conflict, right? Those kinds of frictions are normal inside of people being people and interacting with each other.
Caroline Sinders:
And so you need to have a space that is flexible enough to allow for different kinds of frictions, and also strong enough to be able to respond to different kinds of harassment. And then you need to understand all of those differences. So what is the difference between people having natural friction to growing into conflict, to then the conflict turning into something more like abuse to sort of pun [inaudible 00:31:47], very iconic book. But it’s really important to sort of think about all of those. And so I think for me, if I were to design a feminist social network, I would try to mindfully grow with the size of the community, but also say like, “This is going to have to cut off at a certain point. Maybe we shouldn’t have more than a certain size, but you can make your own.”
Ethan Zuckerman:
So Caroline, the next time we get together, the conversation I want to have is speculative design of the world in which it is as easy to build these sort of small carefully thought about social networks as it is to try to find funding for the next social network for 3 billion people. Carolyn Sinders, it’s been just such a pleasure. Thank you so much for making some time for us. We will be watching and waiting for the new book and for whatever sorts of good mischief you’re up to in the meantime.
Caroline Sinders:
Thank you.