Trigger/content warning: child sexual abuse materials, sexual exploitation of children, and trauma stemming from sexual abuse.
Brian Levine has a storied career as a computer scientist working in cybersecurity. Earler, pivotal work in privacy has given way to his current all-hands-on-deck fight against the spread of CSAM (child sexual abuse material) online.
Ethan and Brian sat down in person for a long talk about what it takes to keep children safe online, what corporations should be doing differently, and why there are no easy answers when you bring in the values that are so often associated with Interent progressivism such as free speech and the spread of small communities.
This is a heavy episode, but an important one. We truly appreciate you joining us to engage in Brian’s critical work.
Brian Levine is the founding director of the Cybersecurity Institute and runs the Rescue Lab here at UMass Amherst. He runs the App Danger project, which was covered last summer in the New York Times.
Transcript
Ethan Zuckerman:
Hey everybody, welcome back to Reimagining the Internet. I’m Ethan Zuckerman. This is one of our rare live recordings. Producer extraordinaire Mike Sugarman is here with me. I’m here with my friend and cherished colleague Brian Levine, say hi Brian.
Brian Levine:
Hi everyone. Great to be here.
Ethan Zuckerman:
I’m gonna introduce Brian in just a second. We are in a somewhat stuffy conference room on a rainy day at UMass Amherst, but we’re taking this time to feature I think some of the most important work being done around a critically important topic, which is child sexual abuse materials and child exploitation online.
It turns out that Brian is one of the leaders in this battle doing truly extraordinary work. It turns out that Brian is one of the leaders in this battle doing truly extraordinary work.
We’re gonna take a chance to learn about a topic that most of us know a little bit about but not a ton about so background here. Brian Levine is director of the Cybersecurity Institute. He’s a professor in Computer science here at UMass Amherst leading the Digital Forensics Laboratory. He’s got a long and distinguished career in computer networking and network security and then in recent years he’s pivoted to topics around child exploitation, child sexual abuse material. Really a focus on crimes against children in digital spaces in that context. He works with the Internet Crimes Against Children Task Force. He ends up doing some work with the FBI and this work gets a lot of attention actually recently featured in the New York Times for some new work that he’s doing.
Before I actually give Brian a chance to say something, let me sort of say disclaimer on this. This is not going to be a happy, shiny episode of Reimagining the Internet. The reason we’re doing this is that a lot of the future visions of the internet that are more decentralized and democratized need to take seriously these issues around child exploitation.
Those issues around child exploitation very quickly turn into other issues around privacy, and around who has control over platforms. I will also disclose Brian and I sometimes disagree on this stuff, and they tend to be really helpful profitable disagreements at least for me. But you may hear the two of us spar a little bit on some of this and on some of the players in the field. Let me just sort of say this is one of the smartest people working on one of the hardest topics out there. Brian welcome. It’s good to have you.
Brian Levine:
It’s again great to be here. Thank you for the warm introduction and I’m looking forward to trying to shed light for each other on these difficult issues that have no clear answers often and I think that’s why we sometimes disagree.
Ethan Zuckerman:
So let’s start with this question of how you got here. Before you started working on questions of child exploitation, you had an incredibly successful career writing papers around computer security. You’ve written about some wonderful difficult wonky topics. But honestly when I started to get to know you about five years ago you had really gone all in on trying to figure out how to combat CSAM child sexual abuse material online and broader patterns of child exploitation online. Tell me about that pivot. Is it as dramatic as it looks to me? What led you there? How did you have to make the shift in your career?
Brian Levine:
So that’s a great question. So when I got out of grad school it was 1999 with my PhD, and for those of you who were around then that was the one of the first peaks of the Internet you know over saturating our society. That was what was called then the dotcom boom and I think that generation of let’s just say grad students and researchers we saw society transformed very quickly. And there was this aspect of we could do anything and I think machine learning is going through a little bit of that now. Anything seems possible and I think we I think society at least some of us see since the internet changed things so dramatically in ways that were unexpected I think there’s a little bit of that going on with ML and its revolutions, but from my perspective when I started be started at UMass as a professor one of the first things that I got into was privacy online.
At the time Obama was not yet president and people could not get medical insurance easily. Sometimes pre-existing conditions would make that hard and let’s see to make a very long story shorter. I used to give a talk where I’d say hey everyone. I’ve got these academic solutions for privacy online and it’s going to hide your IP address, which is maybe we’ll get into this but if you’re going to do an investigation of someone’s horrific activities online as law enforcement, you’re typically going to start with their IP address. So it’s definitely a keystone for the whole internet architecture in terms of investigations and I think everyone understood that back then. So I used to give a talk and I’d say imagine you know unfortunately there was some medical illness that you were worried about getting maybe someone in your family had it and you went to you know, I used to say health.com something generic I probably that that URL exists, but you know, they’re not involved so you you go to some website you you learn about this this medical disease, maybe it’s genetic and you’re a bit worried. So you think I probably should go get health insurance and so you go to the next URL healthinsurance.com whatever it is and little do you know that the same websites are owned by both and they’re going to deny you insurance based on your IP?
Ethan Zuckerman:
Or maybe they’re not even owned by both, but someone’s got a tracker on one of them. The fact that you paid attention to certain bits of content is being sold because there’s very little regulation around online advertising, so you were trying to preserve privacy.
Brian Levine:
Exactly, exactly so I used to give that talk and I would present academic solutions to this kind of problem, and so one day I organized a workshop of academics and and someone said and what one okay, and so one of the areas that I worked in at the time was file sharing networks, which are all peer-to-peer. So I think a lot of your audience might be familiar with BitTorrent and there’s been other versions of that that have been around here and there. And the reason that relates is a lot of the solutions for getting privacy online when that means hiding your IP address are also peer-to-peer and are also no one’s in control of people talking to one another. And you can gain some privacy that way.
So I had organized a talk and then someone said to me it was either during the talk or after the talk and it was someone I knew who worked for a large corporation and they said well, you know, “everyone knows that all the peer-to-peer networks are used to trade CSAM” what we now call CSAM at the time, he said child pornography and I said what what are you talking about and and this was like my naive introduction to when no one is in charge some people will exploit that to their advantage. Now again the reasons I did this were to give people some privacy. So I arrived finally at an ethical dilemma that I don’t think enough people have arrived at.
Ethan Zuckerman:
And let’s just outline that a little bit because I’ve gone down some of the same paths as you have. One of the tools that people use for this is the Tor browser which uses three intermediate relays to make it very very difficult for anyone to figure out an IP address that people are merging from. One of my least cited but best loved works is something that I wrote in the early 2000s called “Anonymous Blogging with WordPress and Tor.” My goal there was helping someone in a repressive nation right online without revealing their IP address. I’ve ended up working with Tor over the years. But yeah, Tor absolutely is a way in which people are disguising their identity and certainly could be used. I think you’re gonna tell me it is used extensively.
Brian Levine:
Yeah, yeah CSAM, what I’ll call CSAM for the rest of the podcast.
Ethan Zuckerman:
Let’s just take a moment and talk about why CSAM because not everyone’s familiar with the term. If you want to sound knowledgeable about this field people don’t use the term child pornography. They use CSAM. Why is that?
Brian Levine:
So the reason they use the term well, let me answer. Why do they use both terms? So to this day the federal statutes that forbid you from possessing or distributing child sexual abuse material or CSAM. They’re still labeled as child pornography. So that term won’t go away until they change the federal law. My understanding is there is an effort to rename things in federal law, but people as you say in the know don’t use that term to distinguish it from adult pornography because child pornography is not pornography. It’s not consensual, if you know to call it actually my least favorite term ever is kiddie porn slightly better than that is child porn and then you know child pornography at least it refers to the federal statute.
But I think it’s important for people to distinguish the fact that every one of these images has captured a moment of abuse for a child, and not only was that awful but the redistribution of that is a harm that lingers into adulthood. One of the things I like to say about that is imagine that when you were a child these images were captured of you, and you grow into adulthood. You know literally every room you enter you wonder if anyone in the room has seen these images. If you have a child that looks like you and that’s often the case for people, you wonder if when you bring your child around in public if others think your child is the one in those images, and so it’s a lifelong trauma. And that has very little to do with adult pornography.
Ethan Zuckerman:
Pornography is a form of expression. Adults can choose to create and disseminate pornography. You can have lots of conversations about whether it’s a good choice or a bad choice, but it’s a choice. Child sexual abuse material is the documentation of an active sexual abuse. And so yeah.
Brian Levine:
It’s a different world.
Ethan Zuckerman:
So you found yourself starting to pay attention to this world based on an interest in ways of essentially disguising IP addresses, ways of maintaining privacy, and your interest in online privacy wasn’t a prurient interest. It was an interest in letting people maintain their privacy, seek medical information and so on and so forth. You found yourself on the horns of an ethical dilemma and you’ve come down quite strongly on one side of it. I have heard you say at times that certain aspects of privacy may end up having to be sacrificed in order to combat child exploitation.
Brian Levine:
Well the way I would phrase it is I still work in privacy. I work in children’s privacy. You know another anecdote I like to say is people care a lot about medical privacy. We care a lot if our records were you know if my doctor’s office was hacked into and someone got all my health records. Gee that would be awful. What if they got a video of the exam of my last doctor visit? Well, of course, that’s all much better than a child exploitation scenario so I think this is a privacy issue.
I think it is a you know, unfortunately you know some people call it a trolley problem. It’s the case sometimes that you can’t let someone have more privacy at the expense of somebody else. And so what I look for is situations where we can help children’s privacy while not removing any sort of rights that adults have or the privacy that they may have. I would say actually that as a society we’ve done that many many many times. And I can point to tangible examples.
I have children. When I drive them around in my car I put them in booster seats. I put them in before that I put them in car seats that are appropriate for children because the cars were not made for them. And it’s a very small chance that they will get into this accident, but if they do it’s horrific and I like that solution because like as a driver, as an adult I don’t have to be in the car seat in that way.
And I think when so much of what we do is online, I think there’s many situations where tech companies have an opportunity to apply, you know the equivalent of car seats to minors who they know are minors and not apply those same restrictions to adults and we’d be better off.
Ethan Zuckerman:
So let’s start with a fairly easy version of this dilemma. So let’s start with maybe a fairly easy version of this dilemma. And I think we’re gonna work our way to a harder version of it. You have a project that’s receiving a lot of attention and praise right now called App Danger. It’s a very simple project. It’s a list of apps that may well be problematic as places where people are soliciting sending receiving child sexual abuse material. When I glanced at this morning, it’s 181 apps listed either on the Google or the Apple stores. How are you compiling this list and what does it mean for someone to be on this list?
Brian Levine:
Okay, so Apple and Google run app stores. They are the only two app stores in existence as far as I know if you own a phone. And most kids own a phone. I mean if you think about it, Apple or I should say Android phones and Apple I mean those are in a lot of ways the largest toy makers in the world. I don’t know if you know any tween that haven’t asked their parents for a phone. And you can look up real statistics on how ubiquitous phones are. So kids are getting software from these places.
As a parent, I am commonly asked by my kids can I install this app? And so where do you go from there? And so you can I, you know, I recommend people to google the app and they might end up using common-sense media. Which is to my knowledge a manual process of looking at an app and there’s some crowdsourced reviews and sometimes for the major apps they will sit down and use the app and get some information from it.
But I think another great place to get information from an app is the app stores themselves. They post reviews from users, my understanding is that those reviews are you know in the case of Apple you can’t post a review without an iCloud account. So Apple has some understanding of who the reviewers are because they’re linked to an Apple device. They’re linked probably to a credit card. Anyway, Apple and Google believe that the reviews are legitimate enough to keep on their own sites about their own app stores.
So what we did is we collected the reviews that indicated some form of abuse and the reason it was important to collect those reviews is if you go to the app stores and you pick an app and then you sort first of all, you’re gonna see an age rating and I hope we come back to that because I think that process is broken. If you go to an app and you sort by I don’t know reviews that indicate this app is not helpful. You’re gonna see one-star reviews that say things like the GUI could be better. I pressed a button and it didn’t work. I lost 20 bucks on this app and things like that.
What we found is that there are reviews that are one-stars that say I was abused on this app as a minor or I was offered child sexual abuse material, we saw reviews that say um, you know, this app isn’t great, I was offered CSAM but you know, I met a good friend here, three stars, and that’s a paraphrase.
So we found it difficult with the Apple and Google app stores to find reviews that indicate abuse. We scraped those apps as best we could because there’s hardly an interface to do that, but they allow it, and then we with the help of some manual review by humans and a little bit of help from a natural language processing or NLP-trained classifier, we were able to pick out reviews that seem to indicate to us and again these were on Apple and Google stores that this app is dangerous in some way. We also found a lot of reviews that indicated there was you know abuse between adults, but we kept that off the app danger project to keep the message clear.
Ethan Zuckerman:
So at the moment the number one listed app on the app danger project is called Whisper correct? I know that this is an app that is a particular interest for you. The app danger project listed is having 400 or more reviews indicating that it might be a dangerous app. Why is Google still listing this app? What has Whisper done? Have they come to you? Have they been in touch with you? And have you found evidence of CSAM being traded on this app?
Brian Levine:
So I know that Google knows about the app danger project because the New York Times asked them about it. I have not talked to Google about the project. The app is listed as age rating teen which I think means 13 above, it might mean 12 and above for Google. If you get on Whisper, I don’t think it will take you long to run through the scrolling of.
So first of all Whisper is an app where you are encouraged to post confessional types almost like tweets back when we had an app that was called Twitter. So what one or two sentence posts where you say things like, you know, I’m thinking about doing this salacious thing, maybe you were gonna embezzle funds from the company you’re with. Maybe you’re looking to have an affair despite the fact that you’re married. Things like that are…
What we’ve noticed and I’m happy to give anyone who’s interested many examples of this because they’re all public. People are advertising that CSAM is available without a wink and a nod. It’s very easy to find and you can look at app danger and people report this and so we got on the app, and scraped and found the same thing. And I don’t you know, I would also say two things.
Okay, so two things, first is that the app is breaking the app store guidelines by allowing or having a moderation process where people can advertise CSAM. Apparently, you know that they easily thwart their moderation process. The second thing to say is there’s a lot of non-illegal whisper posts where people say things like hey, I’m an adult I would like to meet up with another adult and have sex, and way more graphic than what I’m saying here.
This appears to violate Google’s age rating and we have thousands of examples of that. This is from memory, but you know in a study we’ve not yet published about one in five, our estimates are from sampling and with the help of a classifier and some manual review, about one in one in five Whisper posts violate the age rating age rating that Google has given it and many fewer are advertising CSAM but all in all I don’t understand why the app has that rating from Google.
Ethan Zuckerman:
And Brian these are public posts, right? So anyone can go in and see these the fact that you’re able to review Whisper means that these are people publicly saying I am an adult and I am interested in having sex on an app that is rated for teens. Maybe this is appropriate for adults. It probably isn’t appropriate for teen content, and as you’re saying there are people going and saying oh, and I have child sexual abuse material.
Brian Levine:
Correct as well
Ethan Zuckerman:
And so are Google and Apple failing their responsibilities here?
Brian Levine:
Oh, I think so. I mean, I think if, I think if you are advertising to consumers that something is appropriate for ages you know at the teen rating, and you provide guidelines that state what that means and then it’s regularly violated by the content of the app. Then my question is what are they doing about it?
There’s a way of doing this a lot of people are getting behind, it’s called “safe by design” or “safety by design.”
The key, one of the key maxims of that effort is to ask you know, is to evaluate different apps or platforms and so on by what happens on the app. You know the impact the app has in the end rather than the intentions. So I think you know, let’s give them the benefit of the doubt. I’m sure Whisper does not desire this content on there, but it is and so I think there should be a valid, you know, the age rating that Google gives Whisper should be based on the content that is published and not on the desires or intentions of the Whisper.
Ethan Zuckerman:
So let’s talk briefly about social networks. I’m gonna use a phrase that’s gonna sound controversial: responsible social networks. How social networks actually handle CSAM. Facebook routinely reports that it takes down more CSAM than sort of any other players in the space. How does Facebook do that and what’s the sort of the best practice for? Social networks in handling CSAM?
Brian Levine:
Well, so I’ve never worked at Facebook and so this is just based on what I’ve heard third hand. It is calm and I’ll just speak generally rather than speaking confidently about what Facebook does. So the easiest way to recognize CSAM, easiest from a computational point of view, is to look at what the computer scientists who are listening know as a cryptographic hash. For everyone else I’ll just say there’s a unique digital fingerprint that every file has, and if you take a look at the content that’s being shared on your platform you can computationally determine that digital signature and then compare it against a list of signatures that you have. And that’s a really important way of detecting CSAM because you don’t have to have the original CSAM in your possession to do that. And it’s illegal to possess CSAM. So it’s critical that platforms have a way of looking at content that’s been submitted and find the match.
So I think it’s really interesting actually to compare Facebook and Apple when it comes to detecting things so Facebook you may say to yourself. Oh my gosh, like they’re detecting millions of instances of a CSAM apparently being traded on their networks. How do I, what am I quoting when I say millions? They report these instances to the National Center for Missing and Exploited Children. It’s called a cyber tip. You can look up that they regularly do 20 million or so reports a year because they’re doing that checking. So you may say, like wow that sounds like a really unsafe place, and I might say well actually the fact that they’re checking certainly makes it a more safe place than if they didn’t check.
I would also say there’s a group that I think your audience should take a look at called the Heat Initiative, and they are bringing attention to Apple’s practices. I encourage everyone on their own to go to the National Center for Missing and Exploited Children’s website. Most people say that the title is NCMEC and I would be surprised if their website is not NCMEC.org. Google for their cyber tips and you’ll see that you know in a recent year again, just from memory, Apple’s cyber tips numbered in the low two hundredths, and Facebook’s was in the, you know 20 million territory. So people upload things to say iCloud all the time and what the Heat Initiative has is trying to bring attention to is that Apple does not scan those files to see whether they’re known CSAM.
I don’t know why Apple doesn’t do that to my knowledge. They don’t do that. You know, I don’t know why they allow people to use their cloud networks without this safety check. All the other places, all the other companies out there to my knowledge that allow you to store content on their servers in the cloud typically will do this kind of scan. I don’t understand it.
They are, you know, a very large company, and so we could get into you know, I think what people would answer me who are listening perhaps would say, well as soon as you allow Apple to scan content now you’ve propped open the door for them to scan for other content. And I would say to you that’s your opinion like you know, that’s the position Apple has taken perhaps. But I think that children deserve for Apple to take that extra step.
Ethan Zuckerman:
Sure but let’s play that out because I don’t I don’t think it’s a farcical argument. It may be something that I disagree with but there are also databases out there of reported terrorist material and that’s actually a much more complicated and controversial definition. And it’s very hard to review those sorts of ratings, like there’s no one of the tricks with all of this is that if you have a hash database.
Part of the point of a cryptographic hash is that you can’t recreate the image from it, so it’s very hard for me to go and audit a hash database unless I have a set of test images, right? So if I take my role of fall foliage pictures in the Berkshires and I throw them against the database and they’re coming back a CSAM. There’s something wrong with it but it’s actually very hard to go in and sort of challenging. No, this video isn’t a terrorist video. This is actually Hamas. You know someone making a pro Hamas argument and probably is legitimate political speech. Apple may be saying we don’t want to get involved with this whole question of determining speech particularly on iCloud which is supposed to be a secure private area.
Brian Levine:
Well, so I would say that I think to sort of rephrase what you’re saying, if they were to blindly accept this database without knowing the images that went into it. They don’t know what else they’re checking for. but I would say if that’s your I would say two things first, if that’s your opinion you have decided in this trolley problem that it’s better to ensure that free speech is not trampled upon at the expense of the CSAM that is placed on their network. And you know again, I think if you go to the heat Initiative website and you’ll see actual cases of people using iCloud for CSAM distribution, I’m not making this up.
Ethan Zuckerman:
Right. This isn’t theoretical.
Brian Levine:
Yeah, the other thing I would say is there is no room for any other solution here in this realm. So for example, Apple must know about some CSAM that was found on their platforms. How do I know this, because I can again point to you cases where the police have come to them and asked for content that they know to be CSAM on Apple’s network. And Apple could have created their own list in short of CSAM that they know about and to my knowledge they don’t do that. So these companies that don’t trust the government could be creating their own list of CSAM so, you know, why not even that?
Ethan Zuckerman:
So there are lists out there, some of them are shared lists, so for instance Meta, Microsoft, and some others have gotten together and built one shared list. There’s other lists out there. These are lists of hashes of known CSAM. To be clear, they can’t do everything right a novel piece of child sexual abuse material will not trigger this list, right because it’s not it’s not on the list already. And I know that you and I have talked in the past that video is a real challenge with this. If you edit a video, so it’s a second shorter it no longer has the same signature. Do you check individual frames? So this is not a perfect solution, but it is sort of the minimum.
Brian Levine:
It’s the minimum bar. Yeah, okay, it’s fast and it’s low CPU intensive and again, I think there’s no easy solutions here, but you have to ask yourself if you’re going to allow people to use your services. If you put out a playground so that children can play on and you put out a rusty slide or you, I mean if I put up a Playground to make a broken analogy and I made it convenient for people to use that playground to abuse children there’d be just enormous public outrage. And so I think to say that free speech trumps everything including the privacy of victims of rape when they were children, I just think is an untenable position. And we have to work to find a place where we can start to solve this problem.
Ethan Zuckerman:
We’ve been tal[1] king about Whisper, the truth of Whisper is that what you’re sharing is text right? And you’re sharing your sort of willingness to contact someone in trade CSAM. That’s gonna happen on somewhere else our friends over at the Stanford Internet Observatory have been researching the appearance of child sexual abuse material in the Fediverse. So the Fediverse to catch everybody up on this. This is this open, participatory alternative, to systems like the former Twitter. There’s a service called Mastodon, which is a Twitter-like service. You can run your own server and take responsibility for what’s going to be allowed on that. If you go back through the history of Mastodon.
There have been big splits in the community over what’s acceptable content back in 2017, I wrote a paper about the ways in which Lolicon, which is to say non-photographic imagery, illustrated imagery of young people in sexual scenarios was quite popular on Japanese mastodon and was leading American mastodon servers to de-federate. Essentially say we’re not going to share that content. The folks at Stanford Internet Observatory found a couple hundred appearances of child sexual abuse material on Fediverse servers not necessarily enormous numbers, but very problematic. I run multiple mastodon servers. Can I have access to these hash databases to check for CSAM on my servers? I want to do the responsible thing. I don’t think anyone using Small Town, our space for community discussion, is using it to host CSAM, but I want to check and make sure. Why or why can’t I access those hash databases and go ahead and do that?
Brian Levine:
Okay, so I think generally you cannot. This is nothing that I have anything to do with so this might be an uninformed opinion, but to my knowledge, to my knowledge the list of hash values or digital fingerprints of known CSAM is kept within the confines of the trust and safety teams that are running the larger platforms and all law enforcement. And I think, and I’m just guessing here I think the reason is to publish that list would encourage some elements of our society to create a collection of every image on that list like, a you know some of the people from what I hear, and I’m not a sociologist or criminologist, but I’ve often heard that some significant portion of people who collect and possess CSAM are what they call collectors .They sort of obsessively gather as many photos as they can, you can think of. It’s been explained to me as someone who might be a sports fanatic. If I’m into the Yankees I want every baseball card of every season that the Yankees have. So you know people do it this way, so I think there’s a fear of releasing that list. That’s not my decision. I don’t I don’t manage this list.
Ethan Zuckerman:
Well, there’s also there’s a red team for your associate with this as well, right? So let’s say you want to start “CSAM-R-Us” and one of the things that you want to do is not get shut down, so you start a mastodon server and what you do is alter your images digitally, so they’re still visibly CSAM but they no longer are finger printed within this database.
Brian Levine:
Well, there are other methods of detecting CSAM that will still detect that image. You know, it’s very public that there’s something called a perceptual hash that will try to overcome small changes in files that would cause the hash not to line up. This would also overcome your scenario where you cut out a frame or a second of a video those methods sometimes have or usually definitely have a false positive rate that could falsely identify something as CSAM. They require human intervention but they generally cut down on the amount of work. You don’t have to review every single file you review some of them, but what I also wanted to say is there are companies out there that can send all images that are uploaded to you out to them electronically and they will match against this database.
I believe Thorn does this. I don’t think they do it for free. I don’t know if they have a sliding scale, but that’s available to you and I would ask just because okay. And then the third thing you could do is you could allow people to post whatever they want but you put together a moderation team who will take the time with their human eyes to evaluate every picture that’s up before it gets published. If you wanted a fourth option, you could educate your users about what CSAM is and I think that’s an important part about what I’m going to say next and you ask them to report what’s going on. And I think all of these things are challenging by the way because someone who is 17 years and 350 days old if it’s an image of them and a sexual act and they’re nude and all that I mean that’s CSAM and when they turn 18 it’s no longer CSAM.
There are studies that show that it’s difficult for humans to determine the age and etc etc. So I think if, I think what I want to get back to though is if you’re going to put up a platform I don’t think it’s right to say wow checking for CSAM is difficult and by the way, I’m not making any money well, maybe you shouldn’t put up that platform them and your business model puts children in harm’s way.
Ethan Zuckerman:
Right, and so this is you know as promised where we get to places where you and I experience some tension because I’m very interested in this question of can we build a pluriverse of small social media platforms run by their communities?
Brian Levine:
What run means, no economic activity is that what you mean? There’s no charge for these services. I think that’s an important part of your question.
Ethan Zuckerman:
Yeah, so absolutely in some cases, right.
Brian Levine:
So but they are willing to put money in, I’m interrupting you so we’re already disagreed Ethan. But I want to make clear if you don’t mind, can you be clear you don’t want to charge the users any money? But you are willing to say, I mean someone has to pay for those servers. So some money is going to be put into this platform and I would say are you selectively putting money into the servers and not moderation?
Ethan Zuckerman:
So absolutely, I’m going to spin up the scenario and then we will figure out where the questions that have come into this. So we are on the show, in our lab in general advocating I would say for two types of communities that are interesting in this setting. One is communities that treat an online space as a public good. Which means they might fund it the way that you would fund a library or public park. You might pay for it as a taxpayer. You might pay for it in the way you pay for a school and an example there would be PubHubs, which is a Dutch platform trying to build an alternative to Facebook groups around public schools and absolutely they’re putting money into it. They put money into the moderation and such.
A second example would be cooperatives. And so an example here would be Social.coop which is a cooperative Mastodon server run by Nathan Schneider over at CU Boulder. It’s one of the best projects out there. They are asking members to collectively kick in money as well as kick in time around moderation. So in both cases responsible communities are actually being moderated but don’t have access to these tool sets at this point.
So your take on this and I’m preempting, but I’m gonna let you go back in a second, is don’t start up a social media platform without being responsible enough to moderate. We’re in stunning agreement on that one!
Brian Levine:
Great, end of show.
Ethan Zuckerman:
I’m gonna come in and say: but we need tools right because these platforms succeed. You’re suddenly doing this at enormous levels. And it’s very very hard to scale it and it’s just not great work. Like scanning images and then trying to figure out if it has the signs of being seen as something traumatic is enormously traumatic. That’s enormously damaging.
I’ll just tell a very quick story on this. I built one of the first trust and safety teams before we called it trust and safety. And I was doing it for Tripod.com in the late 1990s when it was the eighth largest website in the world. We managed to get as far as a flesh checker which looked for flesh tone pixels which tended to either be nude imagery or close-ups of faces. And we would then put it in an interface called flesh finder where you basically had two buttons either okay, or not okay, and not okay would zip up all of the users files and put them for review for things like CSAM delete them from the server because we just didn’t permit nudity on the server.
For the most part in the late 1990s a good friend of mine started having body image problems and started seeking psychological help based on how much of her time she was spending looking at pornography and looking at unrealistic body images, so there is real trauma sort of associated with this. So one of my questions in all of this is how do we build a future in which if I’m trying to do the right things that I’m being responsible? I have access to the tools that Facebook and others have?
Brian Levine:
Well, so I guess I would say again, I don’t I’ve never worked for Facebook and this is just based on what I’ve read but my understanding is the people that work for Facebook even though they have tools when you know a lot of these tools are there for triage or to put it another way a lot of the tools make the job easier of a human reviewer. You can’t. You often can’t solve this problem by completely taking humans out of the loop, and so I think the people that do moderation for Facebook on these other large platforms are experiencing trauma and I think that you can’t. I personally believe that the problem is checking for content. That’s horrific. That being allowed for publication on your platform causes your moderators trauma.
If the solution to that is not to do moderation or say like well I mean what about the children if you take away the moderation? What about the people who are survivors of that abuse who appear in the images like this is the trolley problem, and I’m not willing to sometimes in security we talk about failing open or failing closed so. What happens when you know, what do you do in the scenario if you’re overwhelmed with content that causes your moderators to quit? Do you fail by shutting down uploads or do you fail by allowing anything to be uploaded? And in the case when you fail by not allowing anything to be uploaded you’re you’re you know in the worst case chilling free speech. But you’re definitely stifling people’s voices and in the case when you’re just allowing anything to go up there until you can get to it well then you’re harming people otherwise.
So I guess the way I would state it is there’s no magic here but I don’t think the key lesson I would want your audience to take away from me is don’t think that you have the right to start a platform without thinking through what you’re gonna do when you get to that scenario. That’s the important piece and if I can say one other thing. I think to get bring this back to the app stores. You know there’s this rags to riches story at least in computer science in the morning when you wake up you have this great idea for a new app that’s gonna make you a millionaire. If you’re good at coding you can code it up that day, and you can get it on the app store maybe that night and start earning money the next day. And that should not be allowed in all cases if you’re going to say put a child at risk.
Ethan Zuckerman:
So let’s talk about maybe the most famous version of that rags to riches story which sets up another very challenging trolley problem, which is WhatsApp. So WhatsApp is an enormously popular service that is now owned by Meta, but it was started independently, probably in the neighborhood of three billion users worldwide. And is one of the most prominent examples of a system that depending on how it’s configured promises end-to-end encryption so in the field that we work in e2e. The trick with end-to-end encryption is that the platform running it disclaims responsibility for the content because they’re able to say…
Brian Levine:
Great position to be in, it’s very economical.
Ethan Zuckerman:
So it’s not hard to imagine a CSAM WhatsApp group where you are sending imagery that ends up encrypted so that it can only be seen by you and seen by the recipients. To the best of your knowledge, how is Meta handling CSAM on WhatsApp?
Brian Levine:
So to the best of my knowledge when you are connected to someone else with WhatsApp and you send them a CSAM image Meta will do what they can computationally to detect that it’s known CSAM. I don’t know if they’re doing some sort of machine learning to detect as of yet unknown CSAM that a machine learning classifier can detect. But my understanding is that’s the basis for a lot of the cyber tips that they’re sending.
If you were an apartment owner, a building owner and you know, you were an absentee landlord and all of your apartments were being used for a you know many many years to commit awful crimes against children and then you said you know what, I’m gonna ensure the privacy of some of the apartments are used for that. But it’s a you know, of course the analogy starts to break. But if you know there’s extensive crime that’s taking place on your platform and your solution to that is to ensure that you can’t see it anymore. I don’t know how that’s a responsible decision.
I think you know what makes WhatsApp different to discuss is hypothetically sure and definitely not against adults talking privately. They’re owed that in society. But if you know something is extensively used. If you know, three out of eight people on the earth are using it and you know that it’s extensively used to harm children. I don’t know what ethical discussions were had there to arrive at the decision that you no longer want to be privy to the fact that these crimes are being committed. I think that’s the issue, not whether adults should have the ability to talk privately.
Ethan Zuckerman:
So before we get into the ethical issue, let’s work with technical issues, right? WhatsApp is saying that in transit your message cannot be intercepted and decrypted. It is unencrypted when you are uploading it into the client. It is unencrypted at the destination of all of it. If WhatsApp wants to monitor the content it can use either a set of known hashes or perceptual.
Brian Levine:
Currently, this is the case.
Ethan Zuckerman:
Currently, this is yeah, and so our assumption is that WhatsApp is doing that.
Brian Levine:
For most users. Yeah currently I hear in a very short amount of time. They’re going to ensure that by default everyone’s messages are encrypted end-to-end and they will no longer be privy to the content that’s being exchanged on their network that is known CSAM or even as I said maybe as of yet unknown CSAM.
Ethan Zuckerman:
But they will be privy to the content in the app.
Brian Levine:
Oh once it makes it to the user’s phone sure. Well, but then they would know it then but they can’t put that list on the user’s phones. I mean they could take a hash of the content and then you know send the hash of the content up to the cloud. That would be another option. I don’t know that they have plans to do that
Ethan Zuckerman:
So interesting. So your sense is that WhatsApp is becoming immune to checking against these sorts of systems that we’ve been talking about?
Brian Levine:
That’s right, and I don’t know why they’ve made the decision. But I want to point out that if they can no longer check the content of I don’t know how many Billions of messages a year. They certainly don’t have to hire very many moderators anymore, so there’s at least a financial incentive for them to stop doing that. And then they can also say that hey, we’re the most secure method of people talking, but I don’t know that they’re gonna make a big press release about the fact that they’re gonna, I mean there’s gonna be an enormous observable dip in the by tens of millions I would think of cyber tips received a year. And you know, I think that’s notable.
Ethan Zuckerman:
And we should be clear that Meta is not the only company doing encryption. I have Signal on my phone which is powerful in encryption. My guess is that Signal is not in fact checking my images. I don’t know. It’s an interesting question. My guess is that their privacy policies are such that they probably are not.
Brian Levine:
Yeah, and like I said, I think that’s interesting for them to do. I mean so, earlier I said I think this is a difficult issue and where we can carve out solutions. I think we should, so I don’t know that we’re gonna carve out a, you know, a solution for all of society to allow adults to have privacy. But also be able to check for CSAM and also trust Facebook and all that.
I will say that you know WhatsApp knows accounts that are underage and I think one of the things they could do is not allow end to encryption between children or between adults and a known child. Why isn’t that something that we can carve out? You know if an adult approached my child in the playground, I don’t know that I’d be comfortable with that. That’s a weird thing but at least I can see it. In fact I think if I sent my kids to the park and an adult started talking to my young child I think probably a random adult would report that.
Ethan Zuckerman:
But I can give you another analogy, which is a child who’s being abused by their parents, you know seeking out an adult for a secure contact to find help in one fashion.
Brian Levine:
And why can’t why would that? Why are you worried about Facebook being privy to that conversation?
Ethan Zuckerman:
I think in general you’re looking at an attacker that can reveal a child to an adult at that point, so you’re looking for, you’re right, Facebook’s not exactly my antagonist at that.
Brian Levine:
Yeah, so I think given that Apple and Google and Android, you know companies know that which is a lot okay, so let me backup. I know for a fact everyone knows for a fact there are children who have registered for adults for these platforms, but I would also say if you know someone is a child then you should not be it’s better to protect children’s privacy by not encrypting their communications end-to-end not allowing connections from Tor from those apparent child accounts.
No child needs Tor Sorry, especially not in the US not allowing. What do children need VPNs for right? So actually increasing a child’s privacy by not giving them access to those tools because some of those apparent children are adults who are masquerading as children to entice and groom. Children and cause the catfishing them sometimes this is called causing them to send pictures up like allowing evidence on scene on the platform for law enforcement to do investigations would would be a huge start and I feel like we could carve that out and separate it out from should adults be allowed to have end-to-end encryption.
Ethan Zuckerman:
So let me try to do a top-level view of the field and then it’s going to turn into the question of what should we be doing as parents and what should we be doing as software developers? So top-level view of the field there are public social media platforms where information is out there in public. Many of those large platforms are using content hashing perceptual hashing and other ways of trying to detect CSAM so they are not hosting CSAM on their platforms.
There are also some platforms that are not doing that and we can probably agree that that’s probably not a very responsible thing to do even for something like a Mastodon server where you might not be able to get access to those powerful software tools there’s the possibility of essentially partnering with a company like Thorn to be able to have your imagery checked. There’s also the possibility of moderation to sort of find a way around a lot of moderation manual motivation to find a way around. In the case of end-to-end encryption there’s probably not a good way to do this without compromising device privacy,
Brian Levine:
So if you demand absolute privacy, and no one can be involved in the communication, you’ve taken such an extreme stance that you’ve limited the ability to insert some other technological tool.
Ethan Zuckerman:
Right, so then if you design WhatsApp so that when you’re uploading an image to send, it’s getting checked against a CSAM checker or something along those lines you may have the problem that you know people now, say oh now WhatsApp is routing me out. I’m going to use Signal instead. Your take on this, the very least is you would probably not allow an encryption for children for children talking with adults something along those lines and making the argument that in the trolley problem trading off that loss of privacy against the danger of child exploitation.
Brian Levine:
Well, I would say you’re gaining privacy in a different way. You’re allowing children to be rescued. You’re allowing law enforcement to perform investigations and again. I’m not naive. I don’t think all children are going to be sign up as children, but I would also add to this if you are a business and you’re providing free services in order to increase your your customers, maybe the free services should not be end-to-end encrypted only those people who register with a real email or you know, gave you a credit card.
You know, like for example, what if you’re Zoom and you provide free ability for people to conference online. How do you know people aren’t getting together and doing things illegal on the zoom platform? If they’re doing that on the free platform, I don’t think that should be end-to-end encrypted. I think that would be a more responsible way even though they’re adults if they’re adults and they want to have end-to-end privacy. Well, then I think people will disagree with me about the more responsible way to do that as Zoom and I’m just picking on them. There’s lots of there’s lots of platforms out there to have conferences I think you should only allow that end-to-encryption if you have any details about about those users and I just don’t think, you know I’m sure some people are gonna complain to me about citizens and other regimes and so on I don’t think any of these tools are good enough to protect citizens and those other regimes especially Tor.
And I think it’s ridiculous to say that Tor is, you know, any, especially, Tor onion services worth anything for people who are in other regimes. And certainly children that are there are not going to be protected by these tools. So I think it’s important to work towards solutions where we can that have a maximal benefit to society.
Ethan Zuckerman:
I think I’m still on the Tor advisory board, and so with that in mind I’m gonna come in and say I actually think Tor is potentially really helpful if you’re in a situation where you have cooperation between a platform and government. And we’ve seen cases where someone is trying to speak on a platform. The platform might well know who the person is based on the IP they’ll freely give that information back to the government. I don’t think anyone should be using Twitter as a secure platform to speak at this point but you certainly can see evidence right now that Twitter is handing troves of data to the Indian government. If you were trying to tweet in India and have some degree of privacy to me, Tor feels absolutely like something that you would want to use in that case
Brian Levine:
Well, so I’ll agree partially and then strongly disagree. I don’t doubt for a minute that there are companies that are uncooperative. I don’t doubt for a minute that there are companies that are cooperating with different regimes that are out there. And using the data that they have on their servers to allow those corporations. You know, to allow those governments to persecute or even just harass different users that are out there to those people, I suggest that Tor is not even close to the solution that you need to protect yourself from a government.
There is no government on the earth from the poorest government to the richest government that can’t get beyond that the security of Tor and I think it’s honestly very questionable that the Tor website does not include, I think it’s objectionable I strongly object to the fact that on the Tor website they suggest that dissidents and journalists use tour to protect themselves from governments. I have in the past asked directly to the people that run Tor that they include on their website this is the amount of resources that it would take to break the security of Tor and if your adversary has that amount of money or more do not use tour to protect yourself and I would argue that’s in like the tens of thousands I think there are delicate essence around the country that could break tour if they needed to to put that between you and a government especially one the size of India or even let’s just say Liechtenstein not to pick on them. I think I don’t know if economically there are powerhouses, but they seem pretty small to me. I think that’s a huge mistake and I think I have a lot of problems with that.
Ethan Zuckerman:
I’m gonna put for a moment aside the questions of how secure and insecure Tor is although, I personally want to pursue that with you at some point in the future. My characterization of this landscape that we’ve been discussing today would now be as follows. There are public social media platforms. There are some tools. They are limited in terms of what they can do as far as detecting CSAM.
Brian Levine:
Even for the likes of Facebook.
Ethan Zuckerman:
Even for the likes of Facebook, right? Perceptual hashing has a high false positive rate.
Brian Levine:
Well, I don’t know if I have a false positive rate.
Ethan Zuckerman:
It has a false positive rate that requires manual review. A content hash of existing content gets you existing content doesn’t get you new content, right? Okay
Brian Levine:
We agree that human moderation is key.
Ethan Zuckerman:
We’re agreed that human moderation is key. That’s right. There is now the emergence of end-end encrypted systems, they may be connecting very large numbers of people because we have systems like WhatsApp that are reaching many billions of people. You can compromise the client or you might argue, make the client more responsible by essentially having it participate in content hashing or perceptual hashing. Our sense is that that is not where those clients are generally going.
Brian Levine:
Yeah, because I think not because if you’re gonna do that on the client you might as well do it on the cloud. You know, I want to point out this end-to-end encryption. You’re already trusting Facebook. Are you examining the code? Are you examining the code signal? Why not trust them to do it in the cloud in the first place, but go ahead.
Ethan Zuckerman:
Where I’m going on this is this sounds like a tough situation, in a situation that is getting worse. What are you telling parents to do in this situation? You’re a parent of kids, and how are you talking to other parents about?
Brian Levine:
Yeah, so whenever I get this question. I say, what are we telling tech companies about this? I don’t think this is a problem that parents need to solve. I think we have a thought in this country, an assumption that the tech companies are doing enough by some virtue of they’re so good at what they provide to me. My Apple watch is so useful that they must be doing all they can, and I’m here to say they’re not doing nearly enough, really any of them and I think we need to hold them to a higher standard.
I think for example, there are laws in this country that provide blanket immunity for the harms that children would receive on platforms provided by the richest companies in the world. And I think that in exchange for this blanket immunity we as a society have gotten nothing. For example I don’t need them to take away that immunity. I don’t. I don’t need to argue that I think for example, I could demand that if you want that immunity, I would like a public transparency report documenting all the harms that have happened on your platform. I would like an annual review of whether the content that appears on your platform matches the age rating that you’ve gotten. I want to know.
I want parents to have more information from these tech companies in order to make decisions. So I think parents can’t do enough right now because they’ve not been supplied with just how dangerous it is out there. I also want to say I don’t want to scare anyone like I don’t you know, this is very akin to cars. In terms of we all have as I said before we all have by law I’ll mention car booster seats and car seats for infants that are required for us to drive children around because we’ve learned that cars are inherently dangerous. And they’re not designed for children. That’s what the booster seat is all about. It puts the seatbelt at the correct height and if we are to adapt to these technologies that are used for people to get around society. Which I think is a form of free speech.
If not, you know free civic action and no one would ever say that you can’t leave your house. I think that’s a kind of free speech then we have adjustments that allow children to be safe. For accidents that are not that common I mean, I’m very fortunate that I don’t know of any children in my life even though you know, friends of friends where a child has died or been hurt in some significant way in a car accident. Now maybe it’s because they all have car seats or the cars have been engineered to be safer.
And so okay, so to go back I think the first thing that that parents need to do is demand that we have better statutory regulation in the society and demand more that the richest companies on the earth who are the world’s largest toy makers give more back to society in exchange for this blanket immunity that they have or that they give it up. But for parents, I think they need to research how dangerous certain apps are.
I think just Googling is a good first step. I think the second thing is very hard to have a frank conversation with your child about what might happen if they are online. And you know, I think there’s better resources than I could list here about what rules one might follow as a parent, but I think I think you need to have those hard conversations. I’ve had them with my kids. It’s difficult, but I feel safer and I think they’re safer for it. Long rant, I’m sorry.
Ethan Zuckerman:
No, it’s a super useful rant and particularly where it ends up. As far as the parenting part of it, I feel very responsible. I think where I’m a little worried about this is that I think child safety often becomes an excuse for whatever we want to do with regulating platforms in ways that may not be particularly helpful. So a piece of legislation that’s making significant progress in Congress to the extent that anything makes progress in Congress these days is the Child’s Online Safety Act or COSA.
Critics of COSA have pointed out that it gives parental control over use of social media in a way that’s probably really damaging to kids who are searching for information about gender identity. There is a danger that some people are going to hear what you’re saying, and say we got to get rid of Section 230 immunity. We got to make platforms do a whole lot more let’s get behind things like COSA, without considering other pieces of this like the fact that if you are a queer 15 year old and if your parents are not supportive of you the internet may be the only place where you are finding affirmative voices supportive voices and what fashion or another. That to me feels like another trolley problem in sort of balancing all this.
Brian Levine:
Yeah, and in fact I previously wrote a report to Congress on behalf of the National Institute of Justice that examined questions related to what you’re saying. And since I’ve written that report, which I finished in 2021, but was published in 2022. Looking back that is one of the big emissions I wish I had as some notion that that is a harm to children who don’t feel safe in their own homes. I think that’s a great thing to bring up but I would say that is also an instance of bringing out children to you know as a reason to pause to give companies something in exchange. Like I would say that I didn’t say that we should repeal section 230, and I didn’t say we should, well what I said was, whatever I said, what I want to say is that I don’t know for sure that these companies should lose their immunity.
What I do believe is that right now society gets nothing back for that immunity. I think something like these transparency reports. The other thing I’d say is if a child does not feel safe in their own home, if it’s the case that a child doesn’t have secure access to their own phone and to end encryption is not going to solve that if a parent has access to a child’s phone that is a different kind of security problem. I think it’s really important for actually not just society but I think it’s a huge problem in computer science right now that too many computer scientists think of computer security as okay this user is going to put some important information on this app such as their communications with others how do we make sure no hackers can get into that.
And I think what you have to ask is what about the other people in their lives who have intimate contact with them because they’re a parent or they’re a step-parent or their friends, and I think that is a you know a really important security analysis, which I think is often just not even in the purview of the academic community. You know what other problems might come so backing out though. Like I don’t see how giving companies blanket immunity and requiring nothing from them protects children who don’t feel safe in their own homes and don’t have secure access to their phones. When you know a parent might have that, so I don’t think that’s all but I agree. It’s another trolley problem and any solution that I would get behind would have to consider kids that are abused or don’t feel like expressing and don’t feel safe expressing who they are for fear of persecution. I think that’s important.
Ethan Zuckerman:
I think for me if there’s a sort of summing up or a takeaway from all of this is that none of this is easy.
Brian Levine:
None of this is easy at all.
Ethan Zuckerman:
None of this is easy to come up with a prescriptive solution. None of this is easy to come up with a technological solution. To almost anyone who’s giving you a blanket statement, you know, don’t one way or the other. Yeah, it’s probably not thinking through it in detail. Brian I just want to thank you for you know having maybe more of a back-and-forth than we often have in the show. I invite all my guests on you know with the disclaimer that we’re having them there because we admire them and love the work that they’re doing. You know here the work that you’re doing is wrestling with really challenging problems, where there just aren’t easy answers and I really thank you for being with us.
Brian Levine:
I appreciate the opportunity to talk to you about it and it was a great discussion, and I don’t know, hopefully we both learned a lot from doing that.
Ethan Zuckerman:
He’s Brian Levine. I’m Ethan Zuckerman. We are here at the University of Massachusetts Amherst. You’re listening to Reimagining the Internet, and I would just say in general is it a little UMass pride. We’ve got really great stuff going on here at UMass Amherst right now and some incredibly smart people. I am so thrilled, and so proud to be here. Colleagues like Brian are part of the reason why so thanks so much.
Brian Levine:
Thank you.