The platform Tumblr has decided to no longer allow adult content. The ban goes into effect Dec. 17, 2018, and to enforce it, Tumblr seems to have deployed a bot that's singularly bad at its job, flagging innocent content as pornographic in nature and prompting both users and tech pundits to wonder why the bot is so bad at what it does. Part of the answer is that moderation with artificial intelligence is an extremely difficult task.
Many of the subtleties involved in deciding what content we find acceptable or objectionable have to be written down in stone, and our track record with doing that isn't so great. In fact, we have trouble identifying something as pornographic in the first place. The late U.S. Supreme Court Justice Potter Stewart summed up the sentiment in a ruling about an obscenity case (Jacobellis v. Ohio) with the phrase "I know it when I see it."
That sentiment has proven as vague in practice as it is in meaning. Here's an example: A picture of a man with an erection must be lewd in nature, right? But what if this is for a medical illustration of priapism, an often painful condition that causes a prolonged erection, and it appears on a medical site? If any representation of a vulva is obscene, does that mean the work of artist Georgia O'Keeffe, whose paintings of flowers are frequently thought to be visual metaphors for female anatomy, needs to be flagged in art history articles?
Social networks and content platforms encounter these situations all the time. For example, in a major PR incident in 2016, Facebook censored the Pulitzer Prize-winning photo of a naked 9-year-old Kim Phuc running in terror from a napalm attack during the Vietnam War; the photo was published by Norway's most prominent newspaper for a relevant article on warfare. By contrast, Twitter users weren't able to persuade that platform to shut down neo-Nazi accounts until the end of 2017. With different philosophies and seemingly arbitrary, context-free rules that can confuse even human moderators, it's no wonder algorithms are having trouble figuring out what to flag.
Tumblr's system appears to be looking for an amount of what it sees as exposed skin in images, or shapes it believes may be nipples or genitalia. Unfortunately, a lot of benign close-ups on non-erogenous parts of the human body are exceeding the threshold for how much bare flesh an image shows, as Dan Fallon writing for Digg noted. Certain objects like trees could also look phallic. And in one seemingly inexplicable case, Fallon wrote, a landscape photographer's very innocent pictures of nature were flagged as problematic, too. This, however, isn't unusual for such algorithms. Other iterations of censor-bots have flagged pictures of dunes and beaches because the color of the sand was similar to the color of skin according to their training data sets.
This systematic error also makes sense when you consider how many skin tones humans have. Colors from a light beige to nearly black all occur naturally, and depending on how an AI is trained or a sensor is calibrated, it might not understand that darker skin colors even exist. As a result, an algorithm trained to spot and censor pornographic images with Caucasian performers might not be able to flag equally explicit images with dark-skinned models. One of the easy solutions for that is to overcompensate, flag everything and justify the false positives as being better than not catching enough adult content, which is what Tumblr appears to have done in the scramble to sanitize its content. Tumblr did not return requests for comment as to whether there was an additional layer to its moderation.
Ultimately, it remains to be seen whether an overzealous censorship algorithm will drive users who don't post adult content from the platform too, or whether it will be dialed down. But Tumblr's very public and aggressive over-flagging highlights some of the many difficulties in moderating online media. So mistakes will continue to be made, and made frequently. Until we can figure out how to address these concerns, humans will need to oversee any truly successful moderation effort.