I love a good chart. I really do. A good chart can take massive, unwieldy amounts of data and condense them down into easy-to-understand graphics that help illustrate a point. For instance, I think Media Matters, my former place of employment, does a great job of using charts to make points about the way certain topics are covered in the press and on social media. Earlier this month, they put together a really compelling research document about the way right-wing media outlets dominate the conversation about transgender-related issues on Facebook.
What makes those good charts, in my view, goes beyond the graphics themselves. At the bottom of the above link, you’ll find a list of how pages were categorized as “right-leaning,” “left-leaning,” or “ideologically nonaligned;” a list of keywords that were included in the data collection, and other relevant info.
But I’m not here to write about good charts today, sadly. I’m here to write about bad charts (and where they miss the mark).
First off, by “bad charts,” I’m not referring to charts that misinform people or are malicious or anything like that. Those exist, for sure. No, what I’m referring to are the charts that may have come from a good place with good intentions, but either don’t make the points they think they’re making or fall short on a methodological level.
This morning, on Twitter, I saw “The Conspiracy Chart: 2021,” a graphic by Abbie Richards, who describes herself as a “mis and disinformation researcher with a focus on TikTok and conspiracy theories.” I’m interested in conspiracy theories and misinformation, so I excitedly dug in.
Something about it seemed… off. Under “Things that actually happened” are, well, things that actually happened. Great. So far, so good. But it’s at the next level up that things start to get murky. Just past the “speculation line,” filed under “We have questions,” are a list of things that are… just odd pairings. “JFK assassination, “Jimmy Hoffa disappearance,” “Iran Contra,” and “Area 51” are all pretty clearly things that do exist or did happen, but with caveats, like… who killed JFK, how involved was Reagan in the Iran Contra scandal, what happened to Jimmy Hoffa, and what’s at Area 51. Yes, there are a bunch of conspiracy theories about Denver International Airport (cursed statues, secret Illuminati tunnels, etc.), but those seem more suited for the “unequivocally false but mostly harmless” category above it. The same goes for “We live in a simulation,” which is more of a philosophical question (What does it mean to exist?) than anything else.
What’s particularly frustrating about the chart is that it explains so little. “Soy boys” is filed under “Dangerous to yourself and others,” placed between “Reality denial” and “The antisemitic point of no return,” but there’s no actual explanation of what it is, what the conspiracy theory is supposed to be, or really… anything. (More details at the footnote below)
But maybe I was totally off-base, and this was actually a good chart. After all, I really don’t think it’s necessarily wrong and I do feel like it was probably made with the best of intentions. To check myself, I reached out to Christopher Ingraham, author of the great The Why Axis newsletter (which you should absolutely subscribe to) and overall chart genius. He summed up the problem with the conspiracy theory chart:
“I think the main problem is there’s no methodology disclosed here and it’s basically just categorized based on subjective vibes,” he tells me. “Which is fine as a ‘this is what I think’ exercise, but I think trying to package that as ‘The Conspiracy Chart,’ like it’s the one true objective resource that people will need, is what’s rustling jimmies here.”
“The conspiracy chart's trying to present qualitative assessments of credibility in an inherently quantitative visual framework,” he adds. “There's a natural tension there, and it's really hard to do it well without meticulous documentation and disclosure of your assessment criteria, which is unfortunately absent from the chart or the maker's online discussions about it.”
Ingraham added that the chart might not be a bad summary or teaser visual for a lengthy whitepaper on misinformation and conspiracy theories, but that it was odd to present it as a standalone thing.
I reached out to Richards prior to publication in hopes of understanding her methodology. I’ll update the web version of this post if I hear back.
On the topic of bad graphics, let’s talk about one genre of them that really bugs me: media bias charts
We’ve all seen them in our feeds. Many people retweet them and, bizarrely, trust them. Yes, I’m referring to the Ad Fontes Media Bias Chart that gets an update from time to time.
You may notice a few things here. For one, the more partisan the outlet (according to Ad Fontes’ extremely confusing and totally subjective methodology), the less reliable it is as a source. Weird how that’s just a perfect correlation. Gosh. I could go one by one through the list and point out why they’re wrong, why lumping The Daily Beast (which has broken hundreds of legitimate news stories and has served as a jumping-off point for a number of extremely successful journalists on their way to outlets like The New York Times or Rolling Stone) with Just the News (an outlet run by John Solomon, whose work at The Hill had to effectively be relabeled as opinion after being repeatedly debunked) in terms of credibility isn’t an apples-to-apples comparison (in fact, this chart has Just the News higher than The Daily Beast). I could explain that Politico, The Economist, Newsweek and The Bulwark aren’t actually left-of-center (The Bulwark was founded by Charlie Sykes and Bill Kristol, for crying out loud!). I could explain that Pod Save America, founded by center-left staffers of Democratic politics is not the left equivalent of Infowars. I could explain that no, you shouldn’t consider Upworthy (a place where I used to work!) as a source of original news.
But what’s the point? Maybe it makes people feel better to believe there’s some sort of balance to the universe, that for every right-wing outlet there must be a left-wing equivalent, that the truth is always somewhere in the middle, and that things like this, which smash news and opinion together for the sake of some sort of faux “objective” rating are less than useless. But it’s false. And it’s not like Ad Fontes is the only place that’s done this, either. Right-wing former CBS journalist Sharyl Attkisson, who once accused the Obama administration of hacking her computer though it turned out she just had a stuck “delete” key, has her own version of this. AllSides has an equally nonsensical chart it pushes.
For the chart-makers in the audience: make sure you’re building your work on top of actual data. For the chart-readers in the audience: ask yourself, “Is this actually data or just vibes?”
The “soy boy” conspiracy theory is essentially a belief that the rise of soy products as vegetarian meat replacement options is part of a plan to emasculate boys and men. The idea goes that because soybeans contain phytoestrogens, consuming soy will cause boys and men to have low testosterone, high estrogen, develop breasts, etc.” As a trans woman, believe me, I wish it was as simple as all of that, but it’s not. If I could toss my actual estrogen prescription in the trash and just chow down on edamame, I’d totally do it, but alas… I cannot. While “soy boy” is sometimes used as part of a conspiracy theory, it’s mostly just a catch-all insult right-wing guys lob at liberals they think are “soft.”
I have never liked the Ad Fontes chart, though I agree it comes from a place of good intentions. Like you say, partisan doesn't mean false. And how Politico gets lumped as LoC means that their methodology is suspect. And how Fox is just slightly RoC and is considered Mostly Analysis is beyond ludicrous. But what do I know? I'm a partisan who adores Wonkette, far LoC be damned.
As a regular reader of the WONKETTE site - poised on the chart BELOW The Palmer Report - I’m really annoyed that it’s not regarded as truthful. It’s definitely opinionated, snarky, and occasionally crude; but the posts are always carefully researched beforehand. I trust what’s posted to be accurate, much more so than I do most of the other sites the designers of this chart regard as more credible.