Pew doesn’t understand news ecosystem well enough to study it

Note: I have added an update, in bold below, since originally posting this.

A study of Baltimore news sources was more deeply flawed than I initially realized.

I blogged Monday about weaknesses in the How News Happens study by the Pew Research Center’s Project for Excellence in Journalism and about the misinterpretation of the report by many journalists and media outlets. After further study of my own and a response from Tom Rosenstiel, director of PEJ, I have concluded that old-media biases by the researchers were so profound that they truly didn’t understand the “news ecosystem” they were studying.

Before I explain why I now totally discount the value of the Pew report, I should admit the flaws in my own blog posts. I have been working on lots of other things this week and I banged out that blog post pretty quickly Monday. I was annoyed at how some reports by media organizations and some individual journalists in tweets and blogs were trumpeting the study’s finding that 95 percent of stories reporting new information “came from traditional media—most of them newspapers.” Too many newspaper journalists crowed with delight over that finding, as though the study had said, “Nyah, nyah, nyah, without newspapers, all those bloggers would have nothing to write about,” which is what a lot of newspaper journalists think. I find that attitude to be one of the biggest obstacles to innovation in the news business, and I might have a bit of a hair trigger in responding to it.

As I noted Monday, the Pew study found a lot of disturbing things about the Baltimore media, too. And the people who were all excited about the 95-percent figure overlooked much of that. Further, the Pew report’s key findings relied on close examination of only six running stories during one week last July. That’s too shallow a study for any serious conclusions (though, in fairness, the study described its research as only a “snapshot”).

I read the report, noted a few more flaws, made a quick check of a blog post I remembered by Mark Potts about the variety of Baltimore media outlets and fired from the hip. I was faulting Pew’s research, using hardly any research of my own. In the process, I did something that I consider pretty shaky for a journalist to do: I speculated. I correctly concluded and noted that the selection of blogs studied by Pew was too narrow. But I didn’t do the research to find a blog that had been excluded. I just picked a blog from Mark’s post and made a hypothetical statement: “If Baltimore Real Estate Investing Blog did a better job of reporting what might happen with the sale of that theater (the report noted that the advance reporting by the media mostly failed to raise the possibility that actually happened), the study wouldn’t know, because its selection of news outlets was too narrow.” (I should add that I only picked the sale of The Senator Theatre from among the six story lines studied because I could imagine the real estate blog possibly writing about it. If I had noticed a blog in Mark’s post that covered education issues, I might have supposed that it could have written about a controversy over state budget cuts, another of the six story lines.)

I don’t approve of journalism by supposition, but I chose the right story line for which to guess that the Pew researchers might have omitted a blog. (Or perhaps that was inevitable. I’m supposing again, but I would not be at all surprised if we could find examples such as I am going to describe shortly for most, if not all, of the story lines studied by Pew.) I did not correctly identify the blog that contributed significantly to the news ecosystem in the Senator story, but there was one. More important, I was wrong to say the “study wouldn’t know” about the blog.

Here is the truth, as I learned Friday in an email from Rosenstiel (a well-respected journalism leader, I should add): The Pew researchers did know about an active blogger who generated a lot of content about the sale of the Senator, and they deliberately excluded that blog from the study. I will tell more about this blogger (Laura Serena, author of astrogirl’s galaxy guide) and why she should have been in the study and why I think this invalidates the entire study. But first, I want to give Rosenstiel his say. I appreciate that he responded to me.

Here is the passage of Rosenstiel’s email message that dealt with my criticism of the study:

The staff that did the research on this particular thread have responded to your note. It is attached.

As you will see, we did capture the Astrogirl posts and were fully aware of them. Based on her own self descriptions, she does not  represent herself, nor does she function, as a news organization. That put her outside the parameters of the quantitative element of the study. She was an interesting voice on the subject of the theater, but she was not a news gathering outlet, and it was the ecosystem of news we were studying here. A different study, looking at the robust debate that occurs in response to news gathering, would include many such voices.

Here is the note from the research staff (I asked for their names, but Rosenstiel simply responded that I could attribute the statement to him):

Steve,

Thanks for the careful reading and response to, our study “How News Happens.” As we say in the introduction: it is “only one attempt at trying to understand who is producing news and the character of what is produced. Additional reports could tell more. But this snapshot was in many ways a typical week—marked by stories about police shootings, state budget cuts, swine flu, a big international soccer game in town and a mix of fires, accidents, traffic and weather.”

Our audit of Baltimore media did identify the Astrogirl blog you cite. As Ms. Serena acknowledges, it is an organ of advocacy that favors the point of view of her friend, the former owner of the theater. She volunteered at the theater, is active in the “Friends of the Senator Theater,” opposed the auction and participated in the protests. The advocacy element takes the blog out of the category of general interest, public affairs news outlets included in this study. A number of the identified blogs and online-only media outlets did fit these parameters. Even as she wished we had included advocacy sites in the study of news outlets, she said, “The fact that the Pew study was flawed by leaving out some new media sources does not detract from this conclusion being essentially accurate. What might change if a larger sample of new media was used is the percentage, but probably just a bit.”

To address your other point, the study wasn’t limited to the six story lines; we did first-level analysis of all content captured for three days during the week. It formed the basis for our analysis of who produced the most content, which were the leading news topics, an analysis of package news reports on television and radio and a breakdown of local, national and international coverage in newspapers. For deeper study, we followed  six storylines related to public affairs that resonated in the media that week and that attracted coverage from multiple sectors.  Note that in our analysis of each story line we sought out and included stories produced by blogs and websites that were not part of the original 60-outlet sample.

Again thank you for your attention to this report. We look forward to conducting further research on the subject.

PEJ

After I posted this Saturday morning, I emailed a link to Rosenstiel, who responded again by email:

Perhaps we weren’t clear. The blogs you talk about was not excluded as you infer. They were captured, examined and included in the chronology, the qualitative part of the study. They did not however include reporting. Thus they were not part of the tally of stories, posts, tweets or other content that compared reporting with new information vs. reporting that was repetitive. They fell into a different of commentary or argument, and that content was examined. Nothing was “excluded.”

This is an argument over semantics. Serena’s blog did include reporting, as you will see if you read on. She was excluded from the quantitative part of the study and she belonged there.

Before I address the bigger issues, one point of clarification on the note’s quote from Serena (which I italicized above), which was taken out of context from a blog post this week: She was not validating the entire study, she was commenting on a particular statistic in the study, that 94 percent of media coverage of the Senator’s sale was driven by official government sources. That was the “conclusion” she referred to.

I’ll tell you how much Serena contributed to the news ecosystem on this particular story shortly, but first, I have to address the central point made by both Rosenstiel and the staff: that they deliberately excluded a blogger who actively engaged on the topic because she was an advocate. She had a point of view, so they couldn’t even consider her part of the study.

Well, what was the study about? I don’t want to mischaracterize it, so let’s use Pew’s title: “A Study of the News Ecosystem of One American City.” And, presuming that these are good journalists and/or academics conducting the study (I can’t confirm that, since the study didn’t identify them and Rosenstiel wouldn’t), I’m going to presume that their first couple paragraphs of the overview will tell us the point and purpose of the study:

Where does the news come from in today’s changing media?

Who really reports the news that most people get about their communities? What role do new media, blogs and specialty news sites now play?

And for good measure, let’s add the headline over the lead graphic (which plays up that bogus finding that 95 percent of new information comes from old media, mostly newspapers): “Who Reported New Information.”

The third question posed in the opening paragraphs of the study — “What role do new media, blogs and specialty news sites now play?” — purports to include blogs, when in fact it includes only a narrowly defined segment of blogs. And part of the answer to the first two questions might well be advocacy blogs. The answer to “who reported new information” in part is advocacy blogs (or if it’s not, we don’t know, because Pew excluded them from the study). And I guarantee you (and will shortly show you evidence) that astrogirl was part of the “news ecosystem” in one of the issues examined by Pew. Her exclusion reveals a bias so profound that I cannot give this study a shred of credibility. I take back what I said Monday about it having some value.

This decision reveals researchers studying the ecosystem the way they wish it was or the way they think it is, not the way it is. Rosenstiel and the Pew researchers took their snapshot of the news ecosystem through the lens of old media, deliberately cropping out key participants. Bluntly, they don’t understand new media well enough to study it credibly.

Here’s why it’s absurd to exclude an advocacy group from a study of the “news ecosystem”: Imagine a study of the national news ecosystem that excluded Fox News, talk radio and Comedy Central. What credibility would that have?

Yes, it galls journalists that polls show lots of consumers regard The Daily Show as a news source. But it is. Beyond the fact that many consumers get national news from the parodies on the show, it does break actual news that other outlets parrot (exactly the news dynamic that Pew documented). Jon Stewart’s interview with Jim Cramer produced news. It was Jon Stewart (actually, I’m pretty sure it was his staff), not the non-comedy sections of the national news ecosystem, who caught Sean Hannity using bogus footage to make a protest look like a bigger deal than it really was.

And Fox reports new information, even if it’s loaded with advocates. Lots of people get their news from talk radio, even if it might not report anything new (and makes news primarily through outrageous statements).

For better or worse, those are all parts of the national news ecosystem, and a study of “where does news come from” would lack credibility if it ignored them.

Here I need to repeat a telling passage in Rosenstiel’s email: “She was an interesting voice on the subject of the theater, but she was not a news gathering outlet, and it was the ecosystem of news we were studying here. A different study, looking at the robust debate that occurs in response to news gathering, would include many such voices.”

This reveals the old-media mindset of separating news from opinion, the news staff from the editorial-page staff (a valid position under which I have worked most of my career; as I noted earlier this week, I come from the same old-media background as Rosenstiel). But Serena was gathering news. She posted videos of news events. She investigated circumstances behind a news event. She published an exclusive interview with a key figure she said had not been talking to the media. Those are news-gathering functions that resemble the work of all the outlets Pew surveyed (and, as Pew noted, the new-media sources it surveyed, many of those didn’t truly gather news, just recycled news gathered by others). By excluding the “robust debate” from the study of the news ecosystem, the Pew study failed to measure the debate’s contribution to that system.

What distinguished Serena from the outlets Pew studied was her involvement and advocacy, not that she didn’t gather news.

Since Pew didn’t examine Laura Serena’s place in the ecosystem, I did (no speculation this time; I did some research, reading all her July posts). Yes, Serena is an advocate. Everything the Pew researchers said about her is true: She volunteers at the Senator, protested its sale and is deeply involved in the issue she writes about. She does not consider herself a journalist. She wrote in one post the week of the flawed Pew study that a Baltimore Sun story “mentions my blog, too. And at least Chris Kaltenbach didn’t call me a journalist, which I thank him for. (Today’s journalism seems to usually be nothing more than trying to pretend to be neutral, while pushing an agenda. I prefer to be called a writer. I don’t pretend to be neutral.)”

Here’s why you can’t exclude her from a study of the news ecosystem on this particular story, though: She was a news source for some in the community. She was gathering and reporting information about the sale and other media were picking up on things she wrote (I can’t tell whether they were citing her opinions or repeating news that she reported, but anyone studying the news ecosystem on this story should be able to tell you). In addition to that citation in the Sun, Serena mentions twice that her blog was mentioned on radio shows. And she was mentioned in a post by the Baltimore Brew blog, quoted in the Pew study (that post includes a photo of Serena, identifying her as The Senator’s “greatest groupie”). That’s at least four mentions elsewhere in the ecosystem. She was not a tree falling in the forest with no one noticing. Astrogirl made a sound and Pew refused to listen.

These mentions by other media tell you two important things:

• The Sun and other players in the news ecosystem considered Serena part of the system.
• The Sun was quoting at least one blog that the Pew report failed to mention. The Pew report mentions only one instance in all six story lines examined where a blog got to a story before old media, and that instance was addressed dismissively, noting that the story only became a big deal when the Sun weighed in. The central conclusion of the report, the one trumpeted by old media, was that all new information comes from old media, primarily newspapers. But now that we know the study deliberately excluded another time when the Sun cited a blog, my question is how many other instances did the study exclude? On a link from astrogirl, I found another blog, the Friends of The Senator, which posted five times during the week studied by Pew. Was that blog also a source for the Sun, but excluded by the biased researchers? Please note that I’m looking at only one of six story lines, and not even digging into that very deep (I’m a thousand miles away, six months after the fact). How many other news sources did Pew exclude, knowingly or by deciding not to look for them?

The Pew study, by examining the whole news ecosystem, rather than taking a narrow view, could have provided a valuable snapshot of the role of advocates play in providing information to the community. Serena wrote in her comment on my blog earlier this week: “People who carefully read through my blog will find that I have documented many facts that were never reported in the rest of the local media on this story. … Did I report original, truthful information that nobody else had bothered to find out? Absolutely.” I don’t know how true her claim is, but I do know that if Pew didn’t examine her contributions, its assessment of the local media is worthless. And what if she is overlooking some instances when she was first with some facts that Pew attributed to media who followed her lead? Then Pew was inaccurate — backwards — on the very recycling phenomenon it purported to measure. I’m speculating here again. I don’t know whether the traditional media cited other bloggers that the study ignored. But I do not claim to have studied the news ecosystem’s coverage of this story.

Let me also note that Rosenstiel and his Pew colleagues were not as strict as their response to me sounds. Rosenstiel justifies the exclusion because Serena “does not  represent herself, nor does she function, as a news organization.” The staff echoes that, noting she didn’t fit the “general interest, public affairs news outlets” studied.

But one of those “news organizations” was the Twitter feed of the Baltimore Police Department. (I noted in my earlier post that the study ignored the news information reported by citizen Twitter feeds.) The police Twitter feed does not represent itself, nor does it function, as a news organization. It represents itself as the “official site of the Baltimore Police Department.” Think about that: A citizen with a point of view is dismissed as not worthy of this study, but a government news release source makes the cut.

Since the Pew folks dismissed Serena’s blog, I took a look at it. The Pew study noted how inaccurately the Baltimore media predicted what would happen with the sale of the theater. One of the four “lessons” from the coverage was “The Press Proved a Poor Predictor.” In a July 29 post, Serena documented with several links to earlier posts how accurately she had predicted several developments in the story.

In the week Pew studied, Serena posted three videos relating to the sale. These were videos she shot, clearly original work. Two posts provide first-hand accounts of the sale (I don’t know how much she had that other media didn’t, but it doesn’t appear that she was recycling their information).

In a July 19 post, she says the Sun “finally acknowledged today that there’s another side to the story.” Again, I can’t verify that the Sun was ignoring this side of the story. But if she’s right, some Sun content that week was recycling information already reported in a blog, exactly the reverse of what Pew reported: “Print outlets drove this narrative.” The study’s graphic reports no new-media stories with significant new information.

Earlier in the month, Serena published what she claims to be an exclusive interview with the Senator’s owner, Tom Kiefaber. Unlimited by the cost of newsprint or the need to appeal to a broad audience with varied interests, she published it as a Q&A, far longer than a newspaper would publish on such a topic. Serena is a friend and defender of Kiefaber, a relationship that troubles journalistic sensibilities, and I would certainly support a study of the differences in news reported by neutral journalists and blogging advocates. I don’t know whether other media used some quotes from the interview (Serena indicates Kiefaber had not been talking to the media), or whether they were able to get their own interviews after she published hers, but in either case, they would have been following her. At the least, she was reporting new information. There’s no question she was providing news about a central figure in this news story, even if that news came through her unabashed avocate’s lens.

I should note that Serena on several occasions cites, quotes and links to stories from the Sun, so I think it’s reasonable to presume that content she presents as original truly is.

A valid study of the news ecosystem would include the contributions of all news sources: traditional media, bloggers functioning as news organizations, non-journalist bloggers gathering and publishing information to support their points of view, citizen Twitter feeds and official sources reporting directly to the public (such as the police Twitter feed).

In a follow-up email, Rosenstiel told me, “I could give you a 10 page methodological statement on why you need to have objective replicable parameters and rules for what is included and not included. But the point is, it wasn’t missed.”

Yes, it was missed. The fact that is was missed deliberately changes it from an oversight to a deliberate exclusion reflecting bias. A 100-page methodological statement wouldn’t make that right. I agree on the need for academic rigor in a study. But I don’t accept academic rigor as an excuse for excluding part of the very ecosystem you purport to be studying. Astrogirl was part of the news ecosystem for this story, and a study that ignored part of the news ecosystem has no value or validity.

I have long respected PEJ’s contributions to journalism (a few years ago, PEJ used several of my handouts for journalism workshops on its web site). A valid study of the news ecosystem in a community would be a valuable contribution to the industry. I hope PEJ does more research. And I hope they get it right this time.

Advertisement