Monthly Archives: March 2011

Google listens… if you shout loudly enough

Yesterday evening I wrote about the bizarre disappearance from Google News of my news site, Berkeleyside. What happens next is either an illustration of the power of digital democracy or an example of the value of friends with fantastic megaphones.

Shortly after my post went up, a number of friends tweeted about it. Dave Winer and Felix Salmon were first off the mark. Felix’s rare use of an exclamation mark (“Berkeleyside axed from Google News!”) seems to have provoked some of his regular followers to retweet, including The New York Times’ David Carr, with over 300,000 followers.  My co-founders on Berkeleyside were doing their own tweeting and that reached plenty of others, from Exceptional Women in Publishing to the Knight Digital Media Center’s hyperlocal maven Michele Mclellan. Online media beacon Dan Gillmor added his voice and this morning local media/tech writer extraordinaire Scott Rosenberg piled on with a direct plea: “If you’re a Googler reading this, pls help!” (I’ve never liked exclamation marks as much as in the last 24 hours.)

I was doing more behind the scenes as well. After I wrote my post last night I emailed Josh Benton, who edits the wonderful Nieman Journalism Lab site at Harvard. Benton had published an earlier item I wrote reflecting on AOL/Patch and local sites. He assigned someone to look into it with Google.

Early this morning I had an email from an engineer on the Google News team. He said he was also a Berkeleyside reader and that, off the record, I should know that the problem was a bug and the Google News team was working to resolve it. No ETA for a solution, but he was happy to help me with a Google News backchannel. I’d earlier posted on a Google support forum thread that I reckoned the problem was a screw-up, not a conspiracy. I’m glad I was right.

At about 3pm Pacific time today we started reappearing in the Google News index. Megan Garber wrote an article on Nieman Lab explaining the kerfuffle. (I wish I’d thought of the headline: “Bug, not snub!” Again with the exclamation mark.) Google’s Jeannie Hornung told Garber, “Google News experienced technical difficulties that may have prevented the indexing of recent articles from some news sources. We believe all issues have been resolved. We apologize to our users and the sites affected.”

Phew.

What lessons can anyone draw from this? First, for all the ascendance of Twitter I’m convinced that my fuller, more reasoned explanation of what happened here on Davos Newbies was crucial. If I had 300,000 followers on Twitter perhaps 140 characters would have provoked someone to action. With less Twitter clout I think the fact that I actually marshaled my arguments in a more discursive form was crucial.

Second, it sure is great to have friends who have bigger megaphones, particularly in the media and tech worlds. The Google support forums where I was posting yesterday were filled with howls from media site owners who had been delisted. No one from Google was replying, so a lot of the forum posts were either grand conspiracy theorists or despairing writers wondering what was going on. From my back channel I’m sure that the Google News team was aware of the bug and was working away. But the cascade of protests on our behalf and the swift resolution today strikes me as more than a coincidence.

Third, it sure is helpful to be producing a vital news source in the Bay Area, where we’re almost bound to have readers who work at Google (and just about any other tech giant you can think of). If we were somewhere else in the world, perhaps we could have gathered the same forces in support, but I think the direct connection to at least one Googler who looks to us for news helped.

Local news: we're at Google's mercy

I spend the bulk of my time these days trying to figure out some of the future of journalism, with the local news site I started with two others, Berkeleyside. We’re unquestionably the leading news source for our city, and we’re widely recognized as such. The San Francisco Chronicle uses us to supply Berkeley news, we’re picked up occasionally by The New York Times, and we’re looked to by, for example, the local public radio station, for the latest on Berkeley.

So it’s not surprising that we’ve also been the prime supplier of Berkeley news to Google News. Until last Saturday.

For reasons as yet unknown, Google News stopped indexing our stories last Saturday. Nothing from us on Google News on the three arrests at Berkeley High today, a story we broke hours ahead of any other news source (although they index several sites that do nothing more than point to our story). Nothing from us on Google News on the power outages in Berkeley today, which was also our scoop.

Was it something we did? That was, of course, my first assumption. Maybe we broke something into our move to a new server, or perhaps our sitemap, the xml document that Google News uses, had been corrupted. A careful check of anything that might be our fault shows that isn’t the problem.

Google, of course, is famously opaque to users, even those that rely on the Mountain View company for parts of their livelihood. There’s no one to call and Google employees only rarely reply on the support forums. “Harvey P”, a Google employee who hasn’t created a Google Profile, did post a notice Attention Publishers: Site Removals on March February 14. But Harvey P implies that publishers removed from Google News would get a notification of removal. We’ve received nothing of the sort.

Scanning through the troubleshooting forum for Google News Publishers, it’s clear we’re not the only ones with this problem. Starting in the middle of last week, an anguished cry went up from sites wondering why Google News stopped indexing them.

Like any sensible search user, I’m delighted Google makes regular efforts to improve the results I use. I want them to get rid of sites that are purely aggregators, that publish nothing original, that really have little to do with “news”. But we meet every one of Google’s criteria for a news site.

Fortunately, there are many ways for people to find their way to Berkeleyside — the Chronicle is a firehose for traffic, Google search still indexes us, Twitter is wonderful, a nice number of people use our iPhone app, and we have a loyal following. But we’re a news site, damn it, and we want and expect to be indexed as such.