I spend the bulk of my time these days trying to figure out some of the future of journalism, with the local news site I started with two others, Berkeleyside. We’re unquestionably the leading news source for our city, and we’re widely recognized as such. The San Francisco Chronicle uses us to supply Berkeley news, we’re picked up occasionally by The New York Times, and we’re looked to by, for example, the local public radio station, for the latest on Berkeley.
So it’s not surprising that we’ve also been the prime supplier of Berkeley news to Google News. Until last Saturday.
For reasons as yet unknown, Google News stopped indexing our stories last Saturday. Nothing from us on Google News on the three arrests at Berkeley High today, a story we broke hours ahead of any other news source (although they index several sites that do nothing more than point to our story). Nothing from us on Google News on the power outages in Berkeley today, which was also our scoop.
Was it something we did? That was, of course, my first assumption. Maybe we broke something into our move to a new server, or perhaps our sitemap, the xml document that Google News uses, had been corrupted. A careful check of anything that might be our fault shows that isn’t the problem.
Google, of course, is famously opaque to users, even those that rely on the Mountain View company for parts of their livelihood. There’s no one to call and Google employees only rarely reply on the support forums. “Harvey P”, a Google employee who hasn’t created a Google Profile, did post a notice Attention Publishers: Site Removals on
March February 14. But Harvey P implies that publishers removed from Google News would get a notification of removal. We’ve received nothing of the sort.
Scanning through the troubleshooting forum for Google News Publishers, it’s clear we’re not the only ones with this problem. Starting in the middle of last week, an anguished cry went up from sites wondering why Google News stopped indexing them.
Like any sensible search user, I’m delighted Google makes regular efforts to improve the results I use. I want them to get rid of sites that are purely aggregators, that publish nothing original, that really have little to do with “news”. But we meet every one of Google’s criteria for a news site.
Fortunately, there are many ways for people to find their way to Berkeleyside — the Chronicle is a firehose for traffic, Google search still indexes us, Twitter is wonderful, a nice number of people use our iPhone app, and we have a loyal following. But we’re a news site, damn it, and we want and expect to be indexed as such.