It is the most ominous number on the Internet: 404.

Error, file not found.

When you’re one man behind one monitor, a 404 message telegraphs little. The site won’t open. But why? Is it under construction? Or are you being blocked?

Thanks to HerdictWeb, beached surfers can find out if other users are also unable to access an unavailable site. An exercise in crowdsourcing, HerdictWeb asks Internet users around the world to report which sites they cannot access. Others in the herd then report if they can see the same site.

When a group of users in one location cannot access a site, HerdictWeb staff can usually always figure out why a site is unavailable, project manager Vandana Aneja said.
Sometimes it’s a server error; sometimes a government has arranged for the site to be blocked within its borders.

Herdict is one of the first concentrated efforts to compile real time accessibility data. The project is Internet guru Jonathan Zittrain’s http://futureoftheinternet.org/> vision, borne out of the OpenNet Initiative.

“At this point the goal is to try and document this information because this information is not available anywhere on the web,” Aneja said. “There’s no way of knowing what websites are available where, for how long they’ve been down, for what reason they’ve been taken down and what people are experiencing in terms of availability of websites in several locations. This data is very interesting and useful in places that actually do censorship.”

HerdictWeb is, to be sure, not an advocacy site. For now, Aneja said, HerdictWeb is a place Internet users can go to document accessibility problems.

The data it gathers, though, has been cited by advocacy groups and journalists. The Wall Street Journal cited Herdict when reporting that YouTube was being blocked in China.

A core staff of six manages Herdict, which has been live since 25 February. It took about nine months for developers to create the site, which includes a game-like interface in which users are shown websites other Herdict reported to be inaccessible. It’s easy to become captivated clicking through to see whether or not sites load. Users may also comment about why sites might not be loading.

This comment section has been useful in solving cases of misinformation. A German website was reported as unavailable by many non-German speaking web surfers until someone in the Herdict community translated the error message on the site, which said (auf Deutsch) that the site was under construction.

Aneja said HerdictWeb, which is in beta, will add more ways for members of its herd to directly communicate with each other. It will launch a Twitter stream next week.

Herdict has already launched a Firefox button so users can quickly report on inaccessible sites without spending time at Herdict.org.

Vandana said the button, also recently made available for Internet Explorer, has been downloaded about 500 times so far.

So far, she said, the most interesting data has come from Saudi Arabia, China and Bahrain.

Since Herdict’s Arabic version launched in early April (HerdictWeb will soon also be available in Chinese, and someday perhaps in Russian), many Saudi users have logged on to Herdict to report inaccessible sites. According to HerdictWeb’s blog, this indicates trends in filtering within the country.

Aneja said for now, Herdict staff rely entirely on the crowd for information about blocked sites. There is little cross-checking on the part of the staff, making Herdict a true crowdsourcing effort.

“The more volume of reports we have the better picture we form,” Aneja said. “That’s the power of the herd.”

Published: April 29, 2009