i’m not sure what the story with oodle’s crawling was, and perhaps there was a throttling issue to be corrected, however i n general, i seriously doubt the % of traffic from search engine crawlers is anything higher than low single-digits (if that) for any of the major portals / search engines out there.

re: content being valuable — i absolutely agree with you. however, being FINDABLE is also pretty important. and furthermore, building applications on top of structured data *does* deliver significant value, across a wide variety of verticals.

you (and craigslist and everyone) always have the right to opt-out of being indexed, but i would argue it’s likely not in most folks’ best interests to not be found on Google or Yahoo… similarly, it’s likely not in most classified listing publishers’ best interests to not be found on search engines that provide info on that domain (or in our case, job listings & job search engines).

like Google & Yahoo, vertical search engine sites like Simply Hired and Oodle (and others in real estate, travel, shopping, etc) deliver millions of users / page views in traffic to data publishers by sending them to sites that might not otherwise be found. if those sites / publishers don’t think that’s of value then it’s certainly their prerogative to remain hidden in the “dark web”, however it seems like a rather Luddite position to take on the matter.

walled garden approaches to data access either by data publishers or portals that host that data, will only result in their marginalization and accelerated irrelevance. aside from the original content owner & data publishers, there are very few of those sources that can be kept proprietary, or can remain so in the future with an open & competitive market for data access & transparency.

for more on the subject of “walled garden” vs “open access”, see our recent blog post:
http://blog.simplyhired.com/archives/2005/10/google_there_ca_1.php

- dave mcclure
http://www.simplyhired.com

I still think it is a high resource cost compared with what comes back. It isn’t so much the cost for the bandwith ($2 per GB) but the fact that the internet must be groaning under the weight of the search and scraper crawlers, especially as ever more come online.

My point is simple: srape and crawl is cheap, anyone can do it: content such as mine and those of other journalists is not cheap to produce. They need to offer something of greater value otherwise they are vulnerable to being turned away at the door…

+ + +

Indeed, Tom is probably only acknowledging the tip of the iceberg in terms of search. The 3.7% number is actually from the “hits” column. (Hits?) A more accurate analysis, for the purposes of judging the true value of search, would involves tagging every visitor who was introduced to the site via a search engine, and all subsequent visits, as search traffic, IMHO. Then there are people who found your site via a search engine, then told a friend, and so on…