ISP Clickstreams
I have to agree with Henry and say the most interesting thing I learnt at Seth Goldstein’s Open Data conference was that the market for ISP proxy logs is fairly well developed, with standard pricing and about a dozen licensees.
David Cancel, a co-founder of Compete.com, is one of them. David says the value is around 40 cents per user per month. Compete.com sources 5m unique non-identifiable click-streams and scrubs them down into 2 million that are roughly representative of the US population. The $1m purchase is done by roughly 12-14 firms at the moment he said.
Another company that rode the ISP proxy logs availability is Hitwise. They found most success when extracting keyword data from the url strings and using that as a center piece for anchoring the data.
Another great speaker on the day was Tony Berkman, who co-founded Majestic Research. I didn’t realize but they are doing great things indexing the web and extracting investable information that plug into earnings prediction models. They also track things like casino foot traffic through third-parties. He said that Hedge funds are starting to do their own things around crawling too.
I imagine it’s not far from the point of Hedge Funds being able to justify the costs of ISP proxy log data too. The intersection of large scale web crawling and equity research fascinates me. If anyone who is working in the area is reading this post, I’d love to hear from you (niki dot scevak at gmail)
