Online intelligence is a relatively new field. How did you become involved in the industry and what has your career path been up to this point?

Bill Tancer: My career path has taken some wild turns. I started my career as a prosecutor in the U.S. Navy JAG Corps. After being transferred to the San Francisco Bay Area in the early '90s, I became fascinated with online and, eventually, Internet services. When I left the Navy I changed career paths and focused on the Internet. Before landing at Hitwise, I worked at a small Bay Area ISP, Pacific Bell's Internet Service, the Gartner Group, NBC's Internet subsidiary, and then, search engine LookSmart.

I first discovered Hitwise and the field of online competitive intelligence while working at LookSmart, a client of Hitwise's. I was fascinated with the amount of observed data that was being captured and the potential to innovate not only online marketing but also our understanding of ourselves.

Out of the massive amounts of data you've collected and analyzed, how did you decide what to put in the book? Why did you organize the book in the way you did?

BT: That was one of the difficult tasks of writing Click. In the process of working for Hitwise over the last 4 and a half years, our research team has unearthed countless fascinating findings. The process of determining which stories to include entailed first deciding on some main themes and then choosing only those stories that best illustrated those themes.

Is there one finding in your years at Hitwise that strikes you as the most interesting or unusual trend you've discovered?

BT: Within the search industry and online marketing circles I'm best known for my ability to cite endless statistics about searches for "prom dresses." I can't claim the initial discovery of the significance of "prom dresses" queries—that finding was made by one of my analysts, LeeAnn Prescott. But our study of the searches for "prom dresses" and their repeating yearly patterns led to our first understanding of the inefficiencies that are created when marketers act on hunch rather than relying on observed behavior.

It is a common misconception for people to view Hitwise as a "Big Brother"—someone spying on their every online move. How do you maintain Internet users' anonymity?

BT: End-user privacy is of utmost importance to Hitwise as well as to me as a researcher. The primary data collection methodology revolves around agreements that we have with ISPs to allow us to monitor Internet behavior at a network level. To ensure privacy, our data partners strip out all identifiable information. The data when it enters our database is aggregated (we don't see individual-level data) and anonymized (no trackable IP addresses or other identifiable information is transmitted to us).

Click is one of the first comprehensive looks at a cutting-edge digital field. Where do you think online intelligence is heading in the future, and how will it help us as a global society?

BT: From a business perspective, the ability to truly understand consumer behavior, motivations, interests, timing, and seasonality will allow businesses to build products and services that are more inline with our demands and to market them in the most efficient manner possible.

Are there still cases when it is better to collect data, conduct surveys, and run polls using non-digital methods? When and why?

BT: Absolutely. The biggest challenge with interpreting observed behavior is the art of understanding user intent behind visits to a website or searches on a particular term. For example, if a search for a presidential candidate shows up in our data, was the searcher interested in voting for that candidate, finding out more information about him or her, or instead looking for negative information to solidify his or her position against the candidate?

What role do you think Google plays in determining the data collected by Hitwise? Do you feel our movements online are somewhat streamlined by the popularity of Google, which can be seen as the gatekeeper to the Internet?

BT: Though we don't gather data from Google or any other search engine (we monitor searches at a network level), Google and other search engines are some of the most valuable catalysts for gathering information on consumer demands and interests. That said, the Internet is constantly evolving, with social networks such as Facebook and MySpace changing the way we communicate and navigate online. Today, search engines continue to be the closest thing to an Internet gatekeeper, given the state of hyper-innovation and adoption of new technologies; however, I'm not sure how sustainable that position will be in the future.

In your research, do you find major differences when you begin analyzing data from outside the United States? Do your findings reinforce or negate the typical stereotypes people hold for different countries around the world? Where are countries most similar? Where are they most different?

BT: There are distinct differences in search habits in the regions we cover (U.S., UK, Australia, New Zealand, Hong Kong, and Singapore), but one of the most remarkable phenomena we've noticed is actually the homogenization of our cultures. Take prom dresses, for example. Prom didn't exist in the UK until recently. We first noticed a surge in "prom" queries in the UK, and as it turns out, our search data predicted the influence that American television has had on English teenagers and their school-end rituals.

In the book, you talk about the difference between data arbitrage and making predictions. What is a good example of arbitrage versus prediction? How exact can arbitrage be, and, as your techniques become more refined, what sort of markets or activities do you think you will be able to arbitrage in the future?

BT: Arbitrage, in the context of my analysis, is simply getting access to data before it is publicly reported. For example, it takes the National Association of Realtors several weeks to gather home sale statistics from around the country; I can see the aggregate movement in visits to real estate sites and purchase related search terms with a much shorter delay. Predictions, on the other hand, require some form of calculation based on observed data (usually from relationships observed from past occurrences) to make an educated guess as to future results.

Given the lack of calculations necessary, data arbitrage is usually more reliable, but since even data arbitrage from observed behavior requires the art of intent-interpretation, it can be faulty as well. A longtime goal of mine (since starting at Hitwise) is to find a way to determine consumer sentiment based on Internet behavior. I don't think we are very far from getting there.

How do you think your findings will change business and marketing strategies?

BT: At a minimum, I think the concepts that we talk about in Click will help us to make more informed, strategic business decisions, discarding the inefficiencies arising from making decisions based on hunches rather than on observed consumer demand.

You say we are in the midst of Web 2.0 and that people are already talking about Web 3.0, a stage where you think a method is needed to filter consumer-generated content for similarity of viewpoint, reputation, and accuracy. How do you see Web 3.0 evolving?

BT: My view of Web 3.0 (and we are still debating the definition of Web 2.0) is that the explosion of available information, both good and bad, will require that we develop a new method to distinguish reliable from unreliable information.

Websites like Google, YouTube, and MySpace exploded in the online world due to Early Adopters and Super-Connectors. Do you have any organizations, individuals, or artists on your radar now for whom you can feel the online "tipping point" approaching? In other words, do you have any educated guesses as to what the next big online phenomenon will be?

BT: I'm currently in the process of applying the Early Adopter analysis that we talk about in Click to a number of different categories of sites, from multimedia to social networks, and even fashion.

Do you find any implications in the work you do? For example, political polls can skew voter decisions before they cast their vote (if they see their candidate is losing, they may decide not to "throw their vote away"). If your data arbitrage about the housing market, or even American Idol votes, is publicized, do you think it will change the outcome of the game? Do you see yourself as being able to control results in any way?

BT: I don't think we're anywhere near that point. I'd prefer to focus on providing the most insightful and accurate data to both marketers and consumers in hopes that, with all the data on the table, we will all be able to make smarter decisions. By the way, Blake Lewis should have won American Idol.

What's next for you?

BT: I can't help myself—I'm back to looking for more insightful online behavioral patterns to share.