A New View of the Web

This blog post by Rand Fink, the CEO of SEOmoz, an SEO and social monitoring startup, presents an interesting view of the web.  Essentially, the author claims two forces govern most of the successful Internet sites, "The Crowd and The Algorithm."  The crowd refers to the collective users of the Internet, who make all of the data, which can then by exploited by sites that use algorithms to make a market out of the data.  This is an interesting perspective on what we study in class as "exploiting network structure."  We see this everywhere:  Google's PageRank exploits the underlying structure of the network, as well as data from past searches and clicks, to present its users with the information they want.  Facebook gives personalized ads, as we will understand better first-hand with Clickmaniac, by using the information of its users.  Netflix uses tons of data and a lot of machine learning to recommend its users movies they would enjoy.  (The blog gives a more articulate and extensive list.)  I like this way of thinking about exploiting network structure, because it really looks at the parts--the exploitation (the algorithm) and network structure (the crowd)--separately.  Better understanding the crowd makes it easier to develop algorithms to take advantage of what it's doing.

But then the author shows a side to this that one may have not considered before.  Services similar to Netflix, Amazon, Yelp, Expedia etc. existed before, although they were in many cases much more difficult to access, and much less personalized (since they couldn't take advantage of network structure).  However, these older services had the advantage of not hiding their methodologies.  Video rental companies like Blockbuster just reccommended popular movies, Consumer Reports would use knowledge from experts to present viewers with the "best products" etc.  (Don't worry I didn't know what Consumer Reports was either).  But now, we don't really completely understand how these new online companies work.  We don't know the details of how Google ranks its pages, and we definitely don't know how Bing ranks images.  The best example given in the blog is: "Why did Amazon recommend buying whole milk with this Badonkadonk Land Cruiser?"  This may not seem like a problem, but the author brings up a good point that we like to use services we understand, and perhaps more importantly investors should think twice before investing in a company that hides its key marketing techniques.  Furthermore, the author points us to new companies that are trying to go back to the older styles of marketing.  For example Quora uses experts, people we can trust in ranking Q&A's, in addition to the crowd and algorithms, and my favorite is that sites like Groupon crowd-source their marketing  (who hasn't learned about a deal from their friends rather than by going on the website itself?) making much of their success based on word-of-mouth, a phenomenon we were used to before the Internet, even though it uses network structure just like many of the newer marketing techniques.

A question one could ask is whether exploiting data from the crowd to form a market is ethical.  In one sense, Google and Facebook are taking advantage of work billions of people are doing (creating web graphs or adding friends), and making millions of dollars out of it.  But why shouldn't that be the case.  Everyone is benefiting from it.  Users want to see more relevant ads rather than irrelevant ones, advertisers clearly want to go through companies like Google and Facebook, and of course the big corporations are happy on their end too.  Hey, they can even try to help make the world a better place by having side projects like Google X.  In some ways, this new trend of making a market out of network structure and massive amounts of data is similar to price discrimination.  Just like how kids pay less at Souplantation, because they're probably going to eat less, Google can choose to show us ads it thinks we'll like more.  But in an obvious way, it's almost the opposite.  Price discrimination takes advantage of knowing its users to make things worse for its users (make its users pay as much as they are willing), whereas online companies take advantage of knowing their users to provide them better services.  I would much rather give my information to a website that will give me better movie recommendations, than a company that will charge me more because it knows my willingness to pay.  I think the fundamental difference is that the people paying for these online companies are completely different from the users.  The people paying to make Google, Facebook, and Amazon work are companies that benefit* from the presence of Google and Facebook.  I like the fact that Google, Facebook, and the like are using our information to make things better for us, and to make them get more money but from a completely different source.  I don't really mind if I don't understand how their algorithms work (from a user perspective; I do mind from a researcher's perspective).  I don't really mind if they're using my information to my benefit.  It's a whole new kind of economics, I just hope they don't abuse it...

*That's not completely true, because a company may feel compelled to advertise online just because its competitors are doing it.