Search Engines and The Data Explosion

Throughout all of human history up until 2003, we created 5 exabytes of data (five billion gigabytes). We now create that much every day. In 2011, we’ll create 1.8 zettabytes of data (a zettabyte is a 1000 exabytes). That’s up from 1.2 zettabytes in 2010, and some have predicted that we’ll be creating over 20 times that by 2020.

In trying to scale to meet this data explosion, our search engines are getting creaky.

While Google has been busy working on building Google+ as its social tool, Microsoft has quietly gone out and cut partnership deals with Facebook and Twitter and started integrating their social data into Bing search results. For example, if you do a search on Bing and you’re logged into Facebook in the same browser then the search results will show which of your friends have liked a certain page. See the example below:



Bing may have a leg up on Google in social today because of the Microsoft deals with Facebook and Twitter, but you also have to keep in mind that Google is going to have more control over its social-search destiny by building its own product. It won’t have to worry about partnership deals going bad or having to ask its social partners for additional API access. Google can just make it happen.