Search Engines and The Data Explosion

Throughout all of human history up until 2003, we created 5 exabytes of data (five billion gigabytes). We now create that much every day. In 2011, we’ll create 1.8 zettabytes of data (a zettabyte is a 1000 exabytes). That’s up from 1.2 zettabytes in 2010, and some have predicted that we’ll be creating over 20 times that by 2020.

In trying to scale to meet this data explosion, our search engines are getting creaky.

While Google has been busy working on building Google+ as its social tool, Microsoft has quietly gone out and cut partnership deals with Facebook and Twitter and started integrating their social data into Bing search results. For example, if you do a search on Bing and you’re logged into Facebook in the same browser then the search results will show which of your friends have liked a certain page. See the example below:



Bing may have a leg up on Google in social today because of the Microsoft deals with Facebook and Twitter, but you also have to keep in mind that Google is going to have more control over its social-search destiny by building its own product. It won’t have to worry about partnership deals going bad or having to ask its social partners for additional API access. Google can just make it happen.

HTTPS, Certificates and Web Security Reconsidered

The Comodo Group, an Internet security company, has been attacked in the last month by a talkative and professed patriotic Iranian hacker who infiltrated several of the company’s partners and used them to threaten the security of myriad big-name Web sites.

But the case is not a problem for only Comodo, which initially believed the attack was the work of the Iranian government. It has also cast a spotlight on the global system that supposedly secures communications and commerce on the Web.

The encryption used by many Web sites to prevent eavesdropping on their interactions with visitors is not very secure. This technology is in use when Web addresses start with “https” (in which “s” stands for secure) and a closed lock icon appears on Web browsers. These sites rely on third-party organizations, like Comodo, to provide “certificates” that guarantee sites’ authenticity to Web browsers.

But many security experts say the problems start with the proliferation of organizations permitted to issue certificates. Browser makers like Microsoft, Mozilla, Google and Apple have authorized a large and growing number of entities around the world — both private companies and government bodies — to create them. Many private “certificate authorities” have, in turn, worked with resellers and deputized other unknown companies to issue certificates in a “chain of trust” that now involves many hundreds of players, any of which may in fact be a weak link.

The Electronic Frontier Foundation, an online civil liberties group, has explored the Internet in an attempt to map this nebulous system. As of December, 676 organizations were signing certificates, it found. Other security experts suspect that the scan missed many and that the number is much higher.

Making matters worse, entities that issue certificates, though required to seek authorization from site owners, can technically issue certificates for any Web site. This means that governments that control certificate authorities and hackers who break into their systems can issue certificates for any site at will.

Experts say that both the certificate system and the technology it employs have long been in need of an overhaul, but that the technology industry has not been able to muster the will to do it. “It hasn’t been perceived to be a big enough problem that needs to be fixed,” said Stephen Schultze, associate director of the Center for Information Technology Policy at Princeton. “This is a wake-up call. This is a small leak that is evidence of a much more fundamental structural problem.”

In the Comodo case, the hacker infiltrated an Italian computer reseller and used its access to Comodo’s systems to automatically create certificates for Web sites operated by Google, Yahoo, Microsoft, Skype and Mozilla. With the certificates, the hacker could set up servers that appear to work for those sites and try to view the unscrambled e-mail of millions of people, experts say. Comodo says it has suspended the Italian reseller and a second European reseller that the hacker also infiltrated.

In a series of online messages teeming with bravado, the hacker described himself as a software-engineering student and cryptography expert and said he worked alone. He suggested he was avenging the Stuxnex computer worm, which was directed at Iranian nuclear installations last year. And he indicated that he intended to use the certificates he created to snoop on opponents of the Iranian regime. “As I live, you don’t have privacy in Internet, you don’t have security in digital world,” he warned.

The certificate system was created at the dawn of e-commerce in the early 1990s before security was a major issue. Security experts say the system is not up to the challenge of today’s immense, commercial and much-attacked Internet. It was designed primarily to let businesses take credit card payments online, and less to confirm the authenticity of Web sites.

The crucial tool available to Comodo and the browser makers — revocation — is ineffective, security experts say. After the Comodo case, Google, Mozilla and Microsoft rushed out patches so their browsers would recognize and reject the bad certificates. But this solution requires many millions of Internet users to update their browser software, which many people never do.

Moreover, because certificate authorities’ servers are seen as unreliable, most browser makers allow users to proceed to an alternative site, and hackers can exploit this weakness, security experts say.

Browser makers have another problem: Faced with a suspicious certificate authority, there is little they can do shy of rescinding it. But if they did that, millions of Web users might encounter troubling error warnings when they visited sites with certificates from that company, causing a cascade of problems for users and site owners. Cutting out a large player like Comodo, which controls at least 95,100 active certificates, could effectively “break the Web,” said Dan Kaminsky, chief scientist at the security firm DKH.

They are effectively “too big to fail,” said Christopher Soghoian, a former Federal Trade Commission technologist who is now a graduate fellow at the Center for Applied Cybersecurity Research at Indiana University. “The problem is that the browser vendors don’t have a small stick, they only have a big stick." He said he could not recall a single instance in which the browser vendors had rejected a certificate authority.

Microsoft and Mozilla said that they would consider removing certificate authority if it was in the best interest of Internet users, and that they remained in talks with Comodo about its security practices. “Participation in Mozilla’s root program is a privilege, not a right,” the company, the nonprofit maker of Firefox, said. Apple, maker of the Safari browser, declined to comment. (Google’s Chrome browser defers to the choices of operating system makers like Microsoft and Apple about which certificate authorities are accepted.)

Mozilla, Microsoft and Google said they would work together and with certificate authorities and the security community on improvements to the system. One approach proposed by Comodo and Google engineers in January would allow Web site owners to specify which certificate authorities may issue certificates for their sites.

An initiative preferred by security experts would overhaul the system more radically. It would give Web sites similar control while securing their certificates within a new encrypted version of the domain name system, the central directory of the Web, making it the de facto central certificate authority through which Web sites could generate their own certificates.

Website and Supermarket Optimization {there are similarities and differences}

The relatively new field of Website optimization uses specialties such as statistics, user experience testing, and cognitive psychology to get visitors to convert (i.e., do what you want them to do, once they've landed on your site). I talk about these topics in my recent article Statistical and Financial Considerations in Website Optimization. There's a link to it at in my selected bibliography at the bottom of this blog, for anyone who's interested.

The optimization of Websites and supermarkets are both data driven tasks and both have the same goal: to capture visitor/customer activities in your Website/store and transform data about these behaviors into actionable management information.

Before getting too caught up in the art and science of Website optimization, it might be useful to pause for a moment and review some of the widely-used practices for in-store marketing and layout. As you do so, try to see the rather conspicuous parallels between what we do (as outlined in my article) and what they do (as outlined below).

But, remember that unique differences exist between how internet and brick and mortar channels can and do make money. For example, search engines like Google, Yahoo and Sphere help Website publishers to find stories by aggregating links to newspaper websites and blogs. In so doing, they wrest ad dollars from them that they think should be theirs. Not to mention the fact that, in so doing, these Websites are taking copyrighted material. Option like this are clearly not available to brick and mortar stores.

Note: The Associated Press and its member newspapers will take legal action against Web sites that use newspaper articles without legal permission, the group said recently, in a clear shot at aggregators like Google.

From a consumers point of view, a supermarket is quite simple; Put what you want into your cart and go through the check-out. Behind the scenes though, psychology is used a lot to define what products and brands you buy in supermarkets. Stands are designed to catch your eye and the store layout is structured to maximize profit.

Eye level marketing

Generally speaking, the most expensive items with high profit margins are placed on shelves that are at shoppers' eye level. Statistics show that you are more likely to see them than the less profitable brands at the very top or near your feet.

Aisle order

Some customers, particularly men, tend to simply shop for what they want, walking down an aisle grabbing what they want, turning back and walking the way they came, this is called the 'Boomerang Effect'. In order to maximize shopper and produce contact time, markets therefore place major items and brands in the middle of aisles ensuring that from any direction the customer doesn't have to walk the farthest to reach them.

Product grouping

Items that complement each other are often found close together to entice you to buy more. You'll often find pasta sauces on the same display as a featured brand of pasta.

Food smells make you feel hungry

Another tactic supermarkets use is the smell of freshly baked bread coming from the in-store bakery. The smell of warm bread makes people feel hungry. When you feel hungry while shopping you are more likely to buy additional items. Most Supermarkets bake their bread early in the morning; however, to entice more customers, some have resorted to pumping out the smell of fresh baking bread to add to the illusion that it is constantly baked through the day.

Essentials at the back

Supermarkets hit upon the idea of placing the essentials, such as bread and milk, at the back of the store. This is in order to make people have to walk past the rest of the produce, and heighten the possibility of impulse buys, in order to get their necessities. (Changing rooms in clothes stores are almost always situated at the rear of the store.)

Attracting children

One American supermarket chain came up with the idea of drawing a hopscotch in the aisle next to the children's cereal in order to make the children play and thus pin Mom & Pop to a point where the children could hassle them for treats.

Irrational Pricing

Irrational pricing is putting the price of items at say 4.99 instead of 5. The reason offered for not instead rounding $4.99 to $5.00 is based on memory processing time. Rounding upward involves an additional decision compared with storing the first digits. Furthermore, due to the vast quantity of information available for consumers to process, the information on price must be stored in a very short interval. The cheapest way to do so, in memory and attention terms, is by storing the first digits. Therefore customers perceive to be getting a better deal than they in fact are.

Point Of Sale

While you are waiting to pay, retailers often install Point Of Sale displays, this is especially prevalent in Supermarkets who install racks of chocolate to tempt bored children waiting with their parents.

Shuffle and Time

Many stores have a policy of regularly rotating the stock. This happens especially in supermarkets where people regularly shop for the same items. The idea obviously is to confront customers with a variety of items aside from their regulars and encourage them to explore areas of the store they may not usually visit.

The longer customers spend in a store the more money they are likely to spend there. Therefore stores work to make sure customers have to spend the maximum amount of time in their stores, placing obstacles constantly in the way of efficient shopping.

Newer areas where a lot more research is needed

In the UK, the British buy almost two-thirds of their wine from supermarkets; and more than a third of all wines sold in America are purchased at grocery stores, even though only 33 of the 50 US states allow supermarkets to sell wine. Wine is now the largest supermarket category in New Zealand and supermarket sales represent around 60 per cent of total wine sales.

As with any product promotion, there is no 'one solution' for supermarket wine departments. Unlike staples such as milk or eggs, wine is a luxury item (although many will beg to differ). This is the first hurdle for retailers. The difficulty of overcoming this hurdle varies from region to region. Additionally, retailers must decide how much effort and expense to invest in this part of their store. Decisions on these matters must also take into account the local competition, be it a wholesaler or a new chic wine boutique. First and foremost, know your customers.

At the end of the day, basic marketing principles will sell more wine than the most experienced supermarket wine steward. The longer customers stay in the wine department, the more likely they are to make a purchase. That can be encouraged by a tasting, music, or warm lighting. One Piggly Wiggly supermarket in Wisconsin features an expansive wine and spirits department that replicates a speciality wine cellar complete with wood shelving with a library ladder and an extensive walk-in cooler. Music can help in more specialist sales settings. Research at Leicester University showed that French music played in a supermarket's wine aisle boosted sales of French wines. The following day, German folk music led to German wines flying off the shelves.

Here, as in Website optimization, there's no substitute for subject matter expertise!