Recommendation engines: What's in Pandora's Box?

Every year I return from Christmas shopping trips saying the same thing: Never again! Why do I subject myself to the crowds, the queues and frustration when I can do the entire thing from the comfort of my keyboard? Yet every year I, like millions of others, masochistically return for more punishment.

This year I finally realised why.

While ruthlessly efficient and valuable, shopping on Amazon.com is a relatively sterile experience. Amazon's collaborative filtering recommendation engine reportedly results in up to 30 per cent of its revenue. What you've bought in the past, what's in your cart, items you've rated, and what other customers have purchased determines the recommendations.

Amazon has integrated recommendations into nearly every part of the purchasing process from initial browsing to checkout, and of course has an analytics-driven email marketing program which the company claims has higher converson rates than on-site recommendations. All great when you know what you're looking for, or responding to Amazon's recommendations in your known areas of interest. Not so good when shopping for gifts or looking for something new and inspiring.

Buying books from Amazon is like preaching to the converted. I really don't need more books explaining the global financial crisis, pop psychology or how close we're coming to destroying our planet through our rapacious consumption.

Much is written about the power of recommendation engines and their ability to know you better than the small retailer with whom you once had a relationship, because they use vast stores of data and predictive models which a human could never draw upon. But this simply misses the point. Amazon might know what I like and things I'm likely to buy, but it definitely does not know what is good for me.

Cosmos Music and Pipe Records in Melbourne were the best 'recommender systems' I have experienced to this day—not because they gave me more of what I liked, but because they took me out of my comfort zone, presented me with something challenging and confronting, and often opened up a whole new world of experience or knowledge for me. I'm not sure how you codify this, but it's certainly not 'people who bought that also bought this'. The best retailers have more in common with art and museum curators, and this is some of what machine-based recommendation engines don't provide online. And that is why the experience is so sterile.

Having failed to find online a single thing my wife needed for Christmas, or venturing into areas of personal taste that I dare not go, I traded my empty online experience for Hong Kong's Hollywood Road, wherein I did manage to find, completely by accident, the perfect gift (an antique Japanese jewellery box to house the previous five birthday and Christmas gifts). And this is the other component missing from the online experience: serendipity.

It's a beautiful word, derived from the Persian word Serendip, meaning Sri Lanka, and having its origins in an old fairy tale about three princes of Serendip. The term means the fortunate discovery of something useful by accident.

Many inventions and scientific discoveries were acts of serendipity. Penicillin was discovered by Fleming accidentally allowing fungus to grow on a bacterial culture, velcro by observing the natural hooks on burdock seeds and how tricky they are to remove from a dog's fur coat. The great offline retailers like Walmart and Ikea build the opportunity for serendipity into their store experiences; hence there is only one way to go through an Ikea store and you can't buy online (unless you live in Macau). You're forced to look at everything.

Part of the early appeal of group buying sites was the serendipity of finding a deal on a product or service you would not normally buy. The most serendipitous online retail site in my opinion was Ebay, but only in its early days. Stumbling across something unique and cool was part of the experience, not simply searching for the lowest price. As it grew in volume of auctions, it lost a lot of its serendipity factor, though the eBay R&D department is reportedly hard at work trying to recreate this experience through intense modelling of its huge data repository, and a refocus on UX design.

The question is, can you create serendipity (or at least the appearance of it) through data modelling?

The closest I've seen any site come to this is the online music discovery service Pandora. Unlike other services such as Spotify, Rhapsody or iTunes Genius, which are heavily based on genre and collaborative filters, Pandora has no concepty of genre, user connections or ratings. It won't recommend any other user's favourite music, just because you share an interest in Jose Gonzalez.

Pandora employs a huge number of trained music analysts who have codified hundreds of thousands of songs from more than 10,000 artists against 400 musical attributes such as rhythm, tempo, chromatic harmony, complexity and type of lyrics. This underlying database, known as the Music Genome project, is designed to take advantage of the Long Tail of musical choice in a digital world, where less than 2 per cent of all music releases make 80 per cent of the revenue. In other words, it should drive sales toward to the more obscure, less popular, but much deeper catalogue. And it works really well.

Even though marketing researchers like Dan Ariely have shown we often behave in ways that are Predictably Irrational, I don't believe all human behaviours can be modelled, no matter how many parameters we measure and how many data points we capture. My hope, like the last thing to be released from Pandora's Box, is that we find better ways of allowing diversity and serendipity to permeate our screen-based life to create an ultimately more satisfying and rewarding experience.

John Merakovsky can be followed on Twitter @Experian.