Building Big Data Solutions when you are not Google or Facebook
The powerful technology behind the capturing and processing of massive amount of data, also known as big data, has become very accessible to the world either by the popularity of open source software or by the aggressive marketing by the vendors. Any business event having “big data” in its title runs houseful which is a testimony to the success of marketing and plain curiosity of humankind about something that has a potential to change the world. Once curiosity is over then people ask what can we do with big data? Is big data a solution looking for the right problem which is yet to be found in my company? Or plain simple do really we have a big data problem?
Silicon Valley tech giants Google, Netflix and Facebook have successfully built business models around collecting data from hundreds of millions of web users and using it to create personalized products, such as online ads and services, such as book recommendations. The internet based user interactions, through web browsers, mobile apps and sensors, generate massive amounts of data, which these giants mine to create our friends lists, movies recommendations or which news site is interesting for us.
What can your company or you do with the help of big data? What do you need to get started?
Successfully solving a problem using big data requires three elements: 1. Technology, 2. Algorithms (Read: People with brains) and the last and the most important one 3. Big Data. Once you have access to all three then you can start solving serious problems with the help of big data.
There are two kind of problems that merit direct attention of big data problem solvers. 1. Creation of deeply personalized products and services that improve the lives of your customers 2. Creation of services that solve the problems of humanity which were impossible to solve due to astronomical cost of solution or the massive size of data. In other words, we have to think beyond the use cases such as pushing relevant ads and recommendations when we think about the technology that promises so much.
Getting big data is the most difficult part in big data projects even when data has to go from one department to another department with the same company. If the big data to solve a problem does not exist then you have to find ways to generate it or discover some unconventional sources for the data. Let’s take two examples where this has taken place:
The car insurance industry is making use of insurance telematics to generate new big data to analyze the driving behavior and using it to deliver deeply personalized service to their customers, so that they can correct their bad driving habits and save money. The customers of insurance companies have accepted such products without much fuss about the privacy.
In the developing world, the most ubiquitous computing device is mobile phone. Mobile network operators have access to the large silos of real time big data. The potential of this big data to solve the problems of humanity is beyond the mindset of these corporates. However, some researchers have got access to the mobile data from these corporate and used it effectively to control the spread of malaria in Kenya. Privacy and national concerns remain major hurdles in harnessing the big value hidden in the big data.
My upcoming talk at Big Data Inspiration Day at SAP Nederland
http://events.sap.com/nl-big-data-inspiration-day/nl/speakers?bc=2%1%0
Read More:
Insurance Telematics:
http://www.octotelematics.com/en/site/pages/Driving-behaviour
http://www.searchautoparts.com/motorage/electrical/exploring-connected-car?page=0,2
Big Data from Cheap Phones:
http://realitycommons.media.mit.edu/pdfs/hcii_txteagle.pdf
Aadhar Program in India:
http://www.informationweek.in/informationweek/interviews/271802/-aadhaar-example-source-technologies-extensive-driven-analytics-achieve-scale-quality