We at Express Analytics often get this question: how do we (a business) go about setting up our Big Data analytics project?
Typically, we get this question from two types of businesses – those modernizing their data architectures and those that have never had traditional BI and data warehousing systems but want to jump straight onto the Big Data bandwagon.
Both situations pose certain challenges; some unique, some common to both categories. So we thought: why not spell things out in black and white in a blog post, which will serve as a primer for both categories.
What is Big Data?
Before going ahead with any analytics project, one must at least have a rudimentary understanding of what Big Data is.
That’s the right starting point, especially for those enterprises that want to get into the game straight away.
To be candid, the word “Big Data” may have been coined recently, but the act of collecting and analyzing data is decades old.
Simply put, Big Data refers to the large volumes of structured and unstructured data that flow into your business every hour and every day.
The now-industry-accepted definition is the three Vs:
Volume: Data today flows into an organization from numerous sources: websites, social media, mobile computing devices, sensors, and so on.
In earlier times, storing this humongous volume was a problem, but new technologies like Hadoop have now resolved it.
Velocity: Denotes the speed at which data flows in and must be processed.
With the advent of the Internet of Things (IoT), for example, data from sensors, RFID tags, or the Edge necessitates that businesses handle data in near real-time. Other businesses continue to handle data in batches.
Variety: Of course, like humans, not all data was made the same. Comes in different formats, sizes, and formats, unstructured, and so on.
The massive scale of Big Data brings challenges such as data storage, scalability over time, noise accumulation, false data correlations, and measurement errors, all of which require new computational and statistical models not available in traditional BI.
Want to Explore Our Business Intelligence and Analytics Services? >>> Speak to us
Now, having grasped the basic definition of Big Data, businesses have to understand that there’s only one fundamental thought running the capturing of that massive volume of data – what do you do with it?
Collecting data is one part of the operation, but the story does not end there. In fact, some may say it starts there.
Big Data Analytics needs to be analyzed for insights and checked for patterns. Why? Because that leads to better decision-making within your organization.
All of your enterprises’ decision-making is now on a scientific, evidence-based platform, not some random gut feeling or superficial market research.
Big data analysis offers interaction with data that was not possible with the earlier, traditional enterprise business intelligence systems.
But here’s the thing about Big Data – it cannot just be said of it that it will provide 360-degree on-screen enterprise-wide high-level insights; that would be oversimplifying.
According to this bunch of researchers, big data analytics projects have no single canonical use case or structure.
Instead, the applications that could benefit from analyzing Big Data cut across industries and involve a wide variety of data sources.
What does Big Data Analysis mean?
I shall simplify it – Big Data analysis does not throw up a single set of results to be implemented across the enterprise.
Also, it does not have one form throughout the organization.
Big Data analysis can take many forms and yield many outcomes across departments within the same organization.
Outputs may result in revenue gains, cost reductions, or risk mitigation.
It can be used in marketing and sales, Customer Relationship Management (CRM) initiatives, and brand monitoring, among other departments.
That’s why, before launching a Big Data Analytics project in your enterprise, specific steps have to be taken.
Because of the complexity arising from the variety and volume of data and its sophistication, project plans need to dovetail with the intended business outcomes.
One of the biggest challenges in this kind of project is to avoid working in silos, ‘cause that’s one of the main reasons for failure.
The fundamental difference between traditional Business Intelligence (BI) and Big Data analytics is that in the latter, an analyst or trained employee finds patterns in data and then answers questions that were never asked in the first place.
That’s how Big Data analysis differs from the traditional BI methods, where answers to questions were sought against structured “predefined” data sets.
We will pause here, but in part 2 of this post, we will discuss how best to get started with a Big Data project, and also the common pitfalls to avoid. So stay tuned.
