Cloud-based data warehouses have returned with renewed vigor. If one were to go back in time to the early ‘80s, when mainframe computers were relatively expensive pieces of hardware, companies would go in for “time-sharing”. This essentially meant that such companies would dial in at a scheduled time to those “remote” computers to get their work done. When PCs became cheap, this stopped.
Data warehousing in the Cloud is similar to the time-sharing concept, but far more advanced technologically. They are becoming increasingly relevant and prevalent in today’s Big Data world for many reasons.
One of them is the need to create such a hosted infrastructure to house that deluge of data pouring in literally every hour.
Traditional data warehouses are still around, but a dipstick survey will reveal that, over the last two years or so, Enterprises have started moving to cloud replacements. Some surveys by leading research agencies, too, confirm this development.
The question, thus, to be asked is – why?
An electronic data warehouse collects multi-channel data, and a company then analyzes it to support management decisions. Early versions of on-premise data warehouses were prohibitively expensive, took years to build, and, because of their monolithic architecture, were not agile.
All of this meant a time lag between data collection and analysis. This often forced IT departments to wrestle with the question: maintain data quality or sacrifice some of it for business agility?
There’s also the question of data warehouse architecture itself at the heart of this conversation of traditional versus the Cloud. To put it simply, conventional warehouses are built on a 3-layer structure, namely, bottom, middle, and top, with an accompanying OLAP (Online Analytical Processing) server.
Today’s Cloud-based data warehouses do not follow that kind of rigid architecture; they are much more flexible. In fact, some tools like Google’s BigQuery, a serverless data warehouse, allocate resources dynamically. Here, clients can upload data either from Google’s Cloud Storage or stream it in real-time.
BigQuery uses ‘Dremel’, which is a query execution engine using a columnar data structure. Dremel scans billions of rows of data using the Colossus file management system.
The latter distributes these files into 64 megabyte pieces among ‘nodes’, which are then grouped into clusters. Essentially, this is a tree architecture that sends queries to millions of machines.
The process of loading data into a warehouse itself has also got a counter today. The traditional method was ETL – Extract, Transform, Load – a tried-and-tested approach that’s been around for over 20 years. It was the only way the data world knew to process large volumes of data and prepare it for analysis.
But ETL’s supreme position was recently challenged by ELT, or the Extract, Load, Transform method.
Let’s quickly understand a fundamental difference:
In ETL or the traditional approach, data is extracted from multiple sources and stored in a temporary staging database. Here, data is structured and converted for the data warehouse. This is a very “structured” approach.
With the ELT method, data, once extracted, is straightaway loaded. The staging database level is eliminated. The data is instead transformed inside the data warehouse itself for analysis.
Each of them, ETL and ELT, comes with its own set of pros and cons. But Enterprises are leaning more and more toward ELT today, given the high costs of infrastructure. Plus, the ETL system often leads to cost overruns and scalability issues.
In fact, Enterprises are also increasingly leaning towards ELT because of the open-source software framework company Hadoop. The latter reduces storage and processing costs as compared with traditional data warehouses. This is also a significant initiative for Enterprises these days, groaning under high infrastructure and licensing costs as they are.
Understand the real business impact of migrating to the cloud >>>> Learn more
Most deploy Hadoop at the warehouse level itself because, as all of us know, a warehouse hosts the most significant amount of data in any company and is also computing-intensive. Overall, the 3 Vs of data as we know them – Volume, Variety, and Velocity – are making many Enterprises move to Hadoop ELT.
So there it is – the fight is on between the traditional data warehouses and today’s Cloud-based architectures that come with lower price tags, far better scalability, and of course, blazing performance. Let’s also not forget to add easy accessibility and team interactions made possible by the Internet.
Let’s take a closer look at all 3:
Pay when you use: That’s what Cloud-based data warehouses offer. Data-intensive Enterprises, such as e-commerce sites, are almost left with no choice today but to move operations to Cloud-based warehouses.
Getting these kinds of businesses’ data processes automated and thus agile, in a cost-effective manne,r is a big plus. Enterprises pay when specific loads go up and downgrade accordingly.
Move up or down: Moving from manual to automated data warehousing in the Cloud, IT teams can not only prototype but also rapidly release the final version of new analytic modules.
What it means is the IT team can fast-track new projects, maximizing an organization’s return on its Cloud investment. The ability to scale up or down, depending on an enterprise's workload and the capacity to grow to even petabyte scale, makes the Cloud a more attractive proposition.
Fast: The Azure SQL Data Warehouse, for example, delivers a dramatic improvement in query performance. In addition, it supports up to 128 concurrent queries and provides five times the computing power of the previous product generation. That’s some high speed.
What Enterprises Need to Ask Themselves before Moving to the Cloud:
- Are we already part of the Cloud ecosystem like AWS or Google Cloud Services?
- How many queries do we really run? Is it five an hour or five a minute?
- How much data is needed?
If the answer is yes for all three, switch to the Cloud.
To sum up, Cloud-based data warehousing coupled with automation definitely expands an Enterprise’s ability to cope with Big Data, but the move must be made only if it is required.
