Which other challenge do data-driven enterprises face in addition to the hurdle of poor or dirty data? The answer is – finding where their data, both structured and unstructured, is stored.
Corporations that rely on copious amounts of data run into this problem sooner or later. Where exactly is the data needed? Where does it reside? How does a team member access it?
In the last "big data" decade, enterprises relying on traditional data warehouses woke up to the "benefits" of Hadoop. After all, the former was unable to control complex hierarchical data types and other unstructured data. But of late, the debate has returned to square one – will Hadoop eventually replace data warehouses, or not?
Some in the industry say Hadoop may not necessarily be the answer to all big data problems.
For the last two years or so, a "new" trend has emerged – instead of centralizing data in HDFS clusters to fulfill the enterprise's data needs, corporations are back to building systems to handle specific data storage, processing, and analytics tasks.
Called a data catalog, this is a way to record databases across an enterprise, add descriptions (metadata), and so on.
Rather than having to "find" the relevant data in an enterprise-wide Hadoop implementation, such catalogs allow users to quickly locate the source of the information they want.
One definition of a data catalog by Gartner calls it a tool that "creates and maintains an inventory of data assets through the discovery, description, and organization of distributed datasets."
Transform Your Business Using Express Analytics' Machine Learning Solutions >>>> Learn More
What is metadata in this process? These are nuggets of information that identify (and help) in locating data that cannot be found in the data itself. Basically, they are keywords used to locate data in a repository or catalog.
Do Data Catalogs Worsen the Data Silo Problem?
There are no sure answers to this. There are studies showing that most respondents claim that the deployment of a data catalog has increased the difficulty of finding their data. Others, however, have claimed the complete opposite.
Data catalog, they say, cuts down the difficulty of finding data, and they would rather have their data residing in silos than face the gargantuan task of hunting it down.
The most significant advantage of a data catalog, according to its users, is that it "delivers context to the corporation's data". It means not only can you find the data about your data, but it also helps you understand the meaning of the data that is about to be used (or not used).
Data catalogs are also of great help in two areas – data governance and data management. It is of great benefit to organizations that operate in heavily regulated fields, such as healthcare, for example.
Some of the other advantages of a data catalog are:
- to capture and manage technical, operational, or BI metadata
- to provide collaboration capabilities that enable the capture of additional user-provided or social metadata
- to help establish the data lineage
Lineage is becoming a priority as enterprises build knowledge about their data.
Nobody wants any degree of chaos around their data, i.e., who collected it, what kind of transformation was done on it, and so on, while building insights. This is where data cataloging helps, too.
Now, here's the thing. Any data catalog is as good as the relevance of the information it keeps. Which means it needs to be constantly updated in a rapidly shifting work environment. This is where enterprises have started applying artificial intelligence (AI) and machine learning (ML) to enhance data lineage.
MLDCs, or machine learning data catalogs, are the new "in" in data cataloging. A Forrester report titled "Machine Learning Data Catalogs Put The Entire Business In Full View," says companies that utilize MLDCs are more than two times more effective at democratizing the use of data and enabling self-service.
We will look at this in part 2 of this blog post.
