A recent post on the use of artificial intelligence (AI) to clean data not only elicited tremendous response, but CTOs and CEOs wrote in to say they wanted an even more detailed post on dirty data and its consequences.
To be candid, the response took us aback. We also interpreted the responses as evidence of the problem's magnitude. Data quality is one of the top three challenges that Enterprises face in their business intelligence programs.
All of you know that to utilize Big Data effectively, it must be viewed through a business lens. The bridge between the two is master data management. This kind of program aggregates data from disparate sources and verifies their veracity to ensure data consistency.
Off the starter’s block, businesses usually perform data asset inventories. This helps establish the baseline for relative values, uniqueness, and validity of all incoming data. In the future, these baseline ratings will be used to measure all data.
Sounds good. Unfortunately, data asset inventories by themselves are not enough. Organizations run into hurdles as they grow, and the more they grow, the greater the chances of data quality being compromised.
Grow your business operations using our data cleaning services >>>>> Talk to Our Experts
There are many reasons for dirty data. At the top of the list is human error. The list is long, ranging from typos to erroneous values entered to duplicate entries.
Just below human error are IT architecture challenges. IT relies on multiple hardware and software platforms and solutions. If their mesh-up is not done correctly, it could cause data problems. What’s more, not updating systems as the Enterprise grows could also add to consistency errors.
Think of it as a pipeline bringing in the crude oil to the refinery. You need to monitor it for leaks, spillages, corrosion, joint failure, metal fatigue, and human error, all of which can lead to contamination or leakage at the ingest stage itself.
Then there is the problem of data decay, which not many businesses factor in.
Here’s a simple example: a client has moved house, and his address on file is no longer valid. Someone on the team needs to be tasked with keeping data current. Multiply such errors by thousands, and suddenly you have a massive problem of dirty data on your hands.
Areas that get impacted because of dirty data
There are both overt and covert consequences of poor data entering your system. Poor data comes with a cost, both tangible and intangible, and monetary as well as to reputation.
The most obvious sphere of business intelligence to be impacted is strategy, followed by decision-making. Any strategy born from dirty data has to fail. It also does not take rocket science to understand that poor data leads to poor decisions.
Dirty data can also lead to system inefficiencies and negatively affect productivity. All of which will eventually strain your Enterprise’s operational costs.
Here’s an example of low trust: The number of cars on the road on a particular date for a specific geography was erroneously filled in as X value rather than Y. The strategy of an automobile manufacturer to introduce electric cars as replacements in that particular market, based on this wrong input, will be flawed.
Another example: Faced with a deadline, a research executive for a detergent company keys in an approximation. The number under the men-women ratio in a particular suburb, without doing the legwork. What will follow next when the company wants to launch a new product is anybody’s guess.
The intangibles are difficult to measure but equally important. Consistent, nasty, or dirty data reporting in an organization can affect employee morale. Having to deal with inaccurate data 90% of the time can be frustrating.
Grow your business operations using our data cleaning services >>>>> Schedule a call
What your organization eventually needs is a single source of truth, from which all internal and external decisions are made to achieve the shared objectives.
In part 2 of this post, we will examine how to preserve data quality and how AI can maintain the integrity of large datasets.
References:
How to Clean Dirty Data – The Life of a Data Janitor
