What's Driving the Need for Data Quality?

In an early report by The Data Warehousing Institute (TDWI), 48% of the respondents claimed that their companies had "no plans" for a data quality initiative. A more recent survey* by TDWI revealed that the percentage of respondents without a plan had dropped dramatically to 24%. Why the change of heart? As usual both carrots and sticks have had their say in the issue.

Comply or die
Perhaps one of the biggest sticks has been substantially increased regulatory compliance requirements: there is nothing more effective than the threat of hefty penalties and/or a stint in jail to get resources mobilized. Whilst many have yet to make the connection between data quality and compliance, those that have done so have ensured that they not only meet compliance requirements, but have also created a much more efficient organization, resulting from all the attendant benefits of high quality data (See my previous articles on ITWeb). Banks, in particular, have had to invest heavily in Basel II initiatives, whilst ensuring that data being used to assess risk is also completely trusted. Others might soon need to comply with Sarbanes-Oxley (SOX), Anti Money Laundering (AML), FICA and RICA, all of which need to be underpinned by high quality, trusted data.

Information is also an asset
The recent realization that an organization's data is an asset that needs much better management, has resulted in a flurry of interest around topics such as Data Governance, Enterprise Information Management (EIM), Master Data Management (MDM) and Customer Data Integration (CDI - also sometimes referred to as "Single View of Customer"). In the retail and manufacturing sector, for example, a great deal of work is being done to ensure that all trading partners use aligned product data, and so projects for internal data alignment and Global Data Synchronisation (GDS) are being planned. But just as tidying up your garage results in more order, in addition to a pile of junk that you take to the dump, so too will any of these initiatives demand that you first identify the dirt within your data and clean it up before trying to improve the way you manage it. Therefore any attempts to 'govern' data or to achieve a consolidated view of whatever your organization considers critical to survival and growth, be it customers, products, suppliers or business events, will require a data cleansing exercise as a pre-requisite.

Intelligent Business
It is by now well known that BI is a top priority for most organizations, and it has become equally well known that dirty data leads to dirty decisions - the need for higher quality has been driven by frequent boardroom frustrations of inaccurate, ambiguous and generally mistrusted information. But because BI relies not only on the data itself but also on its meaning and presentation, it has also become necessary for organizations to re-assess how they model, store and present their information. The subject of metadata management therefore, for years pushed to the back of the queue, is finally nearing the entrance, as companies start to formally define and align the semantics of the data they maintain. Once again data quality enters via the side door, as companies start out to improve the accuracy of their reporting and BI systems, and wind up tackling this deeper, more fundamental issue.

Data Migrations
Another more recent trend has been to profile and cleanse data within legacy systems, before moving it to, for example, a new ERP package. For years data has simply been "mapped then moved", only to cause significant problems in the new system because of quality levels unable to support enhanced functionality. This is bit like moving house but taking all your junk with you! Better to thoroughly clean, consolidate and enhance data before trying to shoe-horn it into its new home, and so data profiling, standardization and matching has become common practice as sub-projects of ERP implementations.

Good Business
Finally, seeing the carrot, some more mature organizations are tackling data quality in its own right, without a parent project or another driver such as compliance. In the end, all will recognize that data is the foundation of our businesses, and that high quality data will lead to lower costs, loyal customers, increased opportunities and better business overall. So whether it's an MDM project resulting in a data quality initiative, or a pure data quality drive leading to sound data governance, data quality is the common thread that binds these all together. Whether you see it as a carrot or a stick, all paths eventually lead to an imperative to do something about improving the quality of the only re-useable resource any organisation possesses - its data.

* "Taking Data Quality to the Enterprise through Data Governance" by Philip Russom. The Data Warehousing Institute.