Putting the Big Q into Data Integration
by Dr. Fern Halper, Partner and Marcia Kaufman, Partner
We recently spoke with an IT executive about the data integration challenges his team is facing due to a recent merger. The situation he described is familiar, but what made this conversation stand out was the emphasis on data quality in his company?s data integration strategy. This executive is trying to move to a standardized way of dealing with company data so that reports of financial, sales, operational, and other information from the newly merged organization are synchronized and consistent. However, he is concerned that once the data comes together, business decisions might be made from unreliable information. His concerns are right on the money. He knows that his data integration effort must include a data quality effort to ensure accurate, complete, reliable, and reasonable information. He is working with the business side of the house to make sure he is providing the right information. He is also thinking about data architecture and data governance issues. This kind of thinking we call data quality with a ?big Q?, or Quality with a business and strategic focus. This is in sharp contrast to what we hear from many IT executives who follow a ?little q? approach to data quality, one that is more technically and tactically focused.
What do we mean by a ?little q? approach to data quality and why are we concerned? When we ask IT executives about how data quality fits in with their integration initiative we often hear the following:
- Data quality has improved as a result of using their ETL (Extract, Transform, Load) tools. While ETL can help companies understand some aspects of data quality (for example, loading records once and only once, knowing that the data in a data warehouse matches the source data), this isn?t the whole data quality story.
- Data quality is under control because all the company?s data and data integration efforts are the responsibility of a centralized IT department. However, IT does not have a data quality strategy in place and they have not deployed software tools designed to manage data quality.
- Data quality isn?t an issue for a division?s data integration initiative because the team believes that data quality is happening somewhere else in their company and it isn?t their problem.
Statements like these have led to our concern about the data quality gap that occurs in many organizations. Data quality needs to be an integral part of any company’s information infrastructure. Poor data quality affects operational efficiencies, decision-making, and reporting, to name only a few of the more obvious targets. And, data quality is at the heart of new regulatory compliance mandates. A sound data quality strategy needs to include traditional quality measures for accuracy, reliability, completeness, consistency, timeliness, reasonableness, and validity. Ideally, it should also include metrics such as interpretability, understandability, usability, and accessibility. The technology strategy needs to incorporate capabilities for data cleansing, profiling, standardization, and monitoring. The strategy also needs to have IT aligned with the business.
While companies have become more aware of the importance of data quality with the advent of numerous government regulations and privacy concerns, many are still in need of a strategy to make it happen. Hurwitz & Associates expects to see more companies stepping up to the ?big Q? approach to data quality now that several key vendors have made data quality an integral part of their data integration offerings. For example, with its recent purchase of Similarity Systems, Informatica has incorporated data quality into their access-discover-cleanse-integrate-deliver data integration lifecycle.
Similarity Systems brings a focus on the business, it deals with other types of data in addition to customer information, and it monitors quality measures over time. IBM?s purchase of Ascential Software as part of their information integration solution is another good example of a vendor stepping up to the plate to make quality an important focus of any integration effort. Over the past several years IBM has made a huge investment in its common platform for information integration and they place an emphasis on data trust. They have divided their approach to information integration into five capabilities: connect, understand, cleanse, transform and federate, with quality playing a critical role. Other vendors who have incorporated data quality into their offerings include SAS/Data Flux and Business Objects recent purchase of Firstlogic, which it will embed in its data integration capabilities.
Data quality needs to be considered a corporate mandate, not just an IT issue. A sound data quality strategy will include both business and technical considerations and should involve both business and IT people. This is quality with a big Q ? and it will become more necessary as a company?s information infrastructure becomes more complex and the demands on it even greater.