Data Quality & Stewardship

As many companies have found out the hard way, data quality issues can quickly undermine the credibility of a BI/DW application, occasionally to the point of driving users away permanently. Fortunately, there is a rich body of knowledge that we can draw upon to reduce or eliminate the risk of poor data quality. In devising the best approach for your company, a key early decision that must be made is whether to attack data quality and stewardship on the basis of a specific BI project, on the basis of a broad BI/DW initiative, or on an enterprise basis.

At the enterprise level, successful companies have embraced and invested in a comprehensive approach that:

• establishes Data Governance at the enterprise level via a charter, assignments of authority with respect to data governance, and specified policies for managing key data as enterprise assets;
• assigns data stewardship responsibility and accountability to specific business user groups, who define, produce, and use the data;
• actively measures and improves data quality in relation to its intended uses and established structure, often with the use of advanced data profiling or ETL tools.

Ideally, the organization creates an environment where everyone understands that data is important to business success, that continuous improvement of data quality is a key business responsibility, and that the company takes a systematic, proactive approach to building data quality into its IT systems.

At the applied level, BI/DW projects often identify and attack data quality and data stewardship issues associated with the source systems that provide data needed for a specific BI application or a related group of applications. This process includes defining the attributes of data quality for purposes of the specific BI application or applications and then systematically analyzing one our more source data files for conformance to the data quality specification. The leading ETL tools all have data profiling functionality, and there are also specialized data profiling and master data management tools that are appropriate for this task.

The process of building out BI applications requires the BI team to assess data quality, and then quality problems can be addressed with business rules during the ETL process or by using root cause analysis to trace the problem back to the original source. This takes the BI team into the realm of data stewardship, whereby data problems are the responsibility of the business owners and users of the IT systems where the data is created. The classic example is data entry problems associated with transactional systems, which may require business process changes and/or training of business unit people.

Ultimately, there are a range of business and technical challenges associated with ensuring that your company’s BI initiatives and ability to drive profits are not undermined by data quality issues. If you are undertaking a critical BI initiative at the enterprise level, your company would probably benefit from a comprehensive, data governance approach. For a more focused BI initiative, data quality is just as important, but it may be more cost effective to attack the problem at the level of the BI application and the sources that feed it. Ultimately, we need to asses the business value of the data and determine the most effective data quality and stewardship approach. DecisionPath offers a structured approach to doing so, and we have used it with proven results at major organizations in a variety of industries.