Recently in Analyzing Data Quality Category

By Elliot King


And while that definition may be good enough in a practical sense for specific issues, it really isn't good enough to diagnose the sources of data quality problems generally. Constructing a general framework for data quality problems can be a useful guide in better identifying and resolving specific issues.

One of the earliest efforts to better understand the nature of data quality problems calls for classifying problems into three general categories--operational, conceptual and organizational. Operational data quality issues are those that are generated through problems with data capture and transmission. Inaccurate data is collected. Data may be missing. Or data may be corrupted through some process, for example.

Conceptual data quality problems occur when data is not well defined or it is inappropriate for its intended use. One of the most famous examples of a conceptual data quality problem (though it is not often thought of in this way) was brought to light in the movie Moneyball.

The basic thrust of the movie was not that the information old-time baseball scouts used to evaluate players was wrong per se; it was they were collecting the wrong data to identify productive players. Batting average, for example, is less useful in determining a player's value than on-base percentage. A pressing new conceptual data problem is the attempt to use electronic patient records to judge medical treatment outcomes.

When operational and conceptual data problems persist over time despite repeated attempts to fix them, organizational data quality problems are usually the culprit. In these cases, wrong, missing and invalid data is not really the problem, but the symptom. Something has to be fixed in the organizational structure or culture.

The point is this--data can be wrong for many reasons and it can't fundamentally be fixed without a general understanding of the error's cause.

By David Loshin


Yet, as more organizations look to merge data sets and feeds from different sources, some challenges remain, particularly with the use of unstructured text (such as that presented via Twitter or Facebook.) People cannot be expected to always conform to your organization's data standards, and often use colloquial terms or their own words to describe ideas that would map to your own dimensional values.

For example, if you wanted to filter out the individuals who prefer to drive "pickup trucks" (one of our standard values), it is not enough to scan for that phrase. Many individuals will refer to their pickup truck using different terms, such as a make and model ("Ford F-150," "Chevy Silverado") or a different name ("light truck") or a nickname ("baby monster"), but these terms have to be linked to the overall classification term.

This is an example of a simple hierarchy, in which one concept ("automobiles") is divided into a collection of smaller classes (the NHTSA classifications). Each of those classes in turn contains other phrases and terms. Within each of those included collections, there may be other inclusive categorization, such as by make and then model.

With a well-defined hierarchy for classification, unstructured text can be scanned for matches with values that live within the hierarchy, and that enables the standardized classification. To round out the example, a Twitter tweet exclaiming the author's love of "driving his Ford F-150" can be scanned, with the model name extracted, located within the make and model hierarchy for pickup trucks, thereby allowing us to register his/her automobile driving preference!

Do you know how accurate your company's data is? How much bad data is costing your organization? How you can get a single, complete view of your customers? Data quality today is a bottom-line issue that businesses must address to stay competitive. To help business managers and executives, marketing professionals, and other non-tech personnel understand what data quality is, and why it's important - we outlined 7 data quality principles in a convenient and easy-to-follow format.

By David Loshin


More concretely, the example we used was mapping individuals to their car purchase preferences, but different applications used different car classifications that did not share the same number of values and the value sets did not directly map in a one-to-one manner. The potential result is confusion in interpreting the results, especially if this classification is just one variable used for creating a customer profile.

One way to address this is to put a standards policy for classification dimensions in effect by selecting a single set of concepts, mapping those conceptual values to a standard single set of values, and then insisting that any application that uses that conceptual domain always use the standard.

This sounds simple, but it actually may entail some effort, since no one person may be aware of all the places that any specific classification domain is used.

This task goes beyond a "data management" activity and essentially becomes a "data governance" one involving a broad solicitation across the community of data consumers to determine the classification dimensions used and the enumerations of values employed within each dimension.

At the same time, the analyst spearheading this effort must have a plan for capturing the classification data, harmonizing values across variant lists, selecting a standard, communicating the standard, and then ensuring that the standard is put into practice.

Establishing good practices and processes for domain harmonization and standardization is an important topic to be considered in upcoming posts, but next time we will look at a growing challenge for classification domains: aligning data from unstructured text with the standard classification dimensions.

By Elliot King


Ironically, the impact of poor data quality on customer satisfaction is often overlooked. According to a survey of members of the Association of Business Process Management Professionals (ABPMP), of the 45 percent who reported that they are working on improving CRM processes, only 38 percent have evaluated the impact that poor-quality data has on the effectiveness of these processes. That statistic is a little frightening.

Poor data quality coupled with the inability to deliver the right data to the customer at the right time damages overall customer satisfaction in a variety of ways. Consider this scenario. Many retailers run multiple promotions at the same time. Each promotion has different rules and restrictions and run for different time periods.

When customers go to pay, if the final price does not reflect the discounts they anticipated, they will not be real happy. The flip side of the coin is true as well. If a sales process does not include a discount where one is due, an opportunity to build goodwill or perhaps close a deal will be lost.

But misunderstandings and the failure to provide the appropriate product are the tip of the iceberg. If point-of-contact personnel do not have confidence in the information they receive, the process for which the data is needed will be slowed and their productivity diminished.

And there is the annoyance factor. Think about your interactions with your telephone or cable provider. You call with a problem. The computer confirms the number you are calling about. If you are lucky and you finally find a human to talk to, how many times do they have to reconfirm all your information--telephone number, address, and so on--especially when you are passed from one person to another? Okay, part of the problem is most likely the lack of overall system integration, but part of the problem is a fear of faulty information.

Poor data quality is not a theoretical issue. It can hurt you in the place that may hurt most--your relationship with your customers.

By David Loshin


And that is the problem: there are a number of different classifications of cars, and without a defined standard, there's bound to be confusion. Here are three examples (I got them from a page at Wikipedia):

• The Highway Loss Data Institute (HLDI) classifies cars into five groups: Sports, Luxury, Large, Midsize, and Small.
• The National Highway Traffic Safety Administration (NHTSA) has eight classifications (based on curb weight of the car): mini passenger cars, light passenger cars, compact passenger cars, medium passenger cars, heavy passenger cars, sport utility vehicles, pickup trucks, and vans.
• The EPA has a car classification based on interior and cargo space: Two-seaters, minicompacts, subcompact, compact, mid-size, large, small station wagons, mid-size station wagons, large station wagons.

While one application might assign a demographic classification based on the HLDI groupings, another application might use the NHTSA classification, but aspects of those classifications don't match: the set of small HLDI cars might include the NHTSA sets of mini passenger cars, light passenger cars, and compact passenger cars.

The absence of a standard within the enterprise for choices of classification may seem irrelevant within siloed functions, but as more business processes are monitored across multiple functions, variant dimensions for classification and analysis will create confusion somewhere down the line.

By Elliot King


Perhaps the most common source of data errors and one of the most difficult to correct is data entry itself. In many cases, humans--typically data entry clerks, customer service representatives or end users via the Web or some other mechanism--initially enter data into the system.

They enter data incorrectly for many reasons. Data entry personnel or customer service representatives may be required to work too fast. The entry screen may be poorly designed. Or, in some cases, data may be incorrectly entered intentionally. For example, in several cases across the country, criminal lab personnel have been found to have entered data into their systems for tests that they did not perform. And a recent study of the use of electronic medical records found that medical personnel may be falsely claiming to have performed specific examinations.

Incorrect data entry, accidental or not, is only one part of the problem. Sometimes the data itself is just wrong. Measurements have been taken incorrectly. Sampling techniques are wrong and so on. As data volumes increase, controlling the quality of the initial data becomes more difficult. Moreover, frequently, data analysis does not require, or cannot use, all the data collected. The techniques used to distill or summarize data may be faulty.

Finally, in many cases, data does not come from a single source. With data flowing into systems from several different directions, data integration is a challenge. In practice, databases are always evolving and as more data is incorporated into different repositories, inconsistencies must always be resolved.

There is some hope. If you pay attention, the amount of incorrect data in your system can be reduced, but probably cannot be totally eliminated. The sources of errors are just too pervasive.