Recently in Analyzing Data Category

By Elliot King


And while that definition may be good enough in a practical sense for specific issues, it really isn't good enough to diagnose the sources of data quality problems generally. Constructing a general framework for data quality problems can be a useful guide in better identifying and resolving specific issues.

One of the earliest efforts to better understand the nature of data quality problems calls for classifying problems into three general categories--operational, conceptual and organizational. Operational data quality issues are those that are generated through problems with data capture and transmission. Inaccurate data is collected. Data may be missing. Or data may be corrupted through some process, for example.

Conceptual data quality problems occur when data is not well defined or it is inappropriate for its intended use. One of the most famous examples of a conceptual data quality problem (though it is not often thought of in this way) was brought to light in the movie Moneyball.

The basic thrust of the movie was not that the information old-time baseball scouts used to evaluate players was wrong per se; it was they were collecting the wrong data to identify productive players. Batting average, for example, is less useful in determining a player's value than on-base percentage. A pressing new conceptual data problem is the attempt to use electronic patient records to judge medical treatment outcomes.

When operational and conceptual data problems persist over time despite repeated attempts to fix them, organizational data quality problems are usually the culprit. In these cases, wrong, missing and invalid data is not really the problem, but the symptom. Something has to be fixed in the organizational structure or culture.

The point is this--data can be wrong for many reasons and it can't fundamentally be fixed without a general understanding of the error's cause.

By Allison Moon
Data Quality Analyst

This new interface in the current Phone Object takes in a phone number and country to determine whether the number is valid for the region. It can also take in a phone number alone and determine what country the number is from - by simply looking at the international access code (these are the digits used to dial into a country).

For example, with +49 3079788829, the Global Phone Object will be able to determine that this number is from Germany, just by looking at the international access code, +49.

In addition, Global Phone Object will also return international geographical information. Just like the current Phone Object, the Global Phone Object will return the latitude and longitude coordinates of the detected region.

It can also return the predominant language spoken in the phone number's region. This field is helpful in preventing phone calls from being lost in translation. By determining the pre-dominant language of the region, phone calls in a call center environment can be distributed to employees who are fluent in the detected language.

The Global Phone Object is a powerful tool that can be used in a multitude of ways to bridge international communication.

-- Allison Moon is a data quality analyst and software engineer for Melissa Data.

-- For more info on Global Phone Object, go to:
http://www.melissadata.com/phone-verification/index.htm

By David Loshin


Yet, as more organizations look to merge data sets and feeds from different sources, some challenges remain, particularly with the use of unstructured text (such as that presented via Twitter or Facebook.) People cannot be expected to always conform to your organization's data standards, and often use colloquial terms or their own words to describe ideas that would map to your own dimensional values.

For example, if you wanted to filter out the individuals who prefer to drive "pickup trucks" (one of our standard values), it is not enough to scan for that phrase. Many individuals will refer to their pickup truck using different terms, such as a make and model ("Ford F-150," "Chevy Silverado") or a different name ("light truck") or a nickname ("baby monster"), but these terms have to be linked to the overall classification term.

This is an example of a simple hierarchy, in which one concept ("automobiles") is divided into a collection of smaller classes (the NHTSA classifications). Each of those classes in turn contains other phrases and terms. Within each of those included collections, there may be other inclusive categorization, such as by make and then model.

With a well-defined hierarchy for classification, unstructured text can be scanned for matches with values that live within the hierarchy, and that enables the standardized classification. To round out the example, a Twitter tweet exclaiming the author's love of "driving his Ford F-150" can be scanned, with the model name extracted, located within the make and model hierarchy for pickup trucks, thereby allowing us to register his/her automobile driving preference!

Do you know how accurate your company's data is? How much bad data is costing your organization? How you can get a single, complete view of your customers? Data quality today is a bottom-line issue that businesses must address to stay competitive. To help business managers and executives, marketing professionals, and other non-tech personnel understand what data quality is, and why it's important - we outlined 7 data quality principles in a convenient and easy-to-follow format.

By David Loshin


More concretely, the example we used was mapping individuals to their car purchase preferences, but different applications used different car classifications that did not share the same number of values and the value sets did not directly map in a one-to-one manner. The potential result is confusion in interpreting the results, especially if this classification is just one variable used for creating a customer profile.

One way to address this is to put a standards policy for classification dimensions in effect by selecting a single set of concepts, mapping those conceptual values to a standard single set of values, and then insisting that any application that uses that conceptual domain always use the standard.

This sounds simple, but it actually may entail some effort, since no one person may be aware of all the places that any specific classification domain is used.

This task goes beyond a "data management" activity and essentially becomes a "data governance" one involving a broad solicitation across the community of data consumers to determine the classification dimensions used and the enumerations of values employed within each dimension.

At the same time, the analyst spearheading this effort must have a plan for capturing the classification data, harmonizing values across variant lists, selecting a standard, communicating the standard, and then ensuring that the standard is put into practice.

Establishing good practices and processes for domain harmonization and standardization is an important topic to be considered in upcoming posts, but next time we will look at a growing challenge for classification domains: aligning data from unstructured text with the standard classification dimensions.

By Elliot King


Ironically, the impact of poor data quality on customer satisfaction is often overlooked. According to a survey of members of the Association of Business Process Management Professionals (ABPMP), of the 45 percent who reported that they are working on improving CRM processes, only 38 percent have evaluated the impact that poor-quality data has on the effectiveness of these processes. That statistic is a little frightening.

Poor data quality coupled with the inability to deliver the right data to the customer at the right time damages overall customer satisfaction in a variety of ways. Consider this scenario. Many retailers run multiple promotions at the same time. Each promotion has different rules and restrictions and run for different time periods.

When customers go to pay, if the final price does not reflect the discounts they anticipated, they will not be real happy. The flip side of the coin is true as well. If a sales process does not include a discount where one is due, an opportunity to build goodwill or perhaps close a deal will be lost.

But misunderstandings and the failure to provide the appropriate product are the tip of the iceberg. If point-of-contact personnel do not have confidence in the information they receive, the process for which the data is needed will be slowed and their productivity diminished.

And there is the annoyance factor. Think about your interactions with your telephone or cable provider. You call with a problem. The computer confirms the number you are calling about. If you are lucky and you finally find a human to talk to, how many times do they have to reconfirm all your information--telephone number, address, and so on--especially when you are passed from one person to another? Okay, part of the problem is most likely the lack of overall system integration, but part of the problem is a fear of faulty information.

Poor data quality is not a theoretical issue. It can hurt you in the place that may hurt most--your relationship with your customers.

By David Loshin


And that is the problem: there are a number of different classifications of cars, and without a defined standard, there's bound to be confusion. Here are three examples (I got them from a page at Wikipedia):

• The Highway Loss Data Institute (HLDI) classifies cars into five groups: Sports, Luxury, Large, Midsize, and Small.
• The National Highway Traffic Safety Administration (NHTSA) has eight classifications (based on curb weight of the car): mini passenger cars, light passenger cars, compact passenger cars, medium passenger cars, heavy passenger cars, sport utility vehicles, pickup trucks, and vans.
• The EPA has a car classification based on interior and cargo space: Two-seaters, minicompacts, subcompact, compact, mid-size, large, small station wagons, mid-size station wagons, large station wagons.

While one application might assign a demographic classification based on the HLDI groupings, another application might use the NHTSA classification, but aspects of those classifications don't match: the set of small HLDI cars might include the NHTSA sets of mini passenger cars, light passenger cars, and compact passenger cars.

The absence of a standard within the enterprise for choices of classification may seem irrelevant within siloed functions, but as more business processes are monitored across multiple functions, variant dimensions for classification and analysis will create confusion somewhere down the line.