Melissa Data's Data Quality Authority

Melissa Data's Data Quality Authority tag:blog.melissadata.com,2008-09-24:/data-quality-authority/2 2012-05-16T16:48:57Z The Melissa Data Authority Blog is a forum for data quality professionals to gain and share knowledge about data quality issues that impact data warehousing, data integration, data enrichment, and data management strategies. Movable Type 4.21-en Get to the Root of Data Quality Problems tag:blog.melissadata.com,2012:/data-quality-authority//2.183 2012-05-16T16:38:52Z 2012-05-16T16:48:57Z

Blog Administrator Elliot King

Safeguarding data quality is a continual and ongoing process. Sadly, no matter how diligent companies are, data quality problems will be created. But why is that?

The root causes of data quality problems are deeply intertwined with daily business operations coupled with the simple truth that businesses are not static. Things change. Time decay is perhaps the most obvious root cause of data quality issues. People move; they die; they get divorced; they no longer have a need for your specific product, and so on. Updating data records to reflect those changes in a timely fashion is difficult since you may not even be aware that the change has occurred. Over time, data that was once right becomes wrong.

Time decay is a serious ongoing problem. But more dramatic quality issues often emerge when organizations grow, change the application infrastructure, or must respond to external demands. Too frequently, for example, when a company purchases another company, there is not enough time to fully merge the information infrastructure. Instead, the IT organization looks for a workaround or stopgap measure to insure the consolidation does not impede ongoing operations. These workarounds bring with them both known and unknown risks to data quality.

A workaround is often the solution of choice in a variety of other situations as well. A company may choose to eliminate certain applications or face a new regulatory requirement. Branch office or remote operation personnel may feel that the central IT staff cannot respond to their needs fast enough. In response to the pressure to "keep things online," an organization may opt for a workaround.

A third root cause of data quality issues is that the company simply has not established a sufficient data quality program. Since every piece of data cannot be verified, the data quality system itself may be flawed. Alternatively, data entry screens may be poorly designed and generate too many errors. And too few companies have standard data dictionaries.

As long as business is a fluid process, data quality issues will always arise. The key is to recognize their source and have programs in place to minimize the number of problems and reduce their impact on business.

]]> Postal Standards and Address Quality - Take 1 tag:blog.melissadata.com,2012:/data-quality-authority//2.182 2012-05-08T16:43:13Z 2012-05-08T16:51:10Z

Blog Administrator David Loshin

The USPS Postal Standard (Publication 28) provides at least some of the specifications we need for address quality. For example,

"The Postal Service defines a complete address as one that has all the address elements necessary to allow an exact match with the current Postal Service ZIP+4 and City State files to obtain the finest level of ZIP+4 and delivery point codes for the delivery address."

The next paragraph provides some additional details:

"A standardized address is one that is fully spelled out, abbreviated by using the Postal Service standard abbreviations (shown in this publication) or as shown in the current Postal Service ZIP+4 file."

A large part of the remainder of the document guides what is valid and what is not valid, as well as the postal standard abbreviations (as mentioned in the definition of standardized). So an address must be complete, which by definition implies that it can be matched with current Postal Service ZIP+4 and City State files.

This match is to obtain the ZIP+4, so the implication is that verification means that a complete address matches the USPS files and has the correct ZIP+4. The address components must be consistent with the postal standard in terms of valid and invalid values. For example, a street address cannot have a number that is outside the range of recognized numbers (that is, if the USPS file says that Main Street goes from 1-104, an address with 109 Main St is invalid). So validation means that the street address is consistent with what is documented by the USPS files. Standardization is also defined by the above reference: it is spelled out, and uses the USPS standard abbreviations.

In turn, the process for address quality would be to:

1) Ensure the address is complete.
2) Ensure that the address values are valid by checking it against the USPS files.
3) Verify the address's ZIP+4 by matching against the USPS fles.
4) Standardize the address according to the USPS standardized abbreviations.

]]> Senate Green Lights Postal Reform - But Is It Enough? tag:blog.melissadata.com,2012:/data-quality-authority//2.181 2012-05-01T18:39:42Z 2012-05-01T18:48:45Z

Postal Reform legislation passed the Senate this month, but according to the Postal Service (and who should know better?) S. 1789 falls disappointedly short of restoring the USPS to financial viability. For the past two years, the Postmaster General and...

Blog Administrator
For the past two years, the Postmaster General and the Board of Governors of the USPS have worked diligently preparing a comprehensive five-year plan to profitability that would enable revenue generation and achieve cost reductions of $20 billion by 2015 - restoring the Postal Service to long-term growth. The USPS is currently losing $25 million a day and has a debt of more than $13 billion.

Following the two days of sessions it took to vote on all the amendments in the bill, PMG Patrick Donahoe stated: "Based on our initial analysis of the legislation passed today, losses would continue in both the short and long term. If this bill were to become law, the Postal Service would be back before the Congress within a few years requesting additional legislative reform."

So where do we go from here? To the House of Representatives with an entirely new bill - H.R. 2309 - and with their own ideas for Postal Reform.

In the meantime? Take a look here at S.1789 to see exactly what the Senators voted for, and against, and then review the USPS Plan to Profitability - 5 Year Business Plan ... It's a good read. After that ... well, we'll keep you posted.

]]> Translating Expectations into Quality Directives tag:blog.melissadata.com,2012:/data-quality-authority//2.180 2012-04-25T18:07:34Z 2012-04-25T18:23:40Z

By David Loshin In my last post, I raised the question about the variety of terms used in describing address quality, and I introduced a set of core concepts that needed to be correct to provide the best benefits...

Blog Administrator David Loshin

In my last post, I raised the question about the variety of terms used in describing address quality, and I introduced a set of core concepts that needed to be correct to provide the best benefits for accurate parcel delivery. Let's look at these more carefully:

1) The item must be directed to a specific recipient party (either an individual or an organization.)
2) The address must be a deliverable address.
3) The intended recipient must be associated with the deliverable address.
4) The delivery address must conform to the USPS standard.

Together these concepts have implications for address quality, and we can start with the first 3 concepts. The first concept implies a direct connection between entities: the sender and the recipient.

The corresponding business rule is relatively subtle - it suggests that the recipient must be identifiable to the sender. Concept #2 is a bit more direct: the address must be a deliverable address. This means that the address must carry enough information to enable a carrier to locate the address as a prelude to delivery. Concept #3 establishes a direct dependence between the recipient and the addressed location, implying awareness of that connection.

Together we can infer more discrete assertions:

• The address must be accurately mappable to a real location.
• The address must contain enough information to ensure delivery.
• The recipient must be a recognized entity.
• The recipient must be connected to the address.

In the next few posts we will figure out what these assertions really mean in terms of transforming a provided address into a complete, validated, and standardized address.

]]> Characterizing the Quality of Address Data tag:blog.melissadata.com,2012:/data-quality-authority//2.179 2012-04-20T16:45:53Z 2012-04-20T16:55:40Z

By David Loshin My company is currently working on a couple of projects associated with address quality and location master data. We are reviewing a lot of the existing documentation that has been collected from a number of different operational...

Blog Administrator David Loshin

My company is currently working on a couple of projects associated with address quality and location master data. We are reviewing a lot of the existing documentation that has been collected from a number of different operational systems, as well as reviewing the business processes to see where location data is either created, modified, or read.

And there are many references to operations or transformations performed on addresses, mostly with the intent of improving the quality of the address.

Curiously, there are a number of different terms used to refer to these different transformations: validation, verification, standardization, cleansing, correction. I am sure there are others. But what do all these things mean? And why are these different terms used if they mean the same thing?

The first step in exploring the answer to this question is reflecting back on the nature of deliverable addresses. When an item is sent to an addressed location, there are some core concepts that need to be right:

1) The item must be directed to a specific recipient party (either an individual or an organization).
2) The address must be a deliverable address.
3) The intended recipient must be associated with the deliverable address.

In addition, there are certain incentives provided to senders when the addresses are completely aligned with the Postal Standard, adding one more concept:

4) The delivery address must conform to the USPS standard.

These directives provide us with some material with which to work for differentiating the different terms used for postal data quality. More next week...

]]> Sometimes Data Quality is the Law tag:blog.melissadata.com,2012:/data-quality-authority//2.178 2012-04-12T20:05:42Z 2012-04-12T20:10:01Z

By Elliot King We have all read the statistics about the real costs that poor data quality represents. And intuitively, we know that bad data is, well, bad. But, in many cases, bad data is more than just bad for...

Blog Administrator Elliot King

We have all read the statistics about the real costs that poor data quality represents. And intuitively, we know that bad data is, well, bad. But, in many cases, bad data is more than just bad for business. Increasingly, good data is required by law.

In 2001, the U.S. Congress added two lines to its major appropriations act that required that the Office of Management and Budget "provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies." These two lines have come to be known as the Data Quality Act (DQA) or the Information Quality Act.

While the DQA applied to federal agencies, it also put a marker in the ground. Regulatory agencies could demand that data given to them meet criteria set by the Congress. Government data had to be accurate, objective and have integrity as a matter of law.

The same notion has spread to corporate America and other specific industries. In one of the most high profile examples, the Dodd-Frank Wall Street Reform and Consumer Protection Act passed in the wake of the financial meltdown of 2008 established the Office of Financial Research with a mandate to improve the quality of financial data accessible to regulators.

One of the primary challenges for financial institutions is to insure that the information they supply to regulators is consistent across all their divisions. Companies are also going to be able to track data flows and usage and develop chains of custody that can be audited. The net result should be that all users of the data will see consistent information. And that may not be easy when data is siloed in different operating entities, which is often the case in those too-big-to-fail financial behemoths.

The regulatory pressure for data quality is being felt elsewhere as well. The American Health Information Management Association, which promotes the technological advancement of health information management systems, has noted that for electronic health records (EHR) to have the positive impact on overall health care that their proponents anticipate, data quality can no longer be a reactive process based on auditing but must be proactively focused on data capture. Standards to ensure that result will be built into the requirements for EHRs.

While many folks in IT chafe at government regulations. However, the Federal government often has been a leader in IT innovation. Think about the Internet. Federal IT initiatives have also been inept--think about the FBI case management system or the Air Traffic Control system. While burdensome, the regulatory demand for higher quality data could easily have a positive ripple effect that will spread widely.

]]> Clean Data is Good Data tag:blog.melissadata.com,2012:/data-quality-authority//2.177 2012-04-09T17:43:39Z 2012-04-09T17:47:15Z

By Elliot King The cliché is as old as computing itself--garbage in, garbage out. And that cliché is as true now as ever, if not more so. Unfortunately, with information flowing into companies from so many sources including the...

Blog Administrator Elliot King

The cliché is as old as computing itself--garbage in, garbage out. And that cliché is as true now as ever, if not more so. Unfortunately, with information flowing into companies from so many sources including the Web and third-party providers, mistakes should not just be expected; they are basically inevitable. Garbage data is going to get in your data systems.

We want to close our eyes to bad data and just pretend it doesn't matter; but that would be a major mistake. Virtually any operation driven by faulty data is suspect. Trends you may uncover could be wrong. Your customer contact efforts could be inappropriate or misdirected.

Data cleansing, the systematic effort to remediate bad data is no trivial task. First, so many different kinds of errors can exist. Mistakes can occur in single source systems as well as multiple source systems. Errors and inconsistencies can be introduced at both the metadata level--the data schema or the information wrapper, in the case of the Web, may be flawed--or at the granular level, where the information itself is just not right.

Just as the information itself, so much can go awry. There can be missing values and misspellings. Information can be entered into the wrong field--a street name in the city field perhaps. Attributes that should be linked aren't--let's say a city without a ZIP code. Records could contradict each other or be associated incorrectly. For example "John Smith" may actually work in payroll and not in human resources.

Not surprisingly, data cleansing has more steps than doing the laundry. The first step is to analyze the data and find the real mistakes--the omissions, the contradictions and the errors. The next step is to re-engineer and validate the new metadata and rules to address those errors. The data has to be transformed, expunging the problems. At that point, the data can be reloaded into the database. And in case that doesn't sound all that daunting, each of the steps generally has a slew of sub processes too.

Of course, none of this had to be done manually. Many different tools have been introduced into the market. Some are more generalized while others specialize in fixing a specific problem such as names and addresses.

At the end of the day, "garbage in, garbage out" sounds really harsh. Maybe you should look at it this way--clean data is good data. But just like your clothes, if you use data it will get dirty, so the cleansing actually never ends.

]]> Where in the End-to-End Should Address Standardization and Correction Happen? tag:blog.melissadata.com,2012:/data-quality-authority//2.176 2012-04-05T17:10:58Z 2012-04-05T17:36:21Z

By David Loshin In my last post, I shared a story about how rampant address validation actually can transform accurate (if not 100% standardized) addresses into inaccurate ones. My client actually noted that with some of the tools they...

Blog Administrator David Loshin

In my last post, I shared a story about how rampant address validation actually can transform accurate (if not 100% standardized) addresses into inaccurate ones. My client actually noted that with some of the tools they have seen, addresses that have been submitted to the product and standardized are then re-submitted to the product, which then reports their own corrected address as being invalid! Huh?

Here are a few reflections:

1) Siloed application development has led to multiple iterations of address correction, often using different tools, or if using the same tool, using different sets of rules, which is probably not the best scenario since it will lead to inconsistencies.

2) Address standardization tools have to be maintained - the rules may change, locations may change, the standards may evolve.

3) The criticality of precision and standardization/validation has to be assessed in the context of how the address (or location) is being used. For example, for delivery purposes accuracy should trump "validity" if a validated address is no longer correct.

4) Find a single tool solution for address standardization and validation instead of using multiple products.

5) Have a single team responsible for managing the solicitation of requirements, definition of rules, and provision of address validation and correction services.

One last thought: push address correction and validation to the earliest point in the business process. If your application solicits an address from the customer, validate it right away and have the corrected address verified directly by the customer. And trust the customer to give you accurate address data - they probably have some experience in ensuring they get their mailings!

]]> Enter the Contact Zone: Where Data Integration and Data Quality Are Simplified tag:blog.melissadata.com,2012:/data-quality-authority//2.175 2012-03-28T20:40:28Z 2012-03-28T20:48:42Z

By Joseph Vertido For many, the concepts of data integration and data quality are separate and have no commonality. But in reality, when you combine them - they create a partnership that excels. Where data quality leaves off, data...

Blog Administrator Joseph Vertido

For many, the concepts of data integration and data quality are separate and have no commonality. But in reality, when you combine them - they create a partnership that excels. Where data quality leaves off, data integration begins, and vice versa. A new product - Contact Zone - fuses these two concepts together into one revolutionary solution for where data integration and data quality converge.

Data integration tools simplify data migration and data warehousing procedures - both of which are concerned with the issue of data management, i.e. keeping data organized. Data quality, on the other hand, is concerned primarily with an understanding of the nature, and validity of the contents of the actual data, i.e. keeping data clean. Maintaining an organized database is not the same as keeping it clean - they are two different approaches to handling data - but they can be combined, or should they?

The short answer is yes.

In essence, data integration allows for the migration of data from a given source to a given destination. Typically, users take advantage of data integration to accomplish data warehousing initiatives - allowing for easy migration and manipulation of data, which ultimately leads to maximizing the efficiency of business intelligence and analytics.

However, Gartner states that "only 30 percent of business intelligence and data warehousing implementations fully succeed." Why? The top two reasons for failure are budget constraints and data quality. So, although the architectural constraints of building a data warehouse can be addressed by utilizing data integration tools, it still leaves the problem of poor data quality - something that most data integration tools handle with mediocrity at best.

That's where Contact Zone comes into play. It's a data integration tool optimized for data quality, allowing you to shoot two birds with one stone.

Contact Zone connects to virtually any source, overcoming an obstacle our clients frequently encounter when implementing data quality, namely there is such a variety of database format and platforms today that the types of environments and combinations can be overwhelming.

Whether you have an IBM DB2 database or PostgreSQL, leveraging Contact Zone allows for data integration for almost any form of database format, while making sure that all data is clean, correct, standardized, and valid.

]]> Validation, Standardization, and Correction: Tool or Process? tag:blog.melissadata.com,2012:/data-quality-authority//2.174 2012-03-27T21:38:25Z 2012-03-27T22:19:57Z

By David Loshin There are all sorts of tools associated with address standardization, cleansing, and validation. As an example, the USPS has a certification program for software vendors, referred to as CASS (Coding Accuracy Support System)™ certification. According to...

Blog Administrator David Loshin

There are all sorts of tools associated with address standardization, cleansing, and validation. As an example, the USPS has a certification program for software vendors, referred to as CASS (Coding Accuracy Support System)™ certification. According to their website,

CASS enables the Postal Service™ to evaluate the accuracy of address matching software programs in the following areas:

(1) five-digit coding
(2) ZIP + 4/ delivery point (DP) coding
(3) carrier route coding
(4) DPV^®
(5) DSF2^®
(6) LACS^Link®
(7) eLOT^®
(8) RDI™ products

CASS allows vendors/mailers the opportunity to test their address-matching software packages and, after achieving a certain percentage of compliance, to be certified by the Postal Service. CASS does not measure the accuracy of ZIP + 4 delivery point, five-digit ZIP, or carrier route codes in a mailer's existing files. CASS enables mailers to measure and diagnose internally written, commercially-available, address-matching software packages. The effectiveness of service bureaus' matching software can also be measured.

There are many vendors selling CASS-certified tools and services. Organizations use CASS-certified tools for address standardization, correction, and validation. End of story, right?

Wrong. Some organizations use many CASS-certified tools for address standardization, correction, and validation, at different places along the processing stream. The addresses are standardized, cleansed, and validated (or not) multiple times. The addresses are changed from their original format, manipulated, and then shoved back into the databases, without considering the actual process dependencies or expectations.

And then you end up with a scenario like this: a process for accepting customer applications including their self-provided addresses, send hard copies of acknowledgements to their self-provided addresses, yet the process includes an elaborate mechanism for managing returned mail. That did not make sense to me: if the organization sends out acknowledgements to the address the customer provided, wouldn't they trust that the customer provided an accurate address?

In fact, the issues was self-created: the provided address passed through at least three different iterations (with different products!) of standardization, correction, and validation, and was transformed from a deliverable ("accurate") address to an invalid one.

So even though the intent was appropriate, the execution of the process got in the way of the results. So I'll throw out two questions: First, is address standardization and validation a tool or a process? And second, at what point and how frequently in the business process flow should address standardization and validation take place?

]]> Cleansing Your Data Using SQL Server 2012 Data Quality Services (DQS) and Melissa Data tag:blog.melissadata.com,2012:/data-quality-authority//2.173 2012-03-26T21:35:07Z 2012-03-26T21:51:44Z

Check out this video featuring a step-by-step/start-to-finish approach to cleanse and enrich address data using the new SQL Server 2012 Data Quality Services (DQS), and Melissa Data's Datamarket Address Check service. From the video, you will learn how to create...

Blog Administrator
Click Here to watch video.

]]> The Relative Distinction of Address Validation, Precision, and Accuracy tag:blog.melissadata.com,2012:/data-quality-authority//2.172 2012-03-20T18:10:46Z 2012-03-20T18:17:37Z

By David Loshin One nice thing about addresses, especially in the United States, is that they have well-defined standards. In previous blog series, I have looked at the process of address standardization and correction, so I won't belabor that...

Blog Administrator David Loshin

One nice thing about addresses, especially in the United States, is that they have well-defined standards. In previous blog series, I have looked at the process of address standardization and correction, so I won't belabor that point. However, many people confuse the differences among a valid address, a precise representation of an address, and an accurate address.

First of all, recall that the main reason for standardizing addresses is based on "delivery" - supporting the general expectation that posted items reach their desired destination. Validation that addresses meet the standard certainly can improve the delivery processing, especially when it comes to sorting the items to be delivered. That is the reason that the Post Office offers discounts in the costs of mailings when the addresses are validated in standard form.

Luckily, there is a lot of leeway when it comes to meeting this expectation. If you provide a street number, street address, city, and state, there is a good chance your item will be delivered even if it is missing a ZIP code.

On the other hand, the address is not necessarily valid in terms of it meeting the USPS standard, since it is an incomplete address. Similarly, what was put into "address line 1" and "address line 2" might have been the reverse of what is specified by the standard (does "suite 2100" go before the street address or not?), but the carrier will still be able to put the letter into the right slot once she gets to the building. In other words, the address does not need to be valid in order to be delivered.

Nor does it really have to be that precise. We might define precision to mean that the address has enough information to direct the carrier to the specific box or slot associated with the entity referenced in the address. An address with a name, a street, and a suite number is more precise than one with just a name and a street. Yet if your local carrier has some knowledge about his route, then items that have less than precise addresses will still get to their destination.

The last term may be the most important one, though: accuracy. If the address associated with the named entity is not correct, the chances of the item being delivered are much lower. And one problem is that these characterizations are not mutually exclusive: an address can be valid (i.e. it is in standard form and there is an actual delivery point associated with it) but not accurate (if the named individual is not associated with the delivery point).

And while the Post Office maintains some information and tools to help synchronize validity and accuracy, sometimes there are kinks in the process, as we will see next time.

]]> Do You Need a Single Version of the Truth tag:blog.melissadata.com,2012:/data-quality-authority//2.171 2012-03-16T18:09:24Z 2012-03-16T18:29:54Z

In a world in which data is flooding into organizations from all directions, it has long been axiomatic that companies should try to develop a "single version of the truth." The idea is that a customer profile, for example,...

Blog Administrator

The impetus to build a single version of the truth comes from many sources. But perhaps the most important is the commonsense notion that companies cannot truly understand their customers, or their performance, or market trends or their operations, unless all the data associated with a given operation is available to all the interested stakeholders. To use even more jargon, data silos must be broken and data integrated across the enterprise.

The classic examples demonstrating the need for a single version of the truth include these: Customers who look great to the sales team, but look terrible to accounts receivable because they are habitually slow payers. Or a supplier is preferred by the logistics group because of its ability to deliver goods just in time, but is an anathema to quality control because of quality issues. Issues like these can be identified before the next sale or purchase is made if everyone in the organization has access to a single version of the truth for a customer or supplier.

The challenge to creating a single version of the truth is two-fold. First, completely integrating all the data about an entity such as a customer, product or even many operational processes is probably beyond the reach of most master data management programs. Sure transactional data can be combined with external data from third parties--think credit reports here--and perhaps even semi-structured data like email. But it isn't easy. And when you add in the local knowledge people have captured in their personal office applications, the task becomes even more daunting.

The second challenge is conceptual. Using the term a "single version of the truth" implies that you can arrive at that destination. But business activities are ongoing. Customers can be great customers until they aren't anymore. Instead of striving to create a single version of the truth, organizations would be better served by building data-robust entities. We can never actually arrive at a single version of the truth. We only can know what we know now and need processes to integrate what we will learn tomorrow.

]]> Synergizing the Process for Locations and Addresses tag:blog.melissadata.com,2012:/data-quality-authority//2.170 2012-03-15T18:04:17Z 2012-03-15T18:20:07Z

By David Loshin I am currently working with a number of clients who are dealing with particularly thorny issues relating to location. While the business drivers are relatively diverse, there are some similarities across all scenarios, especially in the...

Blog Administrator David Loshin

I am currently working with a number of clients who are dealing with particularly thorny issues relating to location. While the business drivers are relatively diverse, there are some similarities across all scenarios, especially in the ways that location is managed from an enterprise perspective. Therefore, in this set of blog entries, we'll look at different

business value drivers, typical usage scenarios, and some ideas for melding process with application for a synergistic lift.

Location is a critical concept in many industries, yet the importance of a standardized approach to managing location is often unnoticed. For example, for some client scenarios, the business driver is reducing risk. Insurance companies like to see their customer base (and therefore, their exposure to certain types of hazards such as floods or earthquakes) spread across different geographic regions. Financial institutions may be subject to different laws (with different penalties for violations) at different geographic jurisdictional levels for the protection of private information.

Healthcare providers need to ensure that protected health information is not inadvertently exposed by being sent to the wrong address.

In other scenarios, the business drivers are financial, such as focusing on customer acquisition, retention or cost management. In some cases, marketing budgets are allocated to local print and media advertising to grow the customer base. In other cases, reducing transport costs by optimizing the supply chain looks at distances between delivery points.

Either way, the underlying desire is precision and correctness in geolocation.

But note that precision and correctness are very separate ideas, and that difference underlies some of the main problems related to address and location management. Next post: Accuracy vs. Precision.

]]> Why You Need a Data Audit tag:blog.melissadata.com,2012:/data-quality-authority//2.169 2012-03-02T22:17:13Z 2012-03-02T22:22:55Z

Everybody makes mistakes and those mistakes have consequences. As enterprises rely more heavily on data to make decisions and drive processes, the quality of that data becomes more critical to the overall success of the organization. In many cases,...

Blog Administrator

In many cases, the impact of bad data is not too hard to identify--direct marketing campaigns with high numbers of email bounce-backs or undeliverable mail; marketing efforts with poor response rates because the offering has not been correctly tailored to address the "wants and needs" of the customers; prospects who turn away because their names are spelled wrong or other information is incorrect.

With the growing need to integrate data from so many sources, both internal and external as well as the dynamic nature of business itself, the chances of introducing mistakes into critical databases is growing exponentially. It's not really a question if there are mistakes in your databases; the question is how many mistakes are there and what kind of negative impact will they have on your business?

The answer to those questions can be determined by conducting a data audit. A data audit is just what the name implies--a systematic look at data to insure that it is what it is supposed to be.

Conducting a data audit is pretty straightforward. First, determine which records and which fields should be examined. If customer records are to be audited, fields such as names and addresses perhaps; buying history, payment history or other critical fields may be included. The field selected should be those associated with a specific process or activity.

Then assess the fields to determine which do not conform to the data dictionary and the business rules that govern them. A wide range of errors may surface. Values may be missing. Values (such as dates) may fall outside of a specific range. Fields may be incomplete or formatted incorrectly.

The next step is to determine the source of the error; how the errors can be rectified (if, in fact, they need to be rectified); and how mistakes can be minimized in the future. Among the most common sources of errors are faulty data conversions and inconsistencies in integrating data from different sources. Not every error always has to be fixed or the underlying process changed. In most situations, there will always be some errors in the data that can be tolerated because they do not have a crippling impact on the business activity being supported.

How often should data audits be conducted? Not surprisingly, the answer is elastic. Data audits should be conducted often enough that faulty data does not impede reaching your business goals.

]]>