http://www.melissadata.com/success/index.htm#Data-Quality
That's why many companies choose to incorporate data quality directly into their data integration architecture. And to be successful, companies should opt for a program that is flexible and agile to meet evolving business requirements.
Microsoft SQL Server provides such a solution - SQL Server Integration Services (SSIS). This service offers a scalable enterprise data integration platform with exceptional Extract, Transform, Load (ETL) and integration capabilities - enabling organizations to more easily manage data from a wide array of sources, including XML data files, flat files and relational data sources.
And, increasingly, more third-party connectors are becoming available which helps further extend the power and capabilities of SSIS, including the ability to cleanse, standardize, dedupe and enrich data seamlessly within the pipeline.
SSIS also is a cost-effective way to enable these data cleansing and enrichment tasks - as it does not require the use of monolithic applications that would exert a lot of time, money and man-power to install and maintain.
Here are some examples of new packages available for SSIS:
This package is a collection of transforms that can be integrated into a workflow to clean, validate and update all kinds of contact data including addresses, phone numbers, email addresses, etc.
Use this package to parse and restructure data into a common format to assist in maintaining consistent data. This package can standardize addresses to a desired format, or to USPS® specifications, which is required for certain postal discounts. This package also includes geocoding, which enriches the value of data by appending geographical information that can help improve everything from marketing to logistics. Geocoding adds latitude/longitude coordinates to a postal address, as well as the Census tract and block which can be used to find additional demographic information. These, and other packages, can be easily dropped into SSIS as native components and plugged into workflows to cleanse and enrich data during ETL processes.
The deduplication of data from disparate sources is an ongoing challenge for organizations of all sizes. This package allows users to compare any data type of any length and from any place in a field to find duplicates fast. The program contains 12 "fuzzy" matching algorithms to find even the not so obvious duplicates - for example, 'Beth Smith' at UDM and 'Smithe, Elizabeth' at United Data Machines. Detecting the most duplicate records helps streamline databases, improve marketing efficiency and achieve a unified accurate view of the customer.
Profiling helps you develop a clearer picture of your data. This package includes several profiling tasks, including column statistics, value distribution and pattern distribution. These tasks analyze individual and multiple columns to determine relationships between columns and tables.
Monitoring Ongoing Data Quality
Databases that contain contact data are never static, because information is changing all the time as customers move, change companies, die or divorce. And new or possibly inconsistent data is coming in all the time from call centers, Web forms and data entry by various departments. That's why it is important to add in a monitoring package in SSIS - to provide real-time data monitoring in an automated process to immediately recognize and correct issues before the quality of data declines. This approach also helps organizations enforce data governance and compliance measures.
Taking advantage of affordable and easy-to-integrate third-party data quality and enrichment packages can help organizations extend the capabilities of their investment in SSIS, helping expand the meaning between data sets, consolidate and validate contact data, and better synchronize business processes through a single view of master data.
We introduced our new SSIS Toolkit of data quality transforms at the PASS Summit in Seattle this week to positive response. Total Data Quality Integration Toolkit (TDQ-IT) is the most affordable, full featured enterprise data integration platform to help IT professionals integrate increasingly complex enterprise data - in all its different formats, structures, and sources, and deliver a single view of critical information assets.
]]>For my inaugural blog I would like to direct you to Jim Harris Obsessive-Compulsive Data Quality post on Data Profiling. He makes several important points regarding the techniques and methodologies in Data Profiling, and since I am not vendor neutral I would point out we provide all the capabilities Jim describes and more for SSIS. Also I recently posted an article on Fuzzy Matching in TSQL on SQL Server. I willbe covering Data Quality, MDM , SSIS and Matching extensively both DIY(Do It Yourself) and using our SSIS Components (Jaro Winkler , nGram) and our MatchUp API(12 algorithms) soon to be released.
Ira
]]>
What is a Web Service?
Software applications written in various programming languages and running on various machine platforms can use Web Services to easily exchange data over computer networks like the Internet. A Web Service is an application that uses open standards for exchanging data between applications or systems.
Sending your data to the Web Service to be validated, verified, cleansed, standardized, and appended with additional information is the next step in the data quality process. The data to be validated is formatted and sent to the Web Service. Once the Web Service processes the request, the data is then returned to you in the same format in which it was sent
Real-Time communication between your Web Services provider and your systems is independent of the programming language or operating system you use. This is usually made possible by using two open standards that are platform independent and vendor neutral, XML and SOAP:
· XML (Extensible Markup Language) is a flexible way to create common information formats and share both the format and the data on the Internet, intranets, and elsewhere.
· SOAP (Simple Object Access Protocol) is a standard protocol for exchanging messages over a computer network built on top of a standard XML format. It is an extensible foundation where complex structures can be used.
Advantages of using a Web Service
Using a Web service has two major advantages: it may be a less expensive option if you process a relatively low number of records; and it eliminates the effort associated with maintaining and updating - it's handled by the Web Service vendor.
Web services, such as a data quality Web service, allow a company to protect their database behind a data quality firewall. Data coming in via call center or online shopping cart is verified and corrected at point of entry - before it enters your CRM system.
Data quality Web services also protect your data from inadvertent errors made by your own data entry personnel. The ability to check, correct and standardize an address "on the fly," before it is even stored, will serve to minimize bad contact data including inaccurate street and email addresses, vulgar names and incorrect phone numbers.
Technology Assessment for Data Quality Web Services
Deciding upon an approach to data quality requires a thorough and realistic appraisal of the technology and resources available within the organization. Ask yourself the following questions:
1. If data quality is to be enforced via a Web site, is it feasible to integrate the technology into the Web site or will the site need a major overhaul to make it happen?
2. Does the volume of addresses to be verified make a Web Service the most economical solution?
3. What would be the best way to ensure that all address data passes through a data quality process before it is used in any way? Where in the business process does data quality fit best or does a new step need to be introduced for data quality.
4. Does the organization have the programming resources to integrate a data quality tool into an existing application?