Recently in Data Warehouse Category

When MDM consolidation is too successful

| No Comments | No TrackBacks

GDPR compliance should mean revisiting--and cleansing--your data files

Want to know the main reason you're liable to run afoul of the new European Union General Data Protection Regulation? Because your Master Data Management view of data across applications and analytics may have been just a little too successful.


Companies that have been extra diligent in that good old "360-degree view of the customer" have a master file chock-full of customer information, all of which is consolidated, but much of which is out of date and duplicated across systems.


And that means your MDM processes are probably filled with rule-busting red flags, courtesy of the new GDPR regulations.


Let's unpack this a bit, examine the key regulatory changes, and see where you're probably at risk.


The EU's GDPR, which becomes effective May 25, addresses consumer concerns about safeguarding digital information, regardless of where in the world it's stored, and erasing it when requested. If you've ever done business with people in any EU country, their information is in your CRM system, marketing automation, or master customer information file.




What data are we talking about here? Just about anything, which pretty much summarizes your potential problem.


Newly protected data not only includes name, address, email, and phone, but also photos, bank details, social networking posts, and individual computer IP addresses. CRM and marketing automation systems normally include birthdays, payment methods, URL visits, and shopping habits, regardless of whether the information is about a person's private, professional, or public life, and those are GDPR-protected as well.


Running afoul of the GDPR is no small matter. It can result in fines of up to 20 million euros--roughly $24.5 million--or alternately 4 percent of a company's entire annual global revenue.


On the upside, the Euro folks are more serious than ever about their citizens' data and privacy.


On the downside, trying to identify your organization's relevant exposure can become an intolerable burden.


While the complete GDPR is extensive, a key element is Article 17, detailing consumers' "Right to Erasure." GDPR requires you to erase any EU citizen information upon request "without undue delay," and with only a handful of exceptions.


Surveys indicate that this Right to Erasure is the most challenging requirement for businesses. And this is where your Master Data Management projects are most vulnerable.


The problem is that data management processes that supposedly have merged siloed files across applications are geared toward inclusion, not efficiency.


It's likely you've got tons of duplications--numerous iterations of the same person, each containing slightly different fields, conflicting data, and even errors from one entry to another. Small wonder: Data entries are made individually over years and via different processes.


Even if entries are flagged, typical MDM false negative results mean truly duplicated records will remain behind, and uncorrected. After all, nobody wants to run the risk of erasing a unique and valid contact.




Consider: You receive a request to have an individual's data erased, you think you've  complied, but in reality have overlooked iterations across applications that are of the exact same contact.


The result: A regulatory nightmare and a potential financial disaster.


What's a chief data officer to do?


First, start with an internal audit conducted by Melissa, one that goes far beyond any Privacy Impact Assessment process you may already have in place. A thorough top-down approach will uncover problems and trends, determine your true risk, and provide recommendations for redress.


But you'll want to go further.


Melissa technology tackles databases to standardize, correct, complete, and verify customer records. If there are any remaining identify questions about who's who in your database, and if any duplications continue to exist, Melissa employs matching engines to resolve them.


You've done your best to assure that 360-degree view of the customer through optimal MDM processes, and congratulations to you. Now, take the next step to protect your company from regulatory snafus.


To ensure you're compliant with the new GDPR regulations, call Melissa.

Record Matching Made Easy with MatchUp Web Service

| No Comments | No TrackBacks

MatchUp®, Melissa's solution to identify and eliminate duplicate records, is now available as a web service for batch processes, fulfilling one of most frequent requests from our customers - accurate database matching without maintaining and linking to libraries, or shelling out to the necessary locally-hosted data files.

Now you can integrate MatchUp into any aspect of your network that can communicate with our secure servers using common protocols like XML, JSON, REST or SOAP.


Select a predefined matching strategy, map the table input columns necessary to identify matches to the respective request elements, and submit the records for processing. Duplicate rows can be identified by a combination of NAME, ADDRESS, COMPANY, PHONE and/or EMAIL.


Our select list of matching strategies removes the complexity of configuring rules, while still applying our fast and versatile fuzzy matching algorithms and extensive datatype-specific knowledge base, ensuring the tough-to-identify duplicates will be flagged by MatchUp. 

The output response returned by the service can be used to update a database or create a unique marketing list by evaluating each record's result codes, group identifier and group count, and using the record's unique identifier to link back the original database record.


Since Melissa's servers do the processing, there are no key files - the temporary sorting files - to manage, freeing up valuable hardware resources on your local server.


Customers can access the MatchUp Web Service license by obtaining a valid license from our sales team and selecting the endpoint compatible to your development platform and necessary request structures here.

Fintech breakthroughs that identify,validate and augment customer data help financial services organizations head off fraud, operate within the rules

One of the most pressing issues facing the financial services industry today is the prevailing emphasis on security and fraud prevention. Big data and analytics are leveraging the power of the Internet, but also offering big, juicy plums for hackers, credit card fraudsters, money launderers and terrorists.


In response, many countries including the U.S. have established Know Your Customer (KYC) requirements, intended to guide financial institutions in heading off fraudulent transactions. A key way for banks to do this is to clearly Identify legitimate banking clients and their business relationships so the bad operators become more obvious and identifiable.


But it can be a challenge. Legitimate customers often have multiple banking relationships with a single institution, with identifying information stored in different formats in a multitude of databases. Various family members also may be account owners, but have different names and live at different addresses, possibly even in different countries.


Linking all these threads together while at the same time correcting name misspellings, standardizing mailing address formats, and parsing precise geolocation IDs can be the stuff of banking compliance nightmares.


Not doing this due diligence, however, can be catastrophic. To be hacked and have millions syphoned out of your customers' accounts is one thing, but to have the government ready and willing to fine you for failing various KYC tests can be just as damaging. Can you say "Loss of reputation and customers?"  



Most notably non-compliant of late have been banks based or doing business in India, but it's a worldwide problem. A New York-based broker-dealer recently was charged with KYC violations and for negligently allowing illegal trading by one of its customers. Perhaps the highest profile case most recently was Morgan Stanley Smith Barney running afoul of a variety of KYC violations. The main charge: Morgan Stanley failed to properly identify an assortment of "red flags" that signaled illegal activity.


Thankfully, technological breakthroughs increasingly are offering their own solutions. Just as the digital world has enabled bad players and victimized banks worldwide, technology is fighting back with sophisticated Know Your Customer tools.


A big step forward is the ability to accurately verify names and addresses. In a global world without borders, technology that verifies, cleans, completes and standardizes names, addresses, phone numbers and emails, and does so globally, immensely aids the process of knowing one's customers.


Technologies also exist that adds in missing contact fields, finds census and area-specific statistical details, and provides precise demographic information. When banks are able to combine census and area-specific details with accurate names and addresses, they'll know pretty closely if a variant player is really a customer or a bad guy.


KYC guidelines are very specific about risky areas prone to scams and schemes. A wide variety of countries are identified by the U.S. State Department for being prone to ignoring money laundering, tolerating suspicious transactions, and generally lacking adequate know-your-customer requirements. Ignoring this, in fact, was one of the issues that burned Morgan Stanley, permitting criminal activities to continue unchecked.


While IP lookup that identifies exactly where a digital communication has come from has been around for a while, new geocoding breakthroughs are able to convert IP addresses into precise latitude and longitude coordinates around the world. Most European and international identify cards also can be verified, along with mobile phone numbers and driver's license information.


One of the essential elements in all this is updating customer data. It's been estimated that accurate contact information deteriorates severely and regularly, and good customers who merely move across town can confuse inadequate screening processes and raise red flags (false positives) when it shouldn't. Today there are a variety of modules banks can use to update customer information quickly and accurately.


The bottom line is this: Financial services institutions now have lots of new reasons to love fintech technology that mitigates KYC concerns by identifying legitimate customers, and flagging the ne'er-do-wells before they can be effective.

Meet Melissa: Global Intelligence

| No Comments | No TrackBacks

Melissa Data is officially now Melissa.


As we welcome our 32nd year in business, we are excited to announce an important change at Melissa Data. We've decided to drop the "Data" from our brand identity. We are simply Melissa now. This is part of a new branding effort to reflect Melissa's growth, and more importantly, the changes in the data quality space. While authoritative data sources power our products and services, we want to continue developing new solutions that deliver data-driven results for better business intelligence.


This forward thinking change is reflected in our new logo with the design emphasis on the "i" for intelligence. You will see this focus on intelligence in our new ID verification services, our industry-specific solutions to help with Know Your Customer initiatives, risk management and compliance, and in our robust customer data management and data integrations platforms.

And, you'll see it in our new website at 

Our goal with this new website is to provide our visitors an easier way to learn about Melissa's services and solutions. Immediately, you will notice streamlined menus, simple navigation, and quick access to the information you need.


We look forward to working together with all of our existing customers on more opportunities and better solutions for global intelligence. Please feel free to reach out and let us know how we can better assist you.

Flagship SSIS Developer Suite Now Enables Data Assessment and Continuous Monitoring Over Time; Webinar Adds Detail for SSIS Experts

Rancho Santa Margarita, CALIF - March 17, 2015 - Melissa Data, a leading provider of contact data quality and address management solutions, today announced its new Profiler tool added to the company's flagship developer suite, Data Quality Components for SQL Server Integration Services (SSIS). Profiler completes the data quality circle by enabling users to analyze data records before they enter the data warehouse and continuously monitor level of data quality over time. Developers and database administrators (DBAs) benefit by identifying data quality issues for immediate attention, and by monitoring ongoing conformance to established data governance and business rules.

Register here to attend a Live Product Demo on Wednesday, March 18 from 11:00 am to 11:30 am PDT. This session will explore the ways you can use Profiler to identify problems in your data.

"Profiler is a smart, sharp tool that readily integrates into established business processes to improve overall and ongoing data quality. Users can discover database weaknesses such as duplicates or badly fielded data - and manage these issues before records enter the master data system," said Bud Walker, director of data quality solutions, Melissa Data. "Profiler also enforces established data governance and business rules on incoming records at point-of-entry, essential for systems that support multiple methods of access. Continuous data monitoring means the process comes full circle, and data standardization is maintained even after records are merged into the data warehouse."

Profiler leverages sophisticated parsing technology to identify, extract, and understand data, and offers users three levels of data analysis. General formatting determines if data such as names, emails and postal codes are input as expected; content analysis applies reference data to determine consistency of expected content and field analysis determines the presence of duplicates.

Profiler brings data quality analysis to data contained in individual columns and incorporates every available general profiling count on the market today; sophisticated matching capabilities output both fuzzy and exact match counts. Regular expressions (regexes) and error thresholds can be customized for full-fledged monitoring. In addition to being available as a tool within Melissa Data's Data Quality Components for SSIS, Profiler is also available as an API that can be integrated into custom applications or OEM solutions.

Request a free trial of Data Quality Components for SSIS or the Profiler API.
Call 1-800-MELISSA (635-4772) for more information.

News Release Library

New Company Magazine Features Data Quality Insights on Merging Duplicate Patient Records into a Golden Record, also Tips on Improving Healthcare Data Warehousing

Rancho Santa Margarita, CALIF. - September 9, 2014 - Melissa Data, a leading provider of global contact data quality and data enrichment solutions, today announced matching and de-duping functionality that solves duplicate records for healthcare database administrators (DBAs). Using tools based on proprietary logic from Melissa Data, healthcare DBAs can consolidate duplicate customer records objectively, unlike any other data quality solution. This and other healthcare data quality challenges are featured in Melissa Data Magazine, the company's new quarterly resource for DBAs and data quality developers.

Healthcare data is characterized by a steady stream of patient records and evolving contact points, warranting a smart, consistent method to determine the best contact information. Melissa Data Magazine highlights a new way to merge duplicate records, based on a unique data quality score that retains the best pieces of data from all of the various records.

"It's essential that healthcare data managers acknowledge data quality challenges up front, implementing processes to cleanse and maintain the trustworthiness of the information that goes into their master data systems," said Bud Walker, director of data quality solutions, Melissa Data. "Our new publication outlines how to ensure this high level of data precision, creating an accurate, single view of the patient. This is known as the Golden Record and is of critical value in healthcare settings - reducing costs, streamlining business operations and improving patient care."

Highlighting industry-specific data quality tools and solutions, Melissa Data Magazine will help DBAs and health information managers adapt to evolving challenges particularly as data becomes more global in nature. Future published issues will feature technologies such as SQL Server development tools, and markets such as retail, ecommerce, government and real estate.

Melissa Data Magazine will be available at the American Health Information Management Association (AHIMA) conference, Booth #723, starting September 27 in San Diego, Calif. Click here to download the healthcare issue of Melissa Data Magazine, or call 1-800-MELISSA (635-4772) for more information.

News Release Library

Justifying Data Quality Management

| No Comments | No TrackBacks
By David Loshin

Last week I shared some thoughts about a current client and the mission to justify the development of a data quality program in an organization that over time has evolved into one with distributed oversight and consequently very loose enterprise-wide controls.

The trick to justifying this effort, it seems, is to address some of the key issues impacting the usability of data warehouse data, as many of the downstream business users often complain about data usability, long times to be able to get the data for their applications, and difficulty in getting the answers to the questions they ask.

The issues mostly center on data inconsistency, timeliness of populating the analytical platform, completeness of the data, the rampant potential for duplication of key entity data due to the numerous data feeds, long times to figure out why the data is invalid, and general inability to satisfy specific downstream customer needs.

Because all of these issues are associated with the context of day-to-day reporting and analysis, we have inferred that the common theme is operational efficiency, and that is the dimension of value that we have chosen as the key for establishing the value proposition. Therefore, our business justification focuses on the data quality management themes that would resonate in terms of improving operational efficiency:

  • Improve proactive detection of invalid data prior to loading into the data warehouse
  • Speed the effort to finding and analyzing the source of data errors
  • Make the time to remediate data issues more predictable
  • Better communicate data errors to the data suppliers
  • Provide business-oriented data quality scorecards to the data users
The goal is to provide reporting with creditable statistics that demonstrate the efficiency of validating data, finding errors and fixing them, and delivering measurably high-quality data.