Data Quality Rules: A Management Primer for Taking Action
What’s the Problem With Data Quality Rules in Your Organisation?
Did you know that right now in your organisation there are typically thousands of data quality rules floating around your applications, data feeds, web forms and other information gateways.
They’re mostly hidden of course, squirrelled away in the depths of ageing software and scripts, created by coders, data pros and subject matter experts (most of whom moved on a long time ago).
The problem with this data quality rule real estate is that:
It’s very difficult to manage in a strategic fashion
Rules get out of date very quickly
Overall it represents a huge cost-base
Standardisation and re-use are practically non-existent
Stewardship and ownership are practically impossible
Tip: if your executives say your company doesn’t need to invest in a data quality rules capability you can tell them that in actual fact they are already paying for one. It’s horribly disjointed, ineffective and costly but it does exist!
Data Quality Rules Management : Why Hasn’t the Organisation Solved it by Now?
There is no clear answer to this question as the reasons are varied.
In case your execs quiz you as to how your organisation got into its present situation, here are some comeback responses:
Application rules are inaccessible, often locked away in 3rd party applications
Experts move on, taking local rule knowledge with them
No-one cares, if the systems are (mostly) working so what?
It’s a complex challenge made difficult by silos and politics
So lots of reasons why it hasn’t been done but perhaps a better question should be – why should it be done?
What are the Benefits of Managing and Re-Using Data Quality Rules Strategically?
I’ve created two very different scenarios to demonstrate the value offered by a strategic approach to data quality rules management.
Scenario A – Haphazard Inc.
Company A has no data quality strategy or governance in place to manage their contact data. As a result it’s a veritable free-for-all. Any new apps follow their own standards (if they even exist). There is no coordination of rules and certainly no error checking beyond some basic mandatory field checks.
Each time a new web form is created, the development team simply use whatever coding language the individual prefers and whatever checks and balances they’ve been taught.
When contact data is moved around the organisation there are no checks for standard data quality dimensions and certainly no checks for context-specific rules that have been stored centrally.
Any defects are picked up and resolved locally with quick, reactive fixes. Upstream systems and data owners are not informed.
No-one in management has visibility of their contact data quality and there is no central reporting of contact data quality levels by business function, application, customer type or other value-added segments.
Year-on-year the cost of bad data quality increases as data volumes grow and the administrative burden of data quality mis-management increases.
Scenario B – Focused and Productive LLC.
Company B has a formal data quality strategy and a series of contact data governance policies in place for ensuring correct management of contact data quality and associated rules.
Whenever new applications or coding requirements are suggested, all requests are routed via the Data Quality Officer so their team can coordinate a response and ensure the correct standards and policies are required. Development and business teams don’t need to store hundreds of policies, the central data quality team simply provide the latest and most accurate resource information.
Over time Company B has created a rich set of data quality rules in a hierarchical format that allows anyone to quickly navigate and copy the relevant rules for the contact data information they wish to store, retrieve, process or amend.
All suppliers and 3rd party IT providers have to adhere to the data governance standards for data quality and contact data application design. RFP’s and tenders reflect these requirements and new applications are marked down when flexible rule design cannot be enforced.
When contact data is moved around the organisation there are automated checks to ensure that the data has not been corrupted. For example, ETL processes utilise corporate standards for contact data checks and where possible calls are made to a data quality API that the data quality team created with their data quality software platform.
Any defects observed are first recorded by the local data steward so that they can apply local knowledge as to the severity of the issue. Local fixes can be applied but the typical course of action is to notify the central data quality team who then route a change request through to the appropriate team on their register so that upstream preventative or tactical fixes can be applied.
Using a central data quality assessment and monitoring strategy means that “repeat offenders” can be picked up quickly and longer term, proactive resolutions implemented.
Senior management receive a personalised report each month of progress in their area. They can quickly drill through into more detailed information to observe how well systems, data sets and even individual users are managing the different types of data, including contact data, within their area of jurisdiction.
Year-on-year the cost of poor quality contact data is coming down sharply as the most common issues are routinely eliminated and prevented from recurring.
Which company would you rather work for?
Data Quality Rules: What Can You Do To Get Started?
Get some buy-in
Firstly, you need to discuss this problem with your superiors and explain some of the traditional costs involved vs the new way of working.
Explain how much of an administrative overhead the old style approach creates. Give some financial examples of how often support teams have to improve contact data manually or field calls from irate customers where information has gone wrong.
Create some A/B comparisons
Put an outline cost together and then outline Scenario A and Scenario B to executives. Explain how you’ll never become a world-class company if you can’t apply the same quality principles for data that your bosses invested in production, workflow and employee training.
Plot the path ahead
Create a simple roadmap initially. Focus on data quality rule discovery, collecting the core rules for your most important data sets. Some rule hierarchies such that people always have a starting point but can go deeper if they require.
For example, your top level hierarchy may well be record completeness – does the contact data have sufficient data to perform a legitimate postal delivery.
Next you may focus on attribute completeness – does the contact data have a complete postal code or email address?
Next you may want to check for accuracy – does the address map against a surrogate trusted source? Can the email address be pinged to ensure it exists?
Transition into a central resource (think library)
Over time create richer and richer rules and publish them centrally. Work with development teams and suppliers to ensure that they use your rule library to enforce standards.
Monitor for continuous improvement (and justification)
Create some initial data quality monitoring processes at key contact data collection points. Spot repeating failures and feed back to the development team so that you can work collectively to create long-term preventions.
Demonstrate the savings even with these simple improvements and push for a longer-term data quality technology and data governance investment.
I know this is a somewhat simplified view but I’m aiming this article at management, not the tech community. There are scores of articles on Data Quality Pro to help you develop the individual concepts further or just contact me (editor@dataqualitypro.com) if you need specific examples.
Don’t underestimate the benefits of re-using and maintaining data quality rules. They can dramatically improve the reach and productivity of your data quality efforts across the whole organisation.
Data Quality Rules: Next Steps for Expansion Within Your Organisation
Check out Data Quality Rules: The Definitive Guide it lists all of the top data quality rules articles on Data Quality Pro and a whole bunch of other resources besides.
Learn more from experts in the Data Quality Pro Virtual Summit