We’re encouraged to treat data as an asset, so why does it so often end up posing a risk to organisations?
The volume of data worldwide is growing exponentially, yet it’s estimated that only 10 per cent is unique data while the remainder is replicated information. Although this might not be quite true of your data, there’s a strong likelihood that the combination of complex information architectures and the presence of unstructured data will result in high levels of duplication and other quality issues.
The Regulator of Social Housing has recently spelt out the necessity of good data, stating, “Boards must have assurance that data is appropriately managed. This will require adequate quality controls and robust audit trails…. implementing appropriate software solutions such as error detection.”
The risks associated with poor-quality data can be far-reaching. If your compliance data is of poor quality, there’s a risk that you will unwittingly cause serious harm or even death to your residents, their friends or your colleagues.
If your personal data has become obsolete or you collect data you don’t need then you will be in breach of the GDPR, potentially causing reputational damage to your organisation, as well as risking a fine from the ICO. And sometimes, someone finds themselves out of a job.
As a former housing officer, I remember the one-bedroom flat that suddenly developed a second bedroom as a colleague was showing the prospective tenant around; the three-bedroom house that had been left unoccupied for 18 months because it had never appeared on void reports; and a tenanted dwelling that dropped off the rent roll for three years before anyone noticed.
Housing downgrades
A review of downgrade judgements reveals that almost all downgrades are linked to inaccurate or poor information. Indeed, the Regulator of Social Housing stated, “Good-quality data forms the cornerstone on which all other assurance of compliance is based, and we would expect registered providers to seek assurance on the quality and integrity of their data in the course of their business.”
Of course, no one sets out to have poor-quality data, and many organisations would say that they minimise the risk by running exception reports and are therefore confident their data is in good shape. But can they be certain of that? We find that while some colleagues will defend data quality, others in the same organisation will be sceptical and will seek further assurance that the data their decisions are based on is robust.
From SQL to 3C Data Logic
Until 2020, whenever we helped people with data-quality reviews, we would use SQL to audit and assess the quality of data, and we would often just check selected sections of data. But in 2020, we decided to look for a tool to support our work. We looked at two tools, one was the market leader and the other was still in its infancy; we chose the latter. Now branded as 3C Data Logic, we chose it because it uses low code to create rules, and because it can investigate unstructured data in emails, document management systems and shared drives as easily as it does structured data in databases.
3C Data Logic has made a massive difference to the delivery of our data services. We used to support customers to: analyse and audit data; curate and clean the data; create new processes; and restore data confidence.
But we would then hit a snag because data typically degrades at two per cent every month or 25 per cent annually. So having cleaned the data, and even with improved data validation on entry, it was always going to get dirty again. Using 3C Data Logic, our customers can continue to monitor their data, ensuring that its quality is always known and assured.
Bespoke explorations
Our staff love the tool because it allows them to focus on what really matters to our customers. And while SQL is needed to pull the data into the tool, once in, the tool itself uses low code, which means that we can investigate the data with our customers in real time. If a customer wants to explore a bit more, we can just do it in front of them, or we can train their business teams to do it themselves; there’s no need for SQL or other technical skills.
Customers analysing data quality using 3C Data Logic have discovered a wide range of issues, including:
- One local authority found over 2,000 tenancies where the rents were illegally set too high and a further 16,000 tenancies where the rents were set too low.
- One housing provider discovered 40 properties with gas appliances listed in one system, which were not having gas services because this was run from another system, while another landlord found 360 (this is a common issue that we have found at all the housing providers we’ve worked with).
- One landlord unearthed 30,000 obsolete people records in their housing system.
- One housing provider found tenants with invalid NI numbers, while another had almost 5,000 tenants with no NI number at all.
- There are often problems with telephone numbers; one housing provider discovered 1,200 invalid telephone numbers (due to being too long, too short or having text in the number field, prohibiting SMS contact).
Additionally, one customer has found that with all its data collated in 3C Data Logic, it doesn’t need a data warehouse and is replacing its KPI system with PowerBI dashboards reporting data held in 3C Data Logic.
As one director of housing and corporate services (and a 3C customer) said, “When we read what others had said about 3C’s Data Logic tool, we were sure that this could play an important part in helping us to achieve our ambition of good quality, reliable data. The initial proof of concept showed us what we could achieve and now we can manage data quality ourselves using 3C Data Logic.”
If you think your data might be more of a risk than an asset, please contact info@3cconsultants.co.uk to arrange a discussion or demonstration.
Claire Bayliss is the chief operating officer of 3C Consultants.
