In the second part of a two-part article (orig. planned as a single feature), Housing Technology quizzed Askporter and Howell Technology Group about dark data and how and why housing providers collect, process and store it yet don’t (or can’t) use dark data for insights or decision-making.
What is dark data?
Askporter’s head of business development, Ben Yexley, said, “Dark data is the vast amount of information that organisations collect, process and store during regular business activities but fail to use for other business-efficiency gains.
“In housing, this can include data types such as resident communications, maintenance records, security footage, sales data and market insights. This data remains largely untapped despite its potential to provide valuable insights and efficiency savings.
“Dark data exists because organisations embrace the notion that all information must be stored so they accumulate massive data-lakes, but most housing providers don’t have the tools they need to use even a fraction of this data.”
How does ‘normal’ data become ‘dark’ data?
Niall Quinn, operations director at Howell Technology Group, said, “Normal data becomes dark data when it isn’t integrated into a housing provider’s data-analytics framework or used for decision-making. This can happen because of a lack of awareness about its potential value, weak data-management practices or technical limitations that prevent its analysis.”
Askporter’s Yexley said, “Normal data becomes dark data when it’s collected and stored but not used. This usually happens when there’s no strategy for data collection, data silos exist or there are legacy systems.
“Data is inherently messy, and most data collected is unstructured and found in a range of disparate sources and formats, making it difficult to use for quick analysis. In short, without a sound strategy or analytical capabilities, normal data automatically turns into dark data.”
How much dark data does a typical housing provider have?
HTG’s Quinn said, “Quantifying the volume of dark data is difficult because it varies greatly depending on the size and data practices of each organisation. For example, I’ve seen entire departments, predominantly front-line workers, using any means necessary to communicate and share valuable data; there is a treasure trove of data being captured but it is usually ‘dark’.
“Instead of using cumbersome CRM tools and processes which, for frontline workers, don’t work well on mobiles or tablets, they turn to other methods to enhance productivity.”
Yexley said, “We estimate that 40-90 per cent of enterprise data remains dark data, meaning a very large amount is simply unused. If this data could be used, it would drive significant improvements at scale, such as autonomously updated asset registers, improved tenant experiences and better regulatory compliance. Housing providers also have a lot of old redundant data; processes are therefore needed to extract meaningful data, keep it up-to-date and create actionable outputs.
“AI technologies for housing, such as Askporter, are changing this. AI can help to engage residents, gather the right diagnostic information and then, in their back-office systems, pull out the meaningful data on residents and property, record it and allocate it to the right staff to take the necessary actions.”
How can housing providers prevent dark data?
Quinn said, “Housing providers should implement robust data governance frameworks that include regular data audits, clear data management policies and the integration of data analytics tools.
“IT departments need to understand the ‘personalities’ of their end-users; it sounds easy on paper but it’s usually hard to implement. We often suggest ‘day in the life’ sessions where someone from IT sits with an end-user within a particular department for a couple of days and studies how they work, helping out where possible and getting hands dirty too, obviously.
Yexley said, “Making use of dark data requires adopting a technology solution that can process and store this data, structuring it in a way that allows it to be reviewed and actually used. Our AI platform provides centralised analytics that can be accessed by housing providers and fed into their existing systems. With this, they can track business-critical workflows in a dashboard that hosts intelligent real-time data. Askporter integrates with housing providers’ existing systems so insights are unlocked across each organisation, solving the problem of under-used data.
“AI-driven tools efficiently process unused data. By integrating these tools, housing providers can automatically sort and classify unused data. AI and large language modelling (LLM) can do the heavy lifting by classifying and categorising dark data.
“Leveraging LLMs can provide housing managers with valuable insights into trends by analysing vast amounts of unstructured data, leading to enhanced decision-making and improved resident satisfaction.”
What are the risks of dark data?
Quinn said, “Aside from the storage, staff and licensing costs of retaining dark data (even if it’s not being used), it represents a missed opportunity to gain a range of insights and efficiencies that could improve housing providers’ operations and decision-making processes. Dark data can also lead to compliance problems if it includes sensitive information that isn’t being properly managed.”
Yexley said, “For best business practice, it isn’t a good idea to have unknown data points in your data repositories. For example, improperly-storing resident information, without proper analysis, can mean missed opportunities for improved service and unresolved problems. At its core, transforming dark data into insightful data involves streamlining knowledge and communication to enhance the efficiency and well-being of residents, staff and properties.”
Housing Technology would like to thank Ben Yexley (Askporter) and Niall Quinn (Howell Technology Group) for their comments and editorial contributions to this article.
