Housing Technology interviewed data integration experts from Aareon UK, Civica, FireAngel Safety Technology, IntoZetta and Northgate Public Services about how and why housing providers should be focusing on data integration and straight-through processing to improve their internal operations and tenant services.
What is ultimate aim of data integration?
Aareon UK’s ERP solution manager, Paul O’Reilly, said, “The ultimate aim of data integration is to improve both staff efficiency and to give a great customer journey. If we start with the experience of a customer who has requested a service from their housing provider, then this should be the thing that informs how we conceive the role of integration. Relevant data may lie in several different places, yet it all needs to combine to create the ‘story’ that provides the customer with a satisfactory resolution to their issue.”
Trevor Hampton, director of housing solutions, Northgate Public Services, said, “There is just no room in today’s digital world for rekeying information from one system to another. Data integration gives housing providers the ability to see the whole picture, ultimately enabling them to offer fully-digital service delivery. It’s only when all the data is connected that they can satisfy customers, contractors and stakeholders’ transactional expectations.”
Andrew Eayres, solution architect, Civica, said, “The aim is to bring together data held in disparate systems. Doing this provides a single view your data so the resulting information is accurate and ultimately allows better decision making. For example, to have a holistic view of a given asset, data is used from your asset management system for planned replacement, or for tenancy turnover, you would access data from your HMS and then add external deprivation data to create a single view.”
IntoZetta’s co-founder and director, Graeme Cox, said, “At the technical level, data integration transforms a housing provider’s typically disparate and fragmented data landscape, sitting across multiple applications, into a coherent, consistent and accurate dataset. This integrated data capability underpins business process and service integration, increasing the levels of automation and reducing the levels of manual intervention through the hand-cranking of stop/start, broken and siloed business processes.”
Why is data integration hard to achieve?
FireAngel Safety Technology’s chief product officer, Nick Rutter, said, “We’re at a stage where the digital integration of data is relatively simple; if you want to streamline datasets and reduce the number of application platforms then integration can be as simple as enabling an API transfer.
“The challenge comes when you want to integrate physical data files or data that’s stored in multiple channels into this system; very often the volume of data is unmanageable and the resources available to oversee this type of integration aren’t available. Add in concerns around confidentiality, GDPR and inaccurate or missing data, and the whole exercise becomes even more complex.”
Civica’s Eayres said, “Data is usually held for specific purposes, in different structures and formats within different systems, from relational databases to Word documents, often in silos, all with their own data models. A housing provider will also probably supplement their core business systems with a variety of informal systems (such as spreadsheets) that become mission-critical data stores. If those challenges aren’t enough, many legacy systems were not designed with data integration in mind; the data may not share a common key and the system may not have a suitable API or other mechanism to facilitate integration.
“Most housing providers’ core applications comprise the ‘big three’ of housing, asset and finance management systems, each with their own databases and data structures, leading to questions about accuracy, which is the lead system and legal constraints about the processing of a particular dataset.
“The challenge is designing systems that are fit-for-purpose while having the ability to integrate the data with other systems without actually duplicating the data – i.e. to integrate data there shouldn’t be a need to hold the same data in more than one system.”
IntoZetta’s Cox said, “There are several unavoidable prerequisites that must be addressed before data integration can be successfully delivered. Unless there has already been a large-scale transformation programme that has consolidated multiple applications into a single new application, it is likely that data is strewn across numerous applications and even some undocumented ‘grey IT’ built up over the years.
“Typically, data is stored across many applications and in different structures, so there isn’t a single view nor a single trusted source. What data sits where and how it moves around the organisation is largely unknown. The overall quality of the data is uncertain and not accurately measured, and there are often different misaligned data identifiers or ‘unique keys’, so linking data across applications isn’t possible. All of these issues will prevent a functioning data-integration capability.
“Over and above the data issues, there is the integration platform and middleware technology itself to design, configure and implement. It must be compatible with multiple application types and integrated with various technologies and database types. In addition, it must be capable of transforming data values and data structures ‘in-flight’ while data and messages are being exchanged, and of course it has to be secure and controlled as well!”
What do housing providers need to do in order to integrate their data across applications?
Aareon’s O’Reilly said, “In the old days, joining up systems and therefore data was very much a case of interfaces. Manual, unreliable and prone to problems, this was, and in some cases still is, a drain on the resources of many IT departments. Things are now changing and the majority of HMS suppliers now offer APIs to some degree or another. These enable easier data integration but they still demand a good deal of technical knowledge so some developer assistance is usually needed.
“Another area often overlooked are the dreaded spreadsheets which enable housing staff to create their own ‘data islands’ that are almost impossible to include in any coherent data strategy.
“A new approach that we are working on at Aareon is the concept of a ‘smart platform’ – a cloud-based platform with a single set of integrations back to the core HMS at one end, and an environment which allows housing providers to ‘plug and play’ digital products, IoT devices, third-party systems and even internally-developed applications into the platform. This approach gives a more usable and comprehensive solution to the problem of integration compared with the traditional, on-premise enterprise service bus (middleware) approach.”
IntoZetta’s Cox said, “Before trying to integrate data across applications and services, a detailed understanding of your data landscape essential. The data must be fully mapped out and be of high quality, so that it can support the integration process through data linking and alignment across the applications.
“Data discovery is needed to identify what data is stored where, the data quality and data lineage, and how it flows to and from both the internal applications and to third parties. This activity, sometimes known as data archaeology, should include all data sources, systems and applications, local databases (incl. grey IT), spreadsheets and even documents.
“A comprehensive data-quality assessment is vital, ideally using an industry-standard set of business rules, to identify where the data-quality issues are across the various business processes and the corresponding impact of each error. This approach will then allow any necessary data cleansing to be carried out correctly.
“Once the data is fully understood, mapped and clean, it’s ready for integration. However, for ongoing and successful integration, the data must be placed under formal data governance and data quality controls to monitor for any degradation that could lead to integration failures.”
Northgate Public Services’ Hampton said, “First, you need a data integration strategy if you want to integrate your data across applications and services. For example, do you use an application-to-application data integration approach, middleware technology or a common data storage approach?
“Second, you need to have access to the right IT skills, either via external data-integration specialists or by training your internal IT staff. In the case of the latter, we’d suggest training a small team rather than an individual to avoid a skills gap if or when that individual leaves.
“Finally, put data-sharing agreements in place before the data integration begins. These include agreements with external stakeholders, such as tenants and contractors, as well as departmental stakeholders because you need an agreed policy about what data is to be shared. These are relatively easy to put together and, in a nutshell cover, what data, how often and in what format.”
How would you define straight-through processing (STP)?
Aareon’s O’Reilly said, “STP is a friction-free process that needs little or no human intervention, providing an immediate resolution for the customer and optimised processing for the housing provider.
“Take the example of a tenant booking a repair. The tenant logs on to the self-service portal or app, reports the damage or fault using diagnostic tools, selects their preferred time and date for the tradesman to call, and has this selection confirmed by an automated text or email. The repair order, because it is low value (in this example), doesn’t need to be authorised, and so is created and issued within the core HMS and sent via scheduling software to an operative’s mobile device. The operative attends the next day, fixes the problem, marks the job as complete, which completes the order in the HMS, sending an automated satisfaction survey by the tenant’s chosen method. Meanwhile, the invoicing data is created in finance system and readied for payment. Throughout that entire process, nobody has had to manually intervene in an IT system, and the tenant gets a speedy resolution to their problem.”
FireAngel’s Rutter said, “STP removes the need for manual intervention, reducing the resource needed for day-to-day functions such as payment transactions. For a housing provider, the introduction of STP allows them to benefit from automating their wider processes, such as letting tenants request maintenance visits, report ASB or access a chatbot powered by FAQs and a knowledge base.”
Civica’s Eayres said, “STP is all about customer requests that can be processed using automation, such as via an electronic form with no manual intervention. STP originated in the banking sector to enable payments and other transactions go through without any manual processing, but it can be applied to almost any service request.”
Do housing providers understand the concept of STP?
Northgate Public Services’ Hampton said, “As a specific terminology, STP is not widely known in the housing sector because historically STP has mainly applied to financial services, but housing providers are more familiar with the concept of data integration. After all, the majority of housing providers have a degree of data integration, perhaps linking their housing or asset management systems with their finance system or their rent accounting system with their tenancy payment engine, while some of the larger housing providers have more sophisticated data integration systems, such as middleware and master data management.
“While our sector is keen to enjoy the seamless experience that STP provides, housing providers know that it can’t be achieved 100 per cent of the time. There will always be instances when it’s impossible to have a fully automated transaction; a good example being a tenant with a pest infestation and a phone call would be needed to get a better understanding of the problem.”
Aareon’s O’Reilly said, “I think the concept of STP is beginning to gain traction now that housing providers can see what’s possible – just look at the truly online business models of companies such as Amazon or Ryanair who have been using this model for years.
“The pandemic and past year of disruption have forced our sector to make changes to the way it thinks about technology. The housing sector has historically lagged behind others but we’re now seeing significant acceleration in housing providers’ plans for digitisation.”
FireAngel’s Rutter said, “I think housing providers’ understanding of STP is growing and, more importantly, their acknowledgement of the benefits of STP is increasing. Self-service is now commonplace in our daily lives, with online banking, chatbot customer service and online shopping driving this acceptance of STP.”
Are housing providers ready for STP?
Northgate Public Services’ Hampton said, “Consolidation lies at the heart of whether or not you’re ready for STP and data integration. We’re still seeing housing providers who lack not only a data integration strategy but also lack a fundamental IT strategy. Having too many disparate IT systems makes joined-up data-sharing unwieldly and expensive, and continuing to rely on paper-based processes makes data integration a complete non-starter.
“Another crucial factor in determining STP readiness is whether the housing provider has mapped out its customer journey from start to finish and thought about what this now looks like in a digital world. If this hasn’t been done then it will be difficult, if not impossible, to identify the consequent data integration needs. The customer journey has changed so much in the last decade that it’s vital to reimagine it in order to understand what the new digital connections look like and where the data integration points need to be.
“If you don’t start to invest in data integration and IT strategies now, then imagine how far you will be behind in five years’ time when looking through the lens of a fully connected, transparent and integrated IoT world.”
What are the advantages and disadvantages of STP?
Civica’s Eayres said, “The main advantages of STP are better customer service by quickly and efficiently handling of tenants’ service requests, combined with lower costs from the removal of unnecessary manual interventions and more ‘first-time fixes’. For example, if a tenant receives prompt automated communications about the progress of their request, it removes the need for them to contact their housing provider. STP also formalises many decision-making processes, thereby removing arbitrary choices at the same time as providing an automatic audit of the outcome.
“The main disadvantages of STP are the difficulties and cost of achieving STP across disparate systems that were perhaps never geared up for facilitating STP in the first place. These are coupled with concerns about accuracy and the suitability of handling particular requests via STP. For example, a housing provider might want to impose manual checks when updating asset information; humans are very good at handling variations and dealing with uncertainty whereas an automated process might not be.
“The key is to identify which service requests or transactional processes most lend themselves to STP, but it doesn’t have to be an all or nothing situation; some processes might need only some degree of STP, still leaving the freed-up resources to focus on exceptions management or adding human involvement where and when it’s needed most.”
FireAngel’s Rutter said, “Housing providers have the almost-impossible task of providing residents with safer homes and supporting a higher quality of living from already-stretched resources; STP can shoulder some of that burden and reduce the volume of administrative tasks. STP can also support the concept of a single source of truth and a ‘golden thread’ of data because all interactions can be digitally logged against the file for each property or tenant. STP also reduces errors and concerns around GDPR and other regulatory requirement because processes should be able to be completed in full without human intervention or error.”
IntoZetta’s Cox said, “STP helps to drive improved service, greater operational efficiency, streamlined business processes, all resulting in higher levels of customer satisfaction. Because the need for manual interventions is reduced, operating costs fall, productivity improves and existing staff resources redirected towards more value-adding activities. However, things can go wrong if there are gaps in the STP design and missing scenarios that will need manual interventions and work-arounds until they have been fixed.”
What factors prevent housing providers adopting STP?
IntoZetta’s Cox said, “Aside from any cultural barriers arising from increased automation, the typical factors preventing housing providers from adopting STP are other competing priorities, cost, complexity and the lack of a clear business case defining its benefits.”
Northgate Public Services’ Hampton said, “The biggest barrier is if the housing provider hasn’t built the right foundations, and for that you need a housing management system that’s fit for purpose. I can’t bang the drum loudly enough here because if your housing systems aren’t functioning as they should then linking up the data will be irrelevant because the quality of the data won’t be good enough.
“Having a flexible, open system is so important, and your systems should ideally provide free APIs for the ultimate flexibility. While we might have been the first to do this, it is nevertheless encouraging to see a number of other HMS suppliers now changing their approach. That said, all housing providers should be mandating this because it’s about ensuring they have the power to control and use their data freely, easily and with no restrictions.
“You also need the vision, understanding and buy-in from top to bottom of your organisation because achieving STP isn’t a short-term process. It’s possible to achieve STP levels of around 15 per cent of transactions within the first year, but to get above 80 per cent of transactions and processes being completed without human intervention will take three to five years.”
Aareon’s O’Reilly said, “Technology can be a barrier, especially if you are trying to build an STP-based solution yourself using tools that don’t lend themselves to the idea. Data integration and reliability are both critical factors, alongside the ‘corporate mindset’; unless you have arrived at a housing provider from another business sector and have already experiences STP in action, it can seem counter-intuitive to relinquish hands-on control of your business processes, but if executed well STP is a significant driver of both efficiency and great customer journeys.”
Housing Technology would like to thank Paul O’Reilly (Aareon UK), Andrew Eayres (Civica), Nick Rutter (FireAngel Safety Technology), Graeme Cox (IntoZetta) and Trevor Hampton (Northgate Public Services) for their comments and editorial contributions to this article.
