‘Big data’ is everywhere these days, even making it onto the agenda of the recent G8 political summit. Housing Technology interviewed a selection of people with an interest in big data to share their view on how big data can be applied to housing providers’ operations.
First of all what is big data? Most people have some understanding of the concept in their own minds but might struggle to explain it if they were asked to define big data. Wiley’s recently-published ‘Big Data for Dummies’ (reviewed on the opposite page) defines big data as any data source that has at least three shared characteristics: extremely large volumes of data, extremely high velocities of data and extremely wide varieties of data.
Chris Coan, managing director of Visualmetrics, said, “Big data is a big bucket – start with video, audio, text, machine, structured, and unstructured data. Combine those in massive volumes with tremendous variety, and you get complexity. The challenge then is how to corral and mine the information to create new opportunities. Therefore the only value of big data is the ability to collect, distribute and analyse the information found within, in a timely manner and with automated processes.”
Continuing the theme of the value of the data, David Mitton, director of Liberata, added, “I like the introductory paragraph of the Wikipedia definition: ‘big data is a collection of data sets so large and complex that it becomes difficult to process’. However, I would extend that to cover all data, big or otherwise, that is difficult to process because if you are unable to make sense of it then it has little value.”
Fad or here to stay?
A couple of years ago, big data could have been dismissed as a passing technology fad or at least only applicable to the largest and most technically advanced global companies. However, like cloud computing, big data has shaken off criticisms of marketing hype and become almost a mainstream technology.
Glen Lewis, UK housing sector manager at Northgate Public Services, said, “The term ‘big data’ may come and go, but the availability of vast amounts of information to organisations and individuals is here to stay. The market for analysing big data is growing rapidly and will continue to innovate and develop over the coming years. We expect to see companies with specialist knowledge of particular datasets emerge as organisations try to make their key business decisions more data-driven.”
David Hall, a senior associate at Anthony Collins Solicitors, added, “Only time will tell if big data has staying power, but ultimately, the answer to the ‘fad’ question will depend on how much your organisation already uses business intelligence, and whether your vision and objectives demand more of what big data has to offer. It would be great to see the sector challenging its suppliers to improve their services by making judicious use of big data.”
Why big data for business?
When asked about the business drivers behind the adoption of big data, Paul Rowley, head of information services at Havebury Housing Partnership put it very succinctly, “It’s simple. There is a single business driver – better decision making.
Dr Andy Murray, chief technical officer of Mutiny Technology, said, “If you can gather intelligence from all of your applications and channels by collecting and analysing the big data that they produce, you can track your users or customers’ behaviour and their real-time data access requirements. Thus, from understanding how these systems are interacting, an organisation can design better and more cost-effective business services.”
Mitton from Liberata added, “Current trends and technology, in the form of social media monitoring, tap into the growing use of social networks and around all sorts of formal and informal communications. This element of big data provides a window into local and national mass opinions, issues, subjects and sentiments. Used correctly, this data can be interpreted and used as a means to deliver positive changes and responses to very current issues.”
This view was shared by Northgate’s Lewis who said, “A key driver is the desire to improve outcomes by providing better, more tailored and customer-appropriate services, while reducing operating costs. A better understanding of the patterns and trends within communities and populations can help to identify groups at risk and their associated risks and also monitor the effectiveness of intervention.”
Technology factors
Big data is not a standalone technological development; it’s the combination of the last 50 years of technology evolution and data management techniques. Furthermore, big data is already part of a wider technology trend, known as SMAC as it comprises sourcing, mobile, analytics and cloud.
Stewart Townsend, business development director at Zendesk, said, “Technology is always changing and in most cases, big data is the remodelling of a data warehouse, and depending on the requirements, it means either a Hadoop infrastructure for rapid large-volume, multi-threaded tasks or overnight batch-process runs that drive an old style business intelligence tool like Cognos.”
Rowley from Havebury Housing Partnership explained, “The technology drivers are easily appreciated with increases in data availability, better interfacing with external data sets through web services and XML, and social media and geographical information creating huge varieties of data. To take advantage of all this, we now have the cheap storage, ‘No-SQL’ database products, scalability and maturity that is required to analyse huge amounts of disparate data.”
Lewis added, “The technology drivers come from an explosion in data sources and the volumes of data being created, in tandem with the advent of web-scale data analytics, allowing companies to process high volumes of real-time, unstructured data.”
Big data in housing
To date, big data has tended to be associated with financial services, retailing and consumer products, pharmaceuticals and utilities; central and local government and housing providers have been slower to adopt big data but as they become more focused on using business intelligence for better decision making, that situation may change. Mutiny Technology’s Murray explained, “There are a number of new applications that can now aggregate and index big data in real time. This enables faster analysis, intelligent search tools and easy-to-understand reporting of the results.”
Hall from Anthony Collins Solicitors, said, “Housing providers who are already looking for sophisticated business intelligence will benefit significantly from big data as it enables more flexible and sophisticated analytics. Big data technologies enable you to ‘mash’ many sources of data, far beyond those produced by existing systems such as the CRM, finance, HR, housing management, maintenance and repair, and care service management applications.”
Is big data right for you?
In its present guise, big data is a concept that most mid-size and large housing providers should be keeping an eye on, with a view to adoption as the technology matures and becomes more commoditised.
Taking a more optimistic position, Coan from Visualmetrics explained, “These technologies will allow housing providers to effectively manage the process of analysing large volumes of data instantly, resulting in satisfying the business needs in terms of real-time reporting. Big data technologies help reduce data silos and allow less complex integration and reporting processes, which ultimately increase the efficiency of the organisation.”
Havebury Housing’s Rowley said, “We are not at the right point as a market sector to need big data, but if you want to look like the big boys and implement it, fine. However, what you must not do under any circumstances is to allow your infrastructure to become incapable of supporting big data when it becomes more widespread.”
Lewis from Northgate added, “Without greater use of big data and the solutions to use it, it is unlikely that the social housing sector will continue to be able to deliver extended services within its communities and increase its ability to tackle the massive challenge that welfare reform brings.”
Business intelligence vs. big data
Most housing providers have some form of business intelligence applications and processes, ranging from basic spreadsheets to full-blown, dedicated business intelligence systems. So, how does big data fit into the scheme of things?
Rowley from Havebury Housing said, “BI will tell you all the places you have void properties. Big data will tell you that those void properties are where people have tweeted about the lack of amenities and that the properties are all on streets with high gradients to get to the local school.”
Zendesk’s Townsend added, “Business intelligence and big data are interlinked. Big data relates to the collection and usage of data, while business intelligence, analytics and reporting enable housing providers to answer questions about the data itself.”
Coan from Visualmetrics explained, “Big data is not replacing business intelligence; it’s enhancing the need for it. With the right big data technologies and the right business intelligence tools, housing providers will be able to analyse data and discover trends that were impossible before.”
Volume, velocity and variety
Relating to the ‘Big Data for Dummies’ book’s definition of big data concerning extreme volumes, velocities and varieties of data, Havebury Housing’s Rowley pointed out, “Don’t forget the fourth ‘V’; veracity. Do your staff trust what’s in your big data system and what comes out of it? Does it stand up to challenge and audit? That is much more important.”
Impact on existing technologies
Depending on your perspective, big data sits either above housing providers’ existing applications or to one side of them; it is the means to the end, not the end in itself.
Rowley continued, “Big data itself has little impact on existing systems. It is the data held within either the housing provider’s IT systems or its data warehouses that will be used in the big data system. However if you get big data, you should expect to collect more data faster and with greater variety. The real-time ideals of big data will rub very uncomfortably against the batch-style reporting technologies used in most housing IT systems.”
Mutiny Technology’s Murray concluded, “Gathering logs, aggregating the data, indexing, storage, searching and display all require systems integration and some dedicated hardware. Add to that investment in a full-time analyst and you can probably make something of big data in your organisation.”
Conclusion
Housing Technology posits that big data is still a fairly ‘extreme’ area of technology for most housing providers, but it will become more mainstream and commoditised over the next 18-24 months and its relevance and affordability will trickle down from its current ‘big ticket’ status for sectors such as banking and retailing. Housing providers certainly shouldn’t rule out big data and keep an eye on developments in this area. In the meantime, we would encourage readers to read ‘Big Data for Dummies’ for an independent view of what big data is all about.
Housing Technology would to thank David Hall (Anthony Collins Solicitors), Paul Rowley (Havebury Housing Partnership), David Mitton (Liberata), Dr Andy Murray (Mutiny Technology), Glen Lewis (Northgate Public Services), Chris Coan (Visualmetrics) and Stewart Townsend (Zendesk) for contributing to this article.
