The big data experiment

When this article is posted on the Inside Housingwebsite, it will take its place among the estimated 26 quintillion bytes of data produced worldwide every day. Tomorrow the amount of new information heaped onto the pile will be even greater.

The pace of change is staggering. According to technology multinational IBM, 90% of the world’s data has been created in the past two years. We all leave trails of the stuff everywhere we go; each time we click on a website or make a phone call, for example, we produce data that, when aggregated and analysed in the right way, can reveal a great deal about our behaviour.

Businesses in the private sector have been leveraging this information for years.

The recommendations Amazon makes to you on its site are not just based on your purchase history, but on patterns revealed by the clicks and page views of huge numbers of your fellow consumers. You could almost say Amazon and its retail bedfellows know what you want before you do.

That’s the promise of what is known as big data – and the social housing sector is about to take its first byte.

For the past 18 months, the housing charity HACT and a group of 16 housing providers with around 400,000 homes between them have been collaborating on a project that aims to aggregate and interrogate the data produced and kept by its members.

“With big data you’re always on the edge of what is possible.”

That metric – the number of properties – is crucial, explains HACT’s chief executive Matt Leach. When it comes to big data, bigger is better. ‘If one housing provider with 20,000 homes isn’t going to get much insight from its relatively limited data, then if we can get 400,000 homes’-worth… perhaps at that point the volume of data will give us insights we can more generally apply,’ he says.

Bringing that data together, storing it securely and converting it into a useable form has been a complex challenge, Mr Leach adds – but he believes this project has the potential to transform the way social landlords operate.

Rent arrears

The project’s first foray into big data analysis has focused on rent arrears. Each of the housing providers involved has collected the information they hold on residents’ rent accounts, stripping away personal information so that individual tenants can’t be identified, before storing it in online ‘vaults’ according to strict security protocols (see box: Security).

The idea is to combine and cross-reference this information with other sets of data – which might include information about the locality, the types of property, information about the tenants and how often they access services – in order to try and find underlying factors that, when spotted early, could help predict whether residents are likely to get into arrears.

‘We have found that by taking the last two years of rent payment histories for people living in social housing, and attaching that to a bunch of other factors, you can get quite close to predicting which people, in broad terms, will be in arrears in a month, three months, six months,’ says Mr Leach. ‘That’s quite exciting, because it points to towards being able to create models that enable you to look at your rent book and find those tenants who need the most support almost before they need it.’

HACT’s team has repairs in its sights, too.

‘Take a new-build scheme – what’s the profile for repairing different property archetypes as it ages?’ asks John Schofield, group director of research and development at Family Mosaic, one of the housing providers signed-up to HACT’s project. ‘How good are we at predicting when boilers need replacing? How can we use data to more accurately predict when things go wrong?’

“The sector as a whole is behind in using data, generally.”

If a resident is unemployed, for example, and spends more time at home, might that mean more wear and tear on the home’s fixtures and fittings? Or could that be connected to the reason they make so many calls to the repairs service?

Once some answers have been extracted from the data, social landlords should be able to target their resources with greater precision and foresight. There are some ethical questions around predicting behaviours (see box: Thought police?), but big data should mean healthier finances – and happier tenants, too.

‘It’s exciting,’ says Amy Redman, business information and research manager at Riverside Group, one of the big data project members. ‘Understanding whether someone is likely to have financialdifficulties, for example, means we can target them with our money advice services rather than waiting and watching people get into arrears.’

Community investment

In a few months’ time, and using data held by a group of 10 landlords who are in the process of signing the necessary contracts, the project also plans to turn its attentions towards housing providers’ community investments. The question to be asked of the datasets is what difference these sorts of projects make to providers’ bottom lines.

Tom Smith, managing director of Oxford Consultants for Social Inclusion, a Brighton-based organisation that takes a data-based approach to improving the performance of public sector organisations, likens the housing sector’s spending here to the old adage about marketing: half the money spent on advertising is wasted, only no one knows which half.

‘Housing associations are in the same boat,’ he says. ‘What I see these data streams helping with is understanding both future need and effectiveness. How much could have been better spent and which of our pounds have had the most impact?’

Crunching the data created in the wake of these kinds of spends could help answer those questions – but getting to the stage where those answers are within reach has not been easy.

Challenges

Collecting and analysing this kind of big data is a complicated business. To give you an idea of the level of expertise necessary, HACT’s latest hire for this project is data scientist Philip Goddard, a post-PhD nuclear physicist whose previous job involved modelling particle collisions inside nuclear reactors.

‘With big data you’re always right on the edge of what is possible in terms of available technology,’ says Mr Smith.

And it’s not just the number-crunching that presents technical challenges. The quality of the raw data – or rather its lack of quality – has proved to be another.

‘The sector as a whole is behind in using data generally,’ says Mr Schofield. ‘We have always used our information systems for operational purposes, and the upshot is the data we have produced has not been of a very high quality for data analysis purposes.’

“There must be a more coherent approach to using data.”

Two housing providers may collect and present information on which tenants have fallen into arrears, for example, in quite different formats and using different systems. Moreover, one provider’s operational procedures might mark a tenant down as being in arrears one day after the due payment date; the other might tick that box 10 days after payment is due.

Compiling and standardising these various sets of data has been a ‘massive job’, according to Mr Leach.

‘Data in housing is incredibly non-standard,’ he says. ‘If you want to make great strides forward, then there needs to be a much more coherent approach to labelling, storing and using data.’

The housing associations involved in the project understand that, Mr Leach says, and are taking steps towards each other in this respect. But this shared approach can only go so far; any top-down mandate to landlords to collect data of a certain kind, or in a certain way, smacks of bureaucratic drudgery, he says, and would suck the life out of the project. And as Mr Schofield points out, front-line staff have enough on their plate without being asked to collect data for projects that might not show fruit for several years.  

Fortunately big data contains its own solution. The greater the amount of data you have to work with, the less the quality of that data matters. ‘If you’ve got enough data, you can get meaningful results through sheer volumes,’ says Mr Schofield.

It looks as though that volume is set to increase exponentially in the near future, thanks to what Mr Leach is calling ‘the next wave of data in housing’ – information collected from new technologies, such as smart meters or heat and moisture sensors, installed in homes.

Within a few years data will become so vital to the sector, he believes, that housing providers could be taking a lead from private sector organisations and appointing chief technical officers and chief information officers to their executive teams and boards. ‘Data will become fundamental to getting value out of businesses in a way it hasn’t over the past 20 years,’ he adds.

Perhaps if someone had crunched the data on housing personnel and organisational structures over the past few years, we might already have seen that coming.

For more on big data, attend the CIH Manchester masterclass ‘The big data question - getting the best out of your data’ on Tuesday 23 June at 2.15pm in Charter 4. Click here to book your place for the conference.