A measure you can’t trust

Are tenant satisfaction scores really all they are cracked up to be? With increasing importance attributed to a landlord’s ‘overall satisfaction rating’, are such figures really reliable as a service quality measure which trumps all others?

Could we even argue that tenant satisfaction statistics render housing inspection redundant?

From the Tenant Service Authority’s viewpoint it would be convenient if the answer to all these questions was ‘yes’. Although - like housing inspection - measuring satisfaction costs money.

At present, housing associations, not the regulator, foot the bill. If associations’ satisfaction ratings are considered authoritative it helps justify the slimmed-down inspection framework and the lighter touch form of future regulation envisaged by the TSA.

However, brand new research on tenant satisfaction measurement suggests that the practice is rather less consistent than is often imagined.

The study, published today and commissioned by London & Quadrant Housing Group from Heriot-Watt University and Ipsos MORI, covers 46 English housing associations which have recently completed tenant surveys. It also analyses the TSA’s 2008/09 tenant satisfaction statistics to identify uncontrollable factors which can influence particularly high or low scores. So is satisfaction up to scratch?

1. Measuring like for like

The research confirms that satisfaction measurement is taken seriously by associations and, generally, is managed professionally. But it also reveals that local practice is not always fully compliant with official guidance, known as STATUS and produced by the National Housing Federation. In a few cases, landlords limited their adherence to the prescribed framework to just a few STATUS questions included in ‘continuous’ or annual surveys.

This is a problem because where, for example, questions are asked as part of a ‘continuous tracking survey’ the sample of interviewees is drawn from individuals who have recently received a service, rather than from a full cross-section of all tenants. Unduly flattering satisfaction scores may result.

2. Method of communication

About a third of the associations polled had conducted their latest survey over the phone, rather than sending self-completion questionnaires through the post. By contrast with the rules for local authorities (which mandate the postal model) this is allowed by the STATUS guidance.

Arguably, however, some tenants may be reticent in giving full vent to their critical opinions when speaking to an interviewer in person. Here, then, it is the guidance rather than landlord practice which is questionable.

3. Tenant representation

Half the associations in the study failed to check whether the people who responded to their polls were representative of their tenants as a whole. These landlords cannot, therefore, know whether their results should be ‘weighted’. Besides contravening the guidance, this matters because different groups within the population have different perceptions.

In particular, older people tend to view landlord effectiveness more positively than others. Unjustly, therefore, a landlord’s failure to measure and correct for the typical over-representation of, for example, older people among survey respondents means that it is likely to post a misleadingly flattering satisfaction score.

4. Arithmetic skills

It turns out that some associations ignore the officially specified formula for calculating the ‘satisfaction performance indicators’, which they submit annually to the TSA. The formula says landlords must count ‘no opinion’ or ‘don’t know’ responses. But some don’t. For a minority of the landlords concerned, this will have produced erroneously high satisfaction scores.

Among the landlords found to have departed from the recommended approach here are a few with ‘overall satisfaction with landlord’ scores in the top 25 per cent of the national ‘league table’. Full compliance with national guidance could have led to these associations recording significantly lower scores, leading to subsequent relegation in the national rankings.

5. Working with what you’ve got

Even if substandard research practice were eliminated, it would remain important to recognise that variations in tenant satisfaction scores are affected by a range of factors outside a landlord’s control. Yes, there is some ‘positive correlation’ between an association’s satisfaction rating and its measured housing management performance. Nevertheless, factors other than ‘management performance’ also influence associations’ satisfaction ratings. Irrespective of their management effectiveness, a landlord’s recorded satisfaction score is associated with the profile of its housing stock and the people it houses.

As shown by this analysis, associations which mainly manage flats tend to report lower tenant satisfaction. There is a similar link where landlords operate in deprived areas. The strongest influencing factor is ethnicity. Landlords with a large black and minority tenant population tend to record significantly lower satisfaction rates. For example, London associations, which tend to work in areas with large BME populations, generally record satisfaction ratings markedly below the national average.

Finally, the recession has depressed national scores for customer satisfaction with public services overall. This is a sharp reminder that such ratings reflect respondents’ perceptions of the broader economic outlook. This challenges the belief that simple monitoring of customer satisfaction trends over time is the ultimate measure of an organisation’s changing performance. Perhaps it’s time for a measure you can trust.

Have your say

You must sign in to make a comment