Reviewing Welfare Reform

Over the last 18 months or so, a steady stream of academic research findings have challenged key arguments put forward by the DWP about the impact of welfare reform. These studies have highlighted evidence that does not support the behavioural assumptions on which impact assessments carried out by the Department are based.

The context in which the public debate is taking place is widespread media and mainstream political support for the necessity of welfare reform. This is coupled with anxiety about the consequences of specific measures such as the bedroom tax or about the impact of benefits changes on particular client groups such as people with disabilities. In recent months UK politicians and pundits have also expressed growing alarm at implementation problems and roll-out delays. But until recently the academic findings have not been drawn together and as a result not featured prominently in the debate.

 Earlier this month, the Scottish Government published an evidence review we carried out on the housing impacts of the post 2010 welfare reforms (1; 2). It was written before two of the more recent contributions to the research base were published, one by Steve Wilcox and also the recent widely reported CCHPR interim evaluation of the bedroom tax for DWP (3). The Scottish Government report is a rare attempt to synthesise a wide range of literature and, alongside the Wilcox and CCHPR reports, it raises fundamental questions.

We readily acknowledge that there are thorny methodological questions. For instance, some of the available evidence comes from research undertaken prior to the introduction of the reforms. This includes not just impact assessments but also studies that estimate effects using secondary data. Some of the research after reform was introduced is incomplete – providing evidence based on initial or interim findings only and some is based on small scale studies. Despite this, there was enough high quality work available, both quantitative and qualitative, to raise serious questions about the scope, direction and implications of UK welfare reform.

Our evidence review was organised around themes agreed with the Scottish Government and findings were split into sections reviewing the research on the private renting reforms (the most long-standing and hence the largest body of evaluative work), social renting reforms to working age tenants, cumulative effects to households, places and markets and the impact on social housing businesses. 

The conclusions can be grouped under six headings:

1. Changes to working age benefits (including non housing related benefits) have increased financial risks for claimants. Large numbers of disabled people in particular have lost entitlement to some or all of their benefits.
2.  Welfare change is interacting with the private rented sector in dynamic ways: will landlords risk letting to benefit-dependent tenants; will they actually stay in that part of the market? Housing Benefit claims by working people have actually doubled since 2009. Rental market benefit cuts have wider effects, not least by forcing those affected to seek lower cost housing further afield. Private renting is the sector with the least residential stability and the cumulative effect on the PRS is likely to exacerbate turnover and weaken both neighbourhood and local service cohesion.
3.  Most social sector studies thus far have focused on the under-occupation charge. The evidence suggests that it is difficult to calculate how much downsizing will occur but that there are strong countervailing forces reducing mobility. Social landlords are most concerned about the financial inclusion issues surrounding whether tenants can cope with responsibility for rent payments following the introduction of Universal Credit that will end direct payments.
4.  The impacts on housing systems are best understood in the wider context of welfare reform reducing incomes and creating new uncertainties for recipients e.g. as a result of changes to ESA and the introduction of PIPs, as well as the sanctions regime. These reforms are important strategic drivers for housing organisations affecting their business plans, development and investment strategies both in the light of lower rental income security and continuing less attractive lending opportunities.
5.  We found evidence of inter tenure and place effects of these reforms on local housing systems. A common concern is that previously settled neighbourhoods and more marginal areas will be threatened by increased turnover and mobility as households seek lower cost housing, thereby undermining local services and community cohesion.
6.  Finally, the review identified evidence gaps. First, there is no robust sense of ‘second round’ effects on behaviour. Second, there remain few if any comprehensive studies of local markets taking account of other drivers of housing outcomes, as well as welfare reform. Third, there is no detailed research evidence on lender attitudes to the new risks they perceive. Fourth, a detailed study of the interaction between sanctions, welfare reforms and housing/labour market processes and outcomes would be valuable

One criticism of the UK government’s welfare reform project, and in particular the housing dimensions of it, has been that it lacked convincing evidence to support it. Another important criticism was that the implicit or even explicit behavioural assumptions concerning responses of affected tenants, private landlords and social housing organisations – were simply not warranted. Our evidence review worked through a considerable volume of research material, and despite the gaps and the other caveats identified (e.g. it is still early in terms of housing responses post reform), there is no doubt that the overwhelming majority of the robust research evidence remains consistent with such criticism.

The outcome evidence on private renting does not support the behaviour change expected in the market and proposed by DWP, Neither have under-occupying social tenants down-sized though it is less clear why. The more recent CCHPR and Wilcox studies on the under-occupation charge also fit with this perspective. Perhaps that is no great surprise but at least now the conclusion is based on evidence rather than conjecture.

Notes

1. Thanks to Des McNulty, a co-author of the evidence review report, who commented on a draft of this post.
2. Gibb, K, Sprigings, N, Wright, S and McNulty, D (2014) Evidence Review of the Impacts of UK Welfare Reform Affecting Housing in Scotland. Social Research Scottish Government. See: http://www.scotland.gov.uk/Topics/Built-Environment/Housing/supply-demand/chma/welfarereformhousingevidence/welfarereformreport
3. Wilcox, S (2014) Housing Benefit Size Criteria: Impacts for Social Sector Tenants and Options for Reform. Joseph Rowntree Foundation: York. Cambridge Centre for Housing and Planning Research (2014) Evaluation of Removal of the Spare Room Subsidy: Interim Report. Department of Work and Pensions: London.

Advertisement

Legacy?

In a few days Glasgow will host its first mega event. The preparations, the visitors and the symbols of the Commonwealth Games are inescapable throughout the city. Even where I live in Lanarkshire, the triathlon is taking place in Strathclyde Park and I’ll be able to walk to the event.

Since the announcement that Glasgow had won 2014 more than six years ago, with academic colleagues we have been debating the Games, the notion of legacy and the pros and cons of committing to such a massive project. It will cost more than £560 million, not counting the disruption and other unaccounted for real costs, and many have argued that this is a high price to pay for what is predominantly a sporting festival. Critics point to the previous evidence of other sports and cultural events, which found little evidence of long-term benefits. Others argue that these mega events represent neo-liberal responses to urban decline and post-industrial regeneration through sport and cultural tourism attached to more or less city boosterism. Contrary to the position stressed by the organising hosts, critics argue that far from being inclusive, they are examples of and reasons why local people lose out when events like the Games are imposed on them.

The response to this from those more favourably disposed or at least willing to test the proposition is multi-faceted. First, the external bodies awarding future games require credible legacy pledges. So, if you are going to bid, there will necessarily be clams being made. Expectations of legacy have also steadily risen with each round of bidding. Are they realistic and have they been modified in the light of the experience of preparing for Glasgow 2014? Second, unlike previous events, Glasgow is different because it has since the awarding of the 2014 Games been planning for legacy, developing a multi-level evaluation framework for the different dimensions of legacy and involving extensive independent research testing both the Government’s and the council’s legacy frameworks. Third, legacy happens after the Games and analysis and measurement will go on for 5-7 years after the event is over. But the evaluation commitment will need to be sustained if these lessons are to be learned.

These are all lessons learnt from previous games in places like Barcelona, Manchester and London. Glasgow is another step on that journey and the 2014 Games is a different stage on the city’s long term regeneration which offer the possibility of impacting positively on that process. But to be clear, these are all propositions to be tested, and, moreover, the multiple nature of the legacy goals (e.g. economic growth, environmental sustainability, inclusiveness, international connectedness, etc.) may well come into conflict with each other and not simply be additive.

As part of my policy knowledge exchange work we have established a new legacy research network which we hope will bring cities and academics together to discuss sharing knowledge and learning about what works and what does not from sporting and cultural events around the world (and also provide a space for robust critical analysis). Initially, we are looking at Glasgow 2014 and we seek to clearly situate legacy in terms of Glasgow and its regional regeneration/development. We are working with the City, its cultural and sport agency Glasgow Life, and representatives of the three Universities in the city. Our first activity was to hold a panel session last week asking what is legacy, what can it mean for Glasgow and what lessons can we draw about its evaluation? We plan to hold an inaugural international conference in Glasgow in the middle of October 2015.

The panel session involved contributions from Bridget McConnell (Director of Glasgow Life), my colleague Ade Kearns (who leads the Go Well programme and the east end community evaluation of 2014 legacy) and Robert Rogerson (Strathclyde University) who for three years has been funded to examine legacy in an innovative partnership involving the same partners who are involved in our legacy network. The main points that struck me from the debate were the following:

1. The evaluation of the legacy will be complex (to put it mildly). Isolating cause and effect and capturing the independent effects of Games-related activities that are genuinely additional, to point to just two issues – will be methodologically challenging in the extreme.
2. The panel were clear that there would be no simple answers confirming or denying lasting legacy effects. As Ade Kearns pointed out, almost any element of the games project investments can be viewed in positive and in negative terms depending on perspective and context. Moreover, the idea that legacy impacts will follow in an emulative way always runs the risk of depending on how the individuals concern choose to respond. Interventions cannot make people be more active, for instance, if they have other countervailing incentives that trump the intervention.
3. A novel feature of the 2014 model is early access to community facilities for local people. This has been successful and is likely to help sustain the facilities investment (as well as make wider local provision available).
4. Ade Kearns also argued that broader regeneration initiatives (in public health and in physical infrastructure via the M74 motorway extension and Clyde Gateway business space investment) would inevitably overwhelm the independent effects of 2014. But that does not mean that legacy’s complementarity to the bigger regeneration programme is not worthwhile.
5. It was also pointed out that the opportunity cost figure is not straightforward because some of the big investments represent the acceleration of investments that were planned but were able to be brought forward and done so together for 2014. For instance, the national sport centre was accelerated while the adjacent velodrome was completely new.
6. The hosting of 2014, assuming it is successful, also places Glasgow in a more competitive position to draw benefits from future bidding for sports and cultural events to the benefit of the city and region. In 2015, for instance, Glasgow will host the World Gymnastics. Will the city develop a capacity to build on this success, how do we assess the returns on such a direction taken strategically, and, who stands to benefit from it?
7. Finally, Glasgow city council has recognised the partnership working benefits of delivering a huge project like 2014. They are keen to not lose these process benefits when normal service is resumed and they are therefore looking at how to mainstream them into everyday council activities.

Like the study of area regeneration, research and evaluation in this field is remarkably difficult and challenging, which makes it so interesting and this is precisely why knowledge exchange and opportunities to work with other cities including to challenge legacy ideas and beliefs – is something that seems really worth doing. You can find out more about the legacy research network at the Policy Scotland website (http://policyscotland.gla.ac.uk).

First things first – there are events to deliver and sport to enjoy. We’ll talk more after the festival is over.

Editing Blues (or, Learning to Love Manuscript Central)

Academics have an ambivalent relationship with peer-reviewed academic journals. In the REF-dominated UK we are all under pressure to place papers in our discipline’s top ranked international journals. That means we have to learn how to play the game and know what works and what does not. We also feed the monster by acting as (hopefully) anonymous referees providing (hopefully) constructive but honest expert reports in an objective disinterested way. Third, some of us may in time go on to act as editors of journals.

In an era beset by technological change, multiple public access controversies, the changing industrial economics or business models of said journals and publishers, as well as the proliferation of new journals – one can forget the other side of the coin of delivering the journal itself.

For my sins I was for over ten years, an editor, managing editor and even latterly editor-in-chief of a big international journal. I stepped down at the end of 2011 but even so I have since found myself involved as co-guest editor of three special issues at other journals and I still sit on four editorial boards. I have also done a stint of editing handbooks and reference works – it is addictive but I am trying to wean myself off it.

I dread to think of the number of papers I read for the journal I edited. Thousands. Typically for each submission we would have two or three external reviews plus the input of two of the editorial team. From this we would seek a consensual decision. I started off very cautious and careful when coming to a decision on a paper. I was always amazed when my senior and very experienced editorial colleague would rapidly conclude whether or not there was a (potentially publishable) paper in what we had been sent. But, I must confess, as time has gone on, one does become more confident, in part because of the on the job experience you build up but also your sharper sense of what the journal is for and what ineffably constitutes a publishable paper – I know it when I see it but I am not sure I could define it.

My main other ‘editorial blues’ lessons to impart from this experience are:

1. Over time an editor clearly reads a lot and can get really on top of a literature to by a form of osmosis. But I think they probably get better in most cases at seeing how effective is a given literature review section of a paper in terms of how it is written rather than somehow mastering that literature in a substantive sense.
2. Never let people off with going over the word limit. Ultimately it damages someone else’s ability to be published given total journal word caps set by the publisher; it is also a good discipline for the author. You grow to hate long papers, yet we had a major battle with authors when we reduced our papers to 8,000 words!
3. Disputation is mercifully rare but when it happens it has to be robustly addressed (as do author disputes re. right to reply, etc.). It is all about natural justice and being seen to be fair (as with all author dealings). A good publisher can be very helpful.
4. Writing clearly is incredibly important (and something perhaps easier to sell in a multi-disciplinary journal where there is a wide ranging audience who have even less tolerance for jargon). There is no better source for good writing than the wonderful Strunk and White ‘Elements of Style’.
5. Most editing work is undoubtedly about choosing referees at one end of the process and reworking those referees comments at the other. We always took the view that we had the right to do this, and that it was in everyone’s interests to, where necessary, tone down or in some cases civilize specific referee reports (academics are capable of saying some shocking things in anonymous reviews).
6. We had excellent administrative staff both locally and at the publishers. It is obviously true that smaller journals can be run well hands-on by a small group of committed editors. However, when you reach the critical mass of large-scale industrialisation (latterly on my watch we had more than 1000 submission a year), you need significant administrative support and you need co-ordinated editing and strong relations between both parts of the system.
7. I hated it at first but as an editor I learned to love manuscript central – the online editing/reviewing/ paper management/submission systems now ubiquitous. It can be customized (I guess if you have some clout with the publisher) and can be improved particularly in the critical area of linking to the flexibility of the reviewer database.
8. I became (and remain) unhealthily obsessed by those authors who insisted on the imputation of meaning from non-significant statistical findings (and also weak significance i.e. to 10%). In the end I instituted arbitrary rules in the journal guide for prospective authors – but did not go far enough.
9. Don’t believe special issues in the journal you edit will be easier to manage. Typically, they are not – though they are often relatively popular and more cited in the end than mainstream issues.
10. And yes, there have been almighty howlers, terrible titles, bizarre papers and even odder covering letters – they can keep the team going but of course cannot be repeated.

It has been said many times before but the system, warts and all, hinges on the timely supply of thoughtful and constructive reviews by referees. This is a widespread problem across academic disciplines and we, like other larger journals, debated several possible approaches to increasing willingness to referee e.g. entering into an implicit contract with authors that if they want to submit a journal article with us they should expect to review other papers as part of the deal.

I am the first to admit that though it has become easier for me to make a decision on most papers, I have written shorter and shorter reports overall. The editor of course cries out for substantive comment from the referee. Editors as referees do have the excuse that they are looking at large bundles of papers each month (I tried to aim for half a dozen short key comments for each one I read – but did not always manage to provide as many). So, next time you find yourself reviewing a paper do try to help the editor with constructive comments. Encourage others to do so too because it might be your paper next in the pile and you’d want helpful comments that point you in a stronger direction.

The future of academic journals is uncertain and the system is under pressure to change to meet the requirements of academics, readership and research funders. And, despite its problems (and many of us at some point will have experienced them at first-hand), peer review is unlikely to go away any time soon. I greatly enjoyed editing a big journal and I am convinced it was a great learning experience for me. But is nice that it is now over and I can happily live with being just a reviewer and an occasional guest editor.