Outcomes and Change: New Thinking in the Third Sector

 

4 October 2012   Reducting Inequalities in Child Health - Outcomes and Change: New Thinking in the Third Sector   Sharon Witherspoon   I am delighted and flattered to be asked to speak here at Gresham College, an institution I have long admired, and that is partly because it does indeed share some of the same enlightenment values that the Nuffield Foundation, of which I am the Director, shares: that is it really believes in public debate and discussion, not in the form of a Punch & Judy show, but really trying to generate light rather than heat.     I want to start by saying that I am not a substantive expert on child health, certainly not in the narrow sense of physical health, but even in the wider sense that encompasses mental health and child wellbeing, where we do quite a lot of funding, I would not claim that I am an expert in the way that the previous speakers have been.     And since you will all have heard my accent, I can admit to being an American by birth, although I have lived here for 35 year by choice and I am not going to be supporting the American welfare system.   But, I am also not going to make the same kind of advocacy-based arguments, using evidence, but very clear advocacy arguments, of the sorts that Helen and Danny and Judy have done, about the need to use really Government public policy to reduce inequalities in order to improve child health. That is not to say I do not believe that. It is just not the point I want to make. I want to talk about charities and the third sector, which, by definition, although they can influence, they can lobby, not in a party political way, but they can influence those debates, but they cannot deliver those alone. So I want to talk about some of the things going on in the third sector, as Helen asked me to do, and to I hope get us to think a bit more about some of the good news stories, but not, I suspect, without getting a frisson of anxiety before I get to the happy news.   Now, just to say, first instance, we are a small funder by comparison with many funders. Our total spend each year is about eleven to twelve million. About seven million of that is in grants to others; the rest are activities like the Nuffield Council or Bioethics, which some of you might or might not know about it. It is not staff costs. And, as this slide shows, we span a range of things, including education, and we fund a lot of science and maths education. We used, for various reasons, to be much bigger than we are. We actually part-funded Jodrell Bank, and therefore we care about science and maths education. But our education work includes a lot of work on early-years, both for educational and other purposes.     We spend about 1.1 million, on children and families directly, including law reform relevant to children, early years in childcare, the child protection system and the evidence related to that, and about another 1.5 million on social policy, much of it research on social policy, much of it for work on poverty and disadvantage. So I would guess, all in all, somewhere between four and five million a year of what we spend is on work relating to children, one way or another. That is a large sum to be giving away if we do not think carefully about how best we can use it to make a difference because, after all, our fundamental aim is to make a difference. In the rather old-fashioned word of our trust deed, it is the “advancement of social wellbeing, particularly by means of scientific research”, and we take that rather seriously.     About two-thirds of what we fund is research, and the rest is practical innovations and evaluations of them. One of the larger projects we have had running for some time now is an evaluation of a family drugs and alcohol court system, which aims to bring together the court-based child protection system with fast, speedy and high quality delivery of drug services and other forms of family help to actually say, within a timeframe that makes a difference to a child, can we deliver some support and can we then make a better judgement about whether the child can stay with their family, birth family, or go into care.   All this activity has made us, and many others in the charitable sector, think much more carefully about how we know if various activities do make a difference. At the simplest: are they actually even good for children? If so, how good, how much difference it makes, and that sounds obvious, but of course, if you think about the charity sector, much of it has grown up from the grassroots level, based on the commitment and passions of people who see a problem and think, quite rightly, they want to do something about it, and I hope all of us can applaud that. But it can lead the sector as a whole to focus on what it is doing – mentoring, mediation, soup kitchens – rather than what is the ultimate aim. Is it actually changing people’s lives? Is it reducing the number of people who need soup kitchens? So, many of us now, over the last decade or so, have been thinking more and more about how the third sector has to face this challenge, given the very clear, passionate, emotional commitment of many of the people who work in it.     To start, of course, I want to flesh out a little bit, not in any length, about what is meant by the phrase “the third sector”. Many of you will know that, but some of you may not, and just as the private sector is diverse, ranging from Barclays Bank, on the one hand, to John Lewis or Apple, to your local newsagent, and the public sector, although it derives its funding ultimately all from public revenues, is also diverse – if you think about a central government spending department versus the Treasury, much less compared to a local council or a quango, but the third sector is just as heterogeneous, and I think one of the reasons that phrase became popular, as opposed to “charity”, is actually to help us think about that a little bit better. I recognise there is all the world of difference between a well-endowed foundation like my own, which works largely by giving money away for activities that link evidence and practice, between that and small charities, local charities, that organise local volunteers to do something for children, or the larger children charities that raise funds for a range of activities, who often have large workforces of professional paid staff and often work with the public sector, and, indeed, are increasingly dependent upon partnerships or contracts with the public sector. I think it is fair to recognise, looking back, it is partly these larger children’s charities that have helped nudge us to think about outcomes more carefully, and perhaps more sceptically.   I want to pay particular thanks to Helen Roberts herself, whose own particular contribution was immense, as she was one of the first who brought a “what works” perspective to the work of Barnardos, which is one of the larger children’s charities. A key question she started asking, with others was, how do we know what works? How do we know whether it does any good at all?  How do we choose where to put our resources? Barnardos raises some money from fundraising from the public – how does it choose where to put that?      I have to say, we acknowledge, in my part of the world, and I think, increasingly, people do, that there is an element of values in all of that, and not only is there, there should be. It would be wrong to think that we lived in a world where we really did have some sort of technically neutral expertise-driven set of decisions about a lot of these issues.     But even if the notion of being wholly evidence-based is not only unachievable, but actually undesirable, we can use evidence better to influence how to make a difference. So what I am going to go through is some examples of how we are trying to grapple with these issues, and to deal with that in particular in the third sector, the whole mass of people who work with a genuine desire to do good, but in a state of imperfect knowledge.   Here are a couple of themes I want to talk about…   Firstly, we have to start emotionally understanding the potentially humbling truth that plausibly good actions may not only be ineffective, they can be harmful. Now, that is so counter-intuitive when you know the emotions or the passions people are bringing to what they are doing, particularly where children are concerned. But I want to give an example of that and how that, you know, is a chastening experience to those of us who give money or the people who work in the children’s sector.   Secondly, we have to acknowledge that some programmes, plausibly good programmes I say – they look like they might be good – are simply more effective than others, and in a world where resources are always finite, and that is not just true because of the crash, or because of the public policy responses to the crash, but in a world where resources are finite, we need to always be thinking about that.   Thirdly, we have to acknowledge that, in vast areas to do with children’s health, particularly in the wider sort of interests that organisations like mine care about, which is to do with mental health and wellbeing in a different sense, we know there is a relationship with depravation and inequality, but we are not as sure about what we might do about it. Yes, we might do some very big public policy things to reduce inequality, but in the meantime, we also do not know how to get into this particular relationship, and that is partly because we just do not have the information about it. I think that means we need to embrace the partial nature of what we know, and embrace it enthusiastically. That is not a reason to get depressed. It is a reason to think smarter about the iterations between what we do and what we find, and I will give you some examples. I think that is an issue most people now understand better, but we do not do much to put it into practice. We very rarely evaluate anything the charitable sector does.   We also need to use that evidence to devise things longer term to keep us on our toes, and I am going to talk about some promising signs there.   Finally, we need to say values do matter. There will be different charities, just as there are different people, in a democracy, that have different values, and one of the reasons we need to be careful about using evidence is because that is a way we can sometimes get agreement on higher level values and then actually say, well, does it work to do what you think, or is the choice of what you are doing actually not going to deliver what you want. A classic example is trying to increase the number of stable partnerships, if you will, or marriages among people who have children in deprived backgrounds. I think there is very good evidence that family stability matters, and it matters for any of us. I am not saying people should never get divorced; I am just saying we would all like it if people had children and stayed in loving relationships. But that the answer to that is marriage counselling, you know, when things are going wrong, or a couple of health visitor classes early on, is pretty underwhelming, so if we think about that, we have to think about how we do things, not just what we are trying to do. That is always an important caution.   And, of course, we have talked a bit about this and I will not belabour the point: we do need to think not just about the small interventions of the small charities but whether any of that can actually lead to discussions about bigger level changes, because, for reasons I think you have heard, there are some reasons to think that some of these things require more thorough going changes.   I am going to just allude to the fact that, for instance, with the Government’s plans for troubled families, which they notionally decided, and the previous Government had a thought about this too, that about 120,000 families were “the problem families” and we were going to intervene in some ways, which are rather unspecified, to do something about them, and these are troubled families – I mean, they are in touch with a range of social services on a range of issues, so I do not deny that, although I do not think the 120,000 estimate is very clear. I think the problem is whether doing something about that requires some of the same arguments about institutional change, and, again, I am going to just end with some thoughts on that.   But first, a good, and I think convincing, example of how a well-meant intervention can do harm, and I always think it is really important for people to spend just a minute thinking about it because, actually, we all like to think that our commonsense will get us through a lot of thinking about this, but it does not always. I have taken it out of the realm of current contentiousness, and am saying I think one of the best examples is the Cambridge-Somerville Youth Study.     That study started in 1935. It was based on a humane view that young boys at risk of becoming delinquent might benefit from some counselling, they called it, but it was actually much more like mentoring, about a couple of hours a month of mentoring, where they worked with the boys, they talked to them about what they could do with their life, they did refer them on to get charitable help if they needed it, or doctors’ appointments, so there was some practical help involved, but a lot of it was a kind of forerunner of the mentoring programmes that are still in use here with young people at risk of delinquency. The programme was unusual in that, because it was done in Cambridge, Massachusetts, so Harvard University, there was a very well-designed study of it.     What they did is they took 506 boys – actually, there were more, but these are the ones they followed up long-term – who were aged five to thirteen. They matched them by whether they were at high or average risk of getting arrested, getting in various kinds of trouble, and a very clear set of criteria they used to get these here are a group of boys that we think are at high risk, here are a group of boys we think are at average risk. They then randomly allocated those to getting a treatment or to having nothing, just watching and interviewing – and there can be effects of that, but I will not go into that. The treatment was this counselling or mentoring, about twice a month, for five years. So not insignificant, but it took a lot of time and money. It did direct boys to various agencies and it unlocked some summer camp participation, as I said, some health and so on, and it was explicitly also aimed at talking about what they could do and what life challenges there were.   Well, what were the results? I have slightly given it away because I said we need to think about things that look plausible but that does not deliver what we want. I think it is fair to say, in the short term, the measures they first used, whether the boys and their families continued to take part, whether they liked it, they liked it, they really did like it – that looked like a success story.     Then, in the 1970s, the study was rediscovered. I do not think it was quite like going into a library and finding an index file full of names of addresses, but it really was picked up again by a different group, and they not only traced the boys and went back to interview them, they got permission to look at official data on arrests, to look at various health treatment services and so on, because they were worried the people they might trace or they might interview might be different from the people who wouldn’t agree to be interviewed. That was true too.   What they found was that there had been no impact on juvenile or adult arrest rates, no difference in the number of serious crimes or the age of first committing crimes, or the age of ceasing crime because, of course, as we now know, many even very troubled young people do stop after that peak period in their early to mid-twenties, no difference in treatment rates for alcoholism or mental health, but the treatment group was worse in the number of crimes they committed, in the self-reported levels of their alcoholism, and they died younger. This was harmful. I remember first reading about this and, actually, your heart quivers – it is a physical reaction…this is harmful, particularly because, when they interviewed them, the subjective evaluations were positive. The men all said, “This was great, this changed my life, you know, it was the best thing that ever happened to me.” There was caution there too.   So what does this mean? Does this mean we really should not like these kinds of interventions and want a short, sharp shock? No, because the social scientists were better than that. They, led by a team, led by Joan McCord, but there were many people involved, they were at pains to try to examine rigorously why the results were the way they were, and the hypothesis they came to that was supported by the evidence said that the damaging effects were caused by the fact that the constant attention and encouragement by the counsellors raised expectations beyond the point that there was any realistic chance the boys could fulfil them. They did not have the practical help to get better education. If they fell and they stumbled into the criminal justice system, they were not picked up. It did not make real differences to their lives. It raised their expectations, and that was the reason the subjective assessment was so high.     But it does make us think about some of the kinds of things we have been talking about, about what does it mean to make a real difference in someone’s social life, and what it meant in this case was it could have given the boys practical tools for coping, vocational education, reading skills, etc. etc. and it also probably suggests we need to think about dose response. Can a relatively shallow, intervention turn people’s lives around?   There is another important lesson here, which is that we are all very reluctant to produce evidence that said we tried something and it didn’t work. Medical trials are acknowledging that problem all the time, although big money means that it is continuing to be an issue, and those of you that read Ben Goldacre will know this, but it has been known for a while, but in the charitable world, we have the same problem: so many people working because they have humane motives and convictions find it very hard to even want to ask the question, “Does it work?” because, of course, they are worried is that going to undo the work of their charity.     These have been some of the problems that underpinned our problems with Sure Start, where actually, there is very good evidence that it does good things, but it could have been so much better if they had paid attention to evidence rather than some of the advocacy, some of it led by the charitable sector, about what they should include in Sure Start.     I will now be going much more quickly, but I do always like to just be clear that that is the reason why we pay attention to evidence. It is not to make careers for social scientists. It is not because we believe somehow that it is magic. It is not because I believe it is always very convincing at getting Government or whoever to do what they ought to do. It is because it improves what we do.   In childcare – someone raised that earlier, and I decided to sit on my hands – there is some really good evidence about early-years’ childcare, but there are some surprising gaps that we would not have if it were a medical issue.     For instance, we know that children from more disadvantaged homes really can have a difference made to their lives if they get good, high quality, early-years’ childcare. It is not the same as reducing inequality, but it can ameliorate that. What we do not really know much about is what “high quality” means. It tends to get reduced to just the number of staff, the staff/child ratio, rather than the quality of staff. We know a bit about it, but we could know more. We really are at the early stages of understanding that, in the context of resource constraints, and there are some eminent social scientists who are working on all of this.     We have not really got as much idea as we would like to have about the outcomes we should be focusing on. If you look at the last Government’s and this Government’s proposals about publically funded or supported early childcare, there is a lot about school readiness, but that is often interpreted in terms of can you recognise letters, can you sign your name, can you tie your shoe, that sort of thing. Now, those are important, do not get me wrong, but there is a growing body of research, which is elegantly demonstrated by the work of Professor Terrie Moffitt and her colleagues, that shows that, independently of IQ or income or cognitive measures, self-control has a long-term impact on life chances – that is health outcomes, ranging from early drug-taking, car accidents, etc. etc. to all sorts of things, health and wealth and wellbeing outcomes. And of course, much of what we know about the early-years’ education in Northern Europe focuses on just those issues: learning to take turns, learning to plan, learning to work in a group to try to do something, part of which is about learning self-control, and self-control in a social setting. Indeed, there is some evidence that being in mixed groups, i.e. having universal nursery education, is not only particularly good for the disadvantaged, it is also fine for the better-off because, for young children, the differences are less stark. So that might make us think about moving from an intervention to a policy.   It all gets rather messy because we really do not know about a lot of this. You know, we do not have the evidence that we could even make a rational case in a very strong and robust way. We have lots of bits of things that some of us read as pointing one way and other people might plausibly say does not.   My view of this is we have to think this is just a vortex, we are caught in a funnel, and the question is the relation between evidence and what we do and how we argue for bigger changes and more evidence in what we do, and we are not going to ever reach a state of perfect knowledge. We should not worry about that, but we can choose what kind of vortex we are in.  So, we either get into a world where some of us disregard evidence or some of us use it badly to under-claim or over-claim, or some of us are at the mercy of events, versus the vortex where we know we are going upwards, we are going up to the light and the air, it is beneficent, and we are really feeling that this is a positive thing. I am not naïve that evidence alone is enough, but I do think it is something we need to be paying more attention to.   So, the last three quick examples.   The first one, I expect to be contentious, but I am going to raise it anyway: the Robin Hood Foundation in New York City, which is funded by hedge fund and financial managers, was funded in the wake of the much smaller financial crash of 1987, and takes money from the rich, though not through taxes of course, through charitable contributions, and makes grants to non-profit groups, with the aim of reducing poverty in New York City, and at least partly with the aim of trying to reduce root causes, so there is lots of work with children and families. In 2008, it gave away about $130 million, so a big amount of money.   In the last few years, it has introduced a novel way of helping it decide what to give money to. I will describe this in full. There is a very nice Harvard Business School case study, for those that way inclined. In essence, they use social science literature about the effects of various things, based on proper evaluation, so the effects of teaching reading or maths, the effects of interventions on health, the effects of buying scanners in health centres, the effects of art therapy versus basketball club – you name it, they do it, and they look at these things, and they say is there evidence. They ask the organisations they fund to collect good information about the number of children they help, the number of staff they pay, the number of unpaid volunteers, and they try to get an estimate – and I mean it, a dollar estimate – of much return there is on various types of programmes.   When you first read this, I am sure you find it worrying: oh my God, we are measuring everything by money, it is absolutely appalling!  Is it not a bit like NICE, the National Institute for Clinical Excellence, which of course has to make everything, or starts out with the notion of quality-adjusted life years, based on empirical evidence about what kinds of things people find challenging or difficult, or make life not worth living versus worth living, and then looks at the cost and then says how much does it cost us to extend life by a year. There is a point at which you say life has an irreducible value, but if you are rationing resources, what do you do? I think it makes one think, and I certainly felt some concern, but when you look at what they do, the Robin Hood Foundation does not use their figures as some absolute religious goal, if we do it ten to one, then we are going to fund it, and if not. But what they have found is that looking at things in this way, (a) keeps them on their toes about evidence, about outcomes in particular, and (b) can inform their judgements – not determine them. There is a very good discussion in the Harvard Business Review case study, but also you can read about it in their work, to make you realise they really are not using it in that way. Let me give you a couple of examples.   You can see that they have got different years here, so they have got the benefit to cost ratio for different years, for different types of programmes, and what you can see is that those estimates jump around. The first set of red circles, 70 to one, versus the fifteen to one return. That could be because the social science evidence was pretty poor and it got better. It could be because the inputs and outputs changed. It could also be because the particular programmes in that area changed. They are not saying there is something really true about 50 to one. What they are saying is: it is in this area and we need to think about that rate of return. If you look at the bottom circles, you can see that health technology may have less of a return than other kinds of public health measures. That is a nice thing to know. So, it does help you talk about it.     I do not know whether it is apocryphal or not, but I was told by someone who knew about them that this had caused quite a frisson in their early days with particular kinds of things they funded. For instance, they used to fund a lot of art therapy, and you can imagine why: it allows children to express their feelings about what are often very frightening parts of life; it brings work to under-employed artists who are popular with grant-givers generally; and it has tangible outputs – all that lovely art. But if you look at it and think about what does it do for their chances of getting out of long-term poverty, compared to maths or English or, you know, basketball or whatever, you might wonder whether it is where you should put your funds, and I gather there was quite a long discussion about that in the early days, about maybe they should not put so much money into this. So, it does help keep our hearts somewhat under the control of our heads.   The second interesting example going on in charitable giving is the Big Lottery, that is the National Lottery, programme on improving futures. This is a programme that is giving £26 million over a five-year period to transform outcomes for children living in families with multiple and complex needs – a bit like troubled families. So each sector project has been awarded up to about 900,000 to deliver their projects. It is aimed at some longer-term transformations. It is not just saying: how can we produce a little change over these years? But it was based on using evidence to help select the projects, something the Lottery I think has been rightly criticised for not doing – it was just “This is evidently a good thing – we must fund it,” as opposed to saying, actually, in some of these children’s programmes, we should be using evidence to say what works better than others, and it is collecting an independent evaluation afterwards, looking at the effects of what those have done and also their cost effectiveness. It is also committed to sharing – I hope they mean pushing – that evidence to a wider group in the public sector and elsewhere.   That illustrates the issue about pluralism and values too, because, after all, Big, or the National Lottery, is funded by a proportion of profits from the lottery and has to be publicly legitimate and accountable. I am not going to refer to the issue of whether lotteries are a form of exploitation of the poor, given who takes part – I really think that’s an interesting issue – but I am referring to the point that, although Big has often given funding courageously to unpopular causes, like migrants or people with mental health problems, it does understand that it’s got to be seen to be serving the nation.   You can see from this map that there is a geographical spread, and I am sure they took that into account in their decision-making, and I am sure they looked at different types of interventions in sectors. But it can show that it is adding value above and beyond the grants by doing this and by using that to push its message.   Finally, two little projects we are funding, or not so little, about childcare, and both are trying to make precisely that move between the small-scale intervention to what can we say about policy.   The first one is at the LSE and it involves looking at a range of international evidence about what we think we know about high quality in early childcare, then it looks at what disadvantaged children in this country actually get, because some, in Sure Start areas, or local authority nurseries, or what used to be local, will get very high quality childcare, and some will not, so we need to look at that. Then it looks at four or five other countries that use universal or tax-funded or you have a voucher system or the market, like we in the United States have, to see what difference it might make if we delivered it a different way, in terms of children’s outcomes. I think that is clearly valuable.   The second is related, but has some overlap, but it looks at the effects not only on children’s outcomes but on mothers’ employment, and that is because, there, we do know quite a lot about why mothers working can be good for even young children – not for infants. That is a good argument for long maternity leave – but that for young children, mental health outcomes improve if mothers generally are working, if they are able to work, and again, I am not talking about forcing people, but if they are able to work. We know it increases their long-term economic benefit, and, crucially, it reduces poverty, and that is really important. So, this study looks at those two outcomes together. Again, it looks much more carefully at the costs of different ways of providing childcare, including whether it makes a difference not only who pays for it, Government or not, but how it is paid – is it paid by Government through taxes, through childcare tax benefits, how efficient are those, in narrow economic terms? Then it looks, again, at the same quality trade-off.  I think the question we are really asking is: does addressing these issues using childcare require rather a larger policy change than simply giving people an extra hour or two of childcare a week? I am not naïve enough to think that is going to change any policies in the short term, but I think we do think this is a useful thing to do to get a range of people across all political parties, and across many childcare workers, thinking about it.   I hope I have persuaded you that asking these difficult questions about what works really matters, and not only that, it should matter, and that it is not just a story of doom and gloom, which many of us working with research have often felt, that research is much better about saying what does not work. There are stories of kind of optimism and hope and it is something I think all of us in the charitable sector can and will be paying much more attention to.       © Sharon Witherspoon 2012