Whitehall is abuzz with discussion about randomised controlled trials in public policy. The dark art has come to the water cooler.
RCTs offer the best way of telling if a social policy intervention does or doesn’t work, because they compare the outcomes for people who received that intervention with those of people who did not. This is a crucial but tough and sometimes costly test to meet – so it needs be seen as part of a longer-term evaluation strategy to move an intervention along the pipeline from innovation to proven impact.
A lot of work is needed before an innovation is ready for this level of scrutiny. You need to be crystal clear about what it is, what it expects to achieve, who it is for, the core components and, importantly, the underlying logic.
There is no point in creating something if it can't be delivered with consistency in a different place – it should be replicable. This is aided by having a manual, training, hands-on-guidance for practitioners and a way of keeping track of quality.
And you need strong indications that it is likely to work.
So what’s involved in a long-term, sustainable impact measurement strategy?
A first step is to work out what you are trying to achieve. What are your outcomes? Select a few – the two or three you think will most likely be affected. That is what you want to measure.
Next, figure out what the intervention is. What are its core components? You want to find out whether this is what practitioners actually do. If they don’t, and you get disappointing results, that may be the reason. Work out how to track these.
Then think about how you will measure outcomes. There are lots of options – structured observation, interviews, official records. A further option is to ask children, parents or professionals to complete a questionnaire.
If you use a questionnaire, explore what researchers call ‘standardised measures’. There are hundreds, covering everything from behaviour and health to emotions and education. Often they are free, brief and simple. Most importantly, they measure what they say they measure.
Lastly, think about when to measure. At minimum, do it before and after the service is used. The difference between these measures will give an indication of the distance travelled. It doesn’t, however, tell you whether what you did was responsible for this change.
This is because there are other influences besides your activities that will affect outcomes. You might be targeting reading but what if children get a new teacher or are simply maturing?
An example illustrates why this is important. One of our evaluations showed children whose parents attended a parenting programme were behaving better at the end than at the beginning. Great – the programme worked! Except that the same happened to similar children whose parents didn’t attend the group. The programme made no difference.
So how can you move from monitoring outcomes to measuring impact? The acid test is whether children who get the intervention do better than similar children who don’t. A comparison group helps us isolate the added value of our efforts.
This point is really driven home by programmes that try to prevent teenagers from taking drugs. As children get older they are more likely to take drugs, so more Year 9s would misuse drugs that Year 8s. If you offered a programme in Year 8 and found that drug use was at the same level a year later that would probably be a good result, but to know for sure you need a group to compare them with.
All things being equal, the best approach is to take the group of people who are right for your intervention and effectively flip a coin to decide who gets the innovation and who gets services as usual. This is ‘random allocation’ – the R of RCT.
Before launching into this type of evaluation you will want to have done the groundwork mentioned earlier, as well as some user testing. Do people use the intervention? Do they like it? What needs adjusting? Qualitative methods such as focus groups are useful here.
And don’t try to do this alone. You will need to commission external research expertise, and pay careful attention to ethics. Rigorous evaluation is not cheap, so it is worth doing the preparation before taking this significant step.
For organisations wanting to make the journey from innovation to proven impact, the Social Research Unit at Dartington's standards of evidence can help. As presented for the Greater London Authority’s Project Oracle, they set out a framework with five levels starting with 'good intentions' right through to 'system ready'. RCTs are needed to get to the top level but the standards also recognise the importance of the other preparatory work discussed above.
The moral: aim to step up a level at a time. Don’t try to go from zero to hero overnight.
Nick Axford is a Senior Researcher at the Social Research Unit at Dartington. More information about their work to support charities to evaluate impact and move along the innovation to impact pipeline can be found here: http://dartington.org.uk/our-work/childrens-service-providers/
Return to Blogs