First used for clinical trials, then for testing in aid projects in developing countries, and next for testing web design, randomised controlled trials are now making it onto the domestic policy scene. The UK government's Behavioural Insights Team released a report last month advocating the use of Randomised Controlled Trials to robustly test public policies. Entitled "Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials", it brought in Ben Goldacre ( the Bad Science boy) and Prof David Torgerson to work with the BIT team.
Domestic policy is still implemented and tested haphazardly, which can incorporate a number of different biases and mean that nationwide policies or policy changes are implemented without truly robust proof that they will be effective.
Randomised controlled trials (RCTs) can better test policy long before it is implemented on a wide scale due to a number of clever design features, including a randomly assigned control group, which enables testers to compare the effectiveness of a new intervention against what would have happened if they had changed nothing. RCTs also remove a number of cognitive biases from research:
1) Selection bias. In research, if an experimenter "knows which group a potential participant may be allocated to, before the participant has been formally recruited to the study, this may affect the decision to recruit them at all." For example, the report mentions a fictional 'back to work' trial where selection bias may occur if researchers consciously or unconsciously choose not to select 'no hopers' to participate in the trial. Another occurrence of selection bias can also occur when allocating participants in the treatment group or the control group. Ideally, all participants should be allocated to the different treatment groups using a random number generator.
2) Confirmation bias: Before the trial has begun, researchers should set out the specific outcome they are trying to achieve, eg get people with outstanding fines to pay up and also specify how they will measure this outcome. "At the end of a trial, there may be a lot of data on many different things, and when there is so much data, it is inevitable that some numbers will improve - or worsen - simply through random variation over time. Whenever such random variation occurs, it may be tempting to pick out some numbers that have improved simply by chance and view those as evidence of success. However, doing this breaks down the assumptions of the statistical tests used to analyse the data because we are giving ourselves too many chances to find a positive result."
3) Experimenter bias: Ideally, the trials should be conducted double blind, so participants should not know if they are in the control or treatment group and those conducting the trials should be people who do not know if they are treating the control group or the treatment group. If they know they are dealing with the treatment group they may make extra effort to get the participant to respond to whatever the treatment might be, more effort than they would make for another person and may also be biased when deciding whether a participant has responded to the particular treatment.
Conversely, RCTs can also stop an ineffective and even damaging policy from being rolled out, saving government money. The report quotes the case of the use of steroids to treat head injuries. Medics believed that steroids would reduce inflammation and swelling, crushing their brain and causing death. However, a formal RCT run in 2005 to test outcomes showed that steroid treatment was more likely to cause death! The results of the trial were so extreme that the trial was halted early to avoid causing more deaths. Sometimes it is better to do nothing, stick to the status quo, than make a costly change.
