In April 2015, the British Council Researcher Links programme and the Social Determinants of Health Network hosted a workshop entitled 'Towards comprehensive evaluation for health and development: promoting the integration of evaluation methods.' This blog reflects on some of the discussions and lessons learned at the workshop about how theories of change can help in the evaluation of complex health interventions. Health systems interventions tend to be complex by definition with many interacting components, and often some degree of flexibility built into the design, meaning that implementation is likely to vary across geographic areas and evolve over time.  Evaluating such interventions is not without challenges.

Theories of change - a useful framework for designing the evaluation of complex interventions

Today, we learnt that a useful starting point when evaluating complex interventions is the development of a theory of change.  A theory of change essentially is a theory about how an intervention works and is expected to lead to intended outcomes, as well as setting out key implementation steps. This is much like a logic model. It is a visual aid as it diagrammatically illustrates the expected implementation pathway. It provides a framework for designing an evaluation and choosing indicators and can be valuable in involving stakeholders in the design phase of an evaluation as well as getting their buy in to the process and ensuring the evaluation will be policy relevant. 

Lucy Lee, a Research Fellow at the London School of Hygiene & Tropical Medicine, gave an inspirational presentation on how to develop a theory of change through an interactive stakeholder workshop.  To develop a theory of change, the following steps are involved:

1. Identifying long-term impact
2. Backwards mapping and connecting the intermediate outcomes or requirements necessary to achieve that goal through causal pathways and explaining why these outcomes are necessary
3. Identifying the basic assumptions about the context
4. Identifying the interventions that your initiative will perform to create your desired change
5. Developing indicators to measure your outcomes to assess the performance of your initiative
6. Illustrating the theory of change visually
7. Writing a narrative to explain the logic of your initiative

This was followed by a presentation by Rishma Maini who reflected on some of the practical challenges such as engaging the right stakeholders, managing group dynamics and gaining consensus on the theory of change pathway during a workshop, and illustrating the value of theories of change to stakeholders.

Process evaluation – critical to the evaluation of complex interventions

One of the benefits of developing a theory of change is that it clarifies the expected mechanisms through which the intervention is intended to lead to changes in outcomes, addressing the ‘hows and whys’ as well as the whats – opening up what is sometimes termed as the evaluation ‘black box’.  To this extent, the approach can be very helpful in shaping a process evaluation which seeks to understand:

a)     Whether a programme was implemented as planned (whether there was implementation ‘fidelity’);

b)     And how a programme works (the underlying causal mechanisms).   

This distinction is important, as it allows the evaluator to differentiate between ineffective interventions (programmes that do not work) and interventions that may be effective but were just poorly implemented (where there was implementation failure). Process evaluation also sheds light on how a programme works and which components were critical to its effect (the core), versus the adaptive periphery (components which are less essential to replicate); as well as the role of context in facilitating or impeding programme functioning.   A theory of change also helps identify (and subsequently test) key assumptions that have to hold for a programme to be effective, as well as anticipate potential unintended consequences which might arise if such assumptions are not satisfied. 

With so much focus nowadays on the outcome evaluation, which aims to show whether or not the intervention has actually worked, process evaluation does not always get the attention it deserves and a theories of change approach is a good way of ensuring it is on the table from the start.