Theory-based evaluation approach: GSDRC Helpdesk Research Report

There is a wealth of literature available on theory-based evaluation and impact evaluation (TBE), but experts disagree on whether TBE is a common and clearly defined approach. Some think a common conceptual and operational understanding has been elusive, while others point to a completely consistent basic concept regardless of slight differences in the use of terminology. Some core features appear consistent across the main accounts of the TBE approach: Opening up the black box to answer not simply the question of what works, but also why and how it worked. This is key to producing policy relevant evaluation. Understanding the transformational relations between treatment and outcomes, as well as contextual factors and aiming to identify the ‘mechanisms’ that make things happen. This goes from asking whether a programme works to understanding what it is about the programme that makes it work. Having two key parts: conceptual (developing the causal model or theory of change that underlies a programme, and using this model to guide the evaluation); and empirical (testing this theory of change to investigate how a programme causes intended or observed outcomes). Being issues led, and therefore, methods neutral. Some of the variations in TBE strategies are: •Approach to types of theory: whether the black box is empty, full of theories or inhabited by people, and the implications for how to accumulate knowledge and establish the theory of change. Approach to causal inference: the realist evaluation approach adopts a generative approach to attribution seen by some as distinct from other (i.e. experimental) designs; other approaches promote the use of a range of techniques and tools to make counterfactual comparisons under the TBE approach. This review highlights the following key points from the literature. Some promote the benefits of applying a TBE approach to experimental designs. Much of the guidance proposes the use of mixed methods (quantitative and qualitative) while leaving open exactly how to go about choosing the appropriate design of mixed methods. Few studies apply the approach in practice. Guidance has been developed on the TBE approach and tools for evaluating complex and complicated programmes.