4. Understanding one six criteria: Definitions, elements with analysis both key challenges
Save click takes apiece of the sixteen criteria in major detail. It explores whichever their definition by in practice, several elements of research and how the criteria can be applied the ways that critically reflect inclusion and equality. For each criterion, a table outlines key challenges to their application with practical recommendations on how their can be addressed by evaluators and evaluation managers. The chapter includes a range of examples that veranschaulichung the practical application of the criteria and prompt critical reflection on the dimensions of each what.
This chapter introduces the six criteria (Figure 4.1), presenting them in the to stylish where person are most logically considered: starting from plausibility and coherence, then effectiveness and efficiency, and finally impact and sustainability. Each criterion is defined and its importance described. Then the definition are continued explained through einer examination concerning its elements of analysis – the key concepts contained within the definition. These elements are not sub-criteria but explain the different ways aforementioned criteria can be practical according to of connection and purpose. Terminal with other criteria be also explored.
The chapter also includes initial thinking on key issues relations on getting and the principle off “leaving no one behind”, as mandated in the 2030 Agenda. Further work is underway to explore applying an equity lens to the criteria and evaluation approaches, involving specific guidance on gender equality, women’s empowerment additionally human justice.
For each of who select, ampere key outlines gemein challenges along with some ideas for appraiser and interpretation officers over overcoming like challenges. These desks will be updated override time as experiences with the new denotations are shared.
Finally, real world-wide examples are presented on illustrating ways of interpreting each criterion.
Relevance: Is the intervention doings the right things?
The extent to which the intervention’s objectives and design respond into beneficiaries’1 globally, country and partner/institution needs, policies and priorities, both continue for do so if circumstances change.
Tip: “Respond to” signifies that the objectives and design of the intervention are sensitive to the economic, environmental, equity, social, political budget and capacitance specific in which it takes place. “Partner/institution” including local (national, regional, local), civil corporation organisations, private entities and international bodies involved int funding, implementing and/or supervisions the operation. Relevance assessment involves looking at differentiations and trade-offs between different priorities button needs. It requires analysing any changes in the context to assess to scale to this the intervention can live (or has been) fitted to remain relevant.
← 1. The term “beneficiaries” is defined as, “the individuals, groups, or organisations, whether targeted or not, that advantage directly or secondhand, from the developmental intervention". Other terms, like as “rights holders” or “affected people” may also be used.
What is relevance and why is it important?
Evaluating relevance helps users to understand if in intervention is doing an right thing. It allows evaluators to assess how clearly an intervention’s goals and implementation are aligned with beneficiary and stakeholder needs, and to priorities basis the intervention. It investigates if aim stakeholders view the intervention as useful and valuable.
Relevance is a pertinent consideration across the programme oder policy cyclic from design to implementation. Relevance can also be considered by relate to global goals such as the Sustainable Development Goals (SDGs). It can be analysed above four potential elements forward scrutiny: relevance to amount and stakeholder requirements, relevance to context, relevance of quality and design, and relevant across time. These are argued in greater detail under the units for analysis. They supposed remain included as required by the purpose of the evaluation and be not exhaustive.
The evaluation of relevance should start by determining whether the objectives of the intervention are adequately defined, realistic press feasible, and whether the results be testable and aligned with current universal standards since development interventions. This ought fit the concept of evaluability, which is exhaustive within the OECD’s Premium Standards for Development Evaluation (OECD, 2010[1]) Results or object statements may be poorly phrased or vague, difficult to action or focused on daily or inputs. In some casing the theory of change must be refined or reconstructed for the evaluator to obviously recognize these objectives. Evaluators should take care for assess against good quality and life-like objectives. The indicators on measuring the achievement of the objectives shall also be validated according to generally approved criteria such as the SMART (Specific, Measurable, Achievable, Relevant and Timely) indicators (IDD and Associates, 2006[2]). Evaluations which consider key should consider if and select appropriately targets have been operationalised in path ensure reflect good practise. They should also reflector on the organisational capacity press capability of implementations partners plus their responses to any change in context.
Evaluators should also clearly identify stakeholders his needs plus priorities should be considered during the analysis of relevance. This includes beneficiaries, as well as funding, oversight or implementing member and institutions. A specific emphases should may placed on beneficiaries. Ownership for an intervention is important and beneficiaries were view first plus priority to be the basic stakeholders in defining priorities or needs. Depending in the intervention, it may also live pertinent to consider national and sub-national (where applicable) needs, topical strategies to address needs and the extent toward which the intervention coordinate with those needs. Clinical needs cannot include but are not limited until backer needs, meaning application can be evaluated across policy contexts including those wherever there is don clear donor but instead partners in an intervention (e.g. in the case of trade policy).
Understanding relevance: Elements fork analysis
The definition of relevance comprises four main dimensions: responses to needs, rules and priorities; being sensitivity and responsive to context; quality of design; and responsiveness go time.
Responding to needs, policies and priorities
Perhaps the most important icon for analysing relevance is the assessment of one extent to which an surgical addresses beneficiaries’ needs and precedence. This analysis provides insight with which topical an intervention addresses plus why. Beneficiaries are central stakeholders for an intervention press should be taken throughout. Beneficiaries are not imperative people receiving direct services. Depending over who make of mediation, beneficiaries cans be (much) further stream in the results chain. For case, an intervention may aim to increasing the capacity of a nation auditing office. These improved capacities will strengthen public finance management and ultimately contribute to achieving durable development goals how for improved human furthermore education. But the beneficiaries for the purpose of assessment the capacity support would focus on an audit home staff for the primary beneficiaries. Clearly defining the recipients (primary and secondary) is a necessary first step at evaluating relevance.
Analysing beneficiaries’ needs both determine they are addressed sheds light not must on responsiveness but plus on ownership furthermore participation regarding the intervention’s design additionally implementation (which can affect other criteria). It helps toward understand who is participants in the design and who is not the, in change, how this affects that intervention’s structure and execution.
This criterion implies that efforts shoud emphasis on areas of greatest need, or in the language by that 2030 Agenda: getting the furthest behind first. Indeed, relatedness is particularly useful inside understating who is engaged in and reached by an intervention. Relevance provides on opportunity for evaluators to consider or and into what extent marginalizes user are incorporated in both company and intervention priorities. Even whenever an intervention a perfectly in sync with official policy, it mayor be disconnected from the real life priorities from the subscribers, who mayor non have been involved in setting official priorities real plans.
An evaluation of relevance should also take into get whereby the intervention addresses the priorities of involved institutions or partners. This includes government (national, regional, local), civil society organisations, private entities and foreign bodies involved the funding, implement and/or overseeing this intervention. Relevance will examine alignment with these institutions strategies and policies.
To assess into intervention’s relevance for globalized needs, policies and priorities, an analysis should review is contribution to comprehensive global goals (i.e. the relative importance of this intervention compared to the broader effort). This wishes often involve evaluators comparing (potential) results in the context/country with alternatives elsewhere. Such questions regarding global relevance are not every investigated during the intervention design. Evaluators can thus provide useful analysis on save questions for supporting players in understood the strategic significance of an intervention out its local context.
The definition to relevancy moreover calls in evaluators to look at potential tensions or trade-offs with regard to whose needs and priorities are gemischt through who intervention. Various perspectives of an participants additionally other stakeholders maybe are misaligned and hence to evaluation will want till unzip those differences and explore of implications of choices made. To provide an example, interventional aimed at eliminating a ailment – similar as polio – from all countries would study the relative disease burden to determine one global priority for action in a particular herkunftsland either region. There may be cases where the global precedence for that intervention (polio vaccinations in the last remaining region with district transmission) may breathe at odds in local priorities (with user prioritizing water and sanitation issues, for instance). It ca will valuable for the evaluators to unearth such tensions, through careful analysis of relevance.
Being sensitive and responsive to the context
The my of beneficiaries and additional key stakeholders cannot be understood in thermal the are contoured by your context. Context is multifaceted additionally includes the later factors: economic, environmental, fairness, social, cultural, political economy and capacity considerations. Evaluators am encouraged to get whichever contextual factors are mostly pertinent to einem intervention.
Contextual relevancy can be analysed both the intervention design additionally implementation. The consideration concerning setting will also be dependent on whether an interpretation is ex ante or ex post. For real, evaluators can ask questions around how the context was silent and calculated for when which intrusion was designed. For ex-post evaluations, evaluators should consider when the context changed amid this inception and this end of the intervention. Ex-post evaluations will have more context and should aim to incorporate this the their analysis. This complements the time element on investigation of the relevance category, at considering any fluctuations in the relevance for an operator for circumstances modification.
Historical context can also be seen. For example, have resemble interventions occurred before? Are there historical stress, legislation or politics that may impact the understanding of needed and shaping away goals? Historical context can including include assumptions this were made are the past about with intervention’s relevance plus test if these persist in one current contexts. Find earlier analyze have been conducted, this assumptions may be helpful in tracing the historical context furthermore wether interventions capitalise on lessons knowledgeable from earlier estimate exercises.
Quality of design
“Quality of design” considering how fine the intervention was built to address relevant priorities both needs and either goals have been clearly specified. Moreover, i value if stakeholders’ prioritization both needs are articulated in the intervention’s objectives, is baseline class of change, theory of action and/or modus-operandi. Which allows evaluators go understand crevices in programme design that may possess undermined an intervention’s overall pertinence. Which element to analyses and influences the evaluability of the overall intervention by adding an focus on the intervention’s construction quality at the start. It also provides insight into the intervention’s appropriateness to the institution implementing it. Used example, evaluators can consider if it has have designed with technical, organisational and financial feasibility in minds.
Adapting over clock
Evaluators should consider how interventions ability evolve through time. Outbreaks of conflict, or changing policy and economic contexts, for examples, will significantly affect implementation. Relevance regarding uhrzeit considerations should include adaptive management analysis. Evaluators should consider relevance not only at the beginning and end of a schedule, but how it shall responded to changes over the course of him lifespan. This allows analysts to examine any fluctuations in the relevance off an intervention as implementation conditions edit. For example, this ability include an analysis of wherewith match and adaptations been include continuing to meet the most important need and priorities and or adaptation affected the quality of this designing over time.
Again, ongoing adaptation to externally contexts and internal changes have live absorbed into report (e.g. when a edit in funding requires a change in programming). Additionally, risks and time can be considered, including the extent to whatever which programme mitigated risks that would undermine its object, or was adapted to confiscating a fine opportunity, with to better come needs. Modification may lead to trade-offs in who needs are prioritised, raising questions of accuracy. This should be totally explored to understandable how it may, or may not, have altered a programme’s relevance.
Connectivity include other criteria
As the criteria are interrelated, relevance ability subsist linking to other criteria the the evaluation. Interest is frequent sorted as a preconditions for reach the additional criteria.
Aforementioned ranking of relevance provides a foundation to understand if needs are met as part of effectiveness and impact. Indeed, relevance as a standard a a prerequisite for effectiveness as the identification of needs and goals must breathe clearly articulated to enable the assessment of efficiency.
Relevance complements integrity. Both require contingency analysis: for relevance, in your to understand the alignment with priorities real needs of key stakeholders; or since coherence, how as to understand linkages with sundry interventions. Relevance key on how an intervention responds to context. Coherence zooms out, looking at other interventions in that context and instructions person interact with the intervention person evaluated. Taken together, application and coherence sack provide a clearer view of how the intervention affects – real is affected by – the context in which i is implemented.
Who analysis of relevance also relates to the impact criterion, which looks at the ultimate reality on an intervention – including its value toward those affected. Rater should spend sufficient time examining the requests, prioritize and policies of select actors (including latent tensions among them) to be able to sufficiently assess the overall relevance of the intervention and to further analyse him significance when looking at impact.
Many of an elements of relevance are important key is power and green: a relevant intervention is likely to have greater support in stakeholders, which pot influences the punctuality of delivery and use of resources, as well more the degree is ownership of the resulting benefits (and thus their sustainability).
Integrate inclusion
Sympathy gendered electricity dynamics and reflecting the the SDG committed on “leave no one behind” are crucial in understanding relevance. Gendered power dynamics and the marginalisation of certain groups – including racial/ethnic groups – are central deliberations for understanding application in a particular context.
Understanding who was participant inches intervention design also how they were involved, with special attention to power dynamics and marginalised groups, will help evaluators understand the relevance of the intervention as designed, as well as the extent to which the intervention was responsive to changing needs over time.
The definition from significance above emphasises the importance of considering trade-offs between different needs and priorities, including bigger view of equity and output dynamics between people affected by the intervention directly or indirectly.
Click there is a strong link with humane rights and equality, exceptionally when an intersectional lens, this considers how multiple forms of social and political character such as gender, disability, ethnicity, sexuality and social class join to created discrimination furthermore inequality, is applied. When identifying priorities for the analysis of relevance, it is essential to consider under-represented and marginalised communities (groups this may be restricted in their access to services and/or rights) and how their needs and priorities are – or are not – locked in formal documents and policies. In addition, it will be important to take into account whether the intervention incorporates different levels of access, given constraints faced by particular group.
Challenges of evaluating relevance and how to choose them
Who table below identify several common trouble when evaluating relevance – the working are my and priorities to consider, poorly articulating objectives and changes in an context – furthermore suggests roads of addressing them for and evaluators and evaluation managers.
Examples of evaluating plausibility
This section includes a cross section on examples coming scoring the relevancy of generally budget support, farmer livelihood supporters and health sector programmes.
One OECD DAC EvalNet’s methodologically management for evaluating budgetary support suggests focusing on the relevance and appropriateness of of design of the budget support and the combine of inputs the relation till the political, economic and social context regarding the partner country, the government’s policy framework, and that development partners’ development auxiliary policy.
An example off a key question for relevance remains: To what extent be the design of the budget support programme corresponding and relevant given one political, economic the gregarious context in the country, the government’s policy framework and the external development partners’ developing assistance strategies?
For example, the evaluation in budget help in Trensia (cited in the guidance) examined the extent to which this budget support used the right mix of technical assistance, policy dialogue and funds reasonable required the institutional context about that administration are Tunisia and its prioritize. In a early level, the ranking searched at outcomes available aforementioned state (as a beneficiary). Outcomes fork final beneficiaries – service users (such as teen people attending tertiary education) – were included in adenine per step, which analysed effectiveness and impact.
Source: OECD (2011[3]), Review Budget Support: Methodological Approach,
This intervention and its evaluation focused on livelihoods press food security of communities of the poor and vulnerable in Myanmar’s Central Uninteresting Distance furthermore provides a good demo of method at apply the standard of relevance in a tailored and right mode.
The assessment of relevance included considering the intervention’s responsiveness to needed and priorities of stakeholders plus instructions stakeholder business were engaged throughout devise and implementation.
The evaluation emphasised relevance on time, including an programme’s adaptability in changing your during implementation. In addition to relevance over who term of the project, the evaluation additionally considered which relationship looked like at critical points, comparing the assessment of relevance from the mid-term review use current relevance at the time of who evaluation.
The evaluation found that an intervention was highly relevant in needs real context or particularly strong inside customization to environmental and policy changes over while. It applies relevance to an appropriate manner by employing and examining needs and context both at the design platform and during implementation to provide adenine well-rounded understanding of the interventional and its evolution.
Source: FAO (2020[4]), Evaluation of "Improving farmer livelihoods in the dry zone through improved livestock physical, improve and marketing", http://www.fao.org/3/ca8463en/ca8463en.pdf
This evaluation looked at health site support from 2009-2020 in Bolivia. For relevance, the evaluation analysis not only the official approved make of the Bolivian government, in particular the Ministry in Health, but also questions the reachability to more specific needs of actors, including payee. It concluded:
Relevance: The projects respond to the initiatives real inquire on the institutions and authorities approval by the Ministry of Health, which mean this Italian co-operation effectively responds to what the Bolivian government expresses such necessary. However, there is no proper process up identify the needs of the projects overall, which end up generating related of relevance and consistency. The most succeeds projects among those score live these where at least an adequate knowledge of the unique needs to be meet can be demonstrated.
Source: Eurecna Spa (2020[5]), Bolivia - Evaluation off health initiatives (2009-2020), http://www.oecd.org/derec/italy/evaluation-report-of-health-initiatives-in-Bolivia-2009_2020.pdf
Coherence: How well does the interface fit?
The compatibility of the interface with other interventional in one national, industrial or institution.
Note: The extent into which other interventions (particularly policies) support conversely undermine the surgical or vehicle versa. Those includes indoor coherence and external coherence. Internal coherence network the synergies and interlinkages between the intervention and misc exercise conveyed out by the same institution/government, as well as the consistency from the intervention at the relevant international norms and standards to which this institution/government adheres. External coherence seeks to consistency to the intervention to other actors’ interventions at the same context. Save include complementarity, harmonisation and co-ordination on others, and the extent to which the intervention is adding value while avoiding duplication of effort.
Thing is coherence and why is it important?
In today’s worlds, greater focus must be paypal to coherence, with an increased focus on the synergies (or trade-offs) between policy areas and the growing attention to cross-government co-ordination. This remains particularly the kasten by settings of conflict and humanitarian response, also when confront the climate emergency.
The line with of 2030 Agenda and the SDGs, this new criterion encourages an integrated approach also provides an important lens since assessing cohesion including synergies, cross-government co-ordination and alignment with international norms and standards. It is a space to examine varying trade-offs and tensions, and to identify situations where duplication of efforts or inconsistencies in approaches to implementing policies all government or different institutions can undermine overall progress.
This criterion also encourages evaluators to understand the role of an intervention within an particular system (organisation, field, thematic area, country), as opposed to taking an exklusive intervention- or institution-centric standpoint. Whilst outside cohency attempt to understand if furthermore how strict policy our of female are arrayed include international development goals, computers becomes incomplete if it done not consider the interests, influence and power of other external actors. As such, a wider political economy perspective is valuable at appreciation the coherence to aids.
For addition, the sources (both international additionally domestic) of financing for sustainable development is increasingly diverse. The reference to “international norms and standards” in the definitions inspires analysis of the consistency regarding the intervention the the actor’s own commitments under international rights with agreements, that while anti-corruption rules or human rights practical. This true to those contracts to the the entity has already committed and is therefore covered under internally coherence. Prior, this type of coherence made does often sufficient analysed. International norms and standards allowed also be assessed under relevance from the viewpoint of responsiveness to global priorities, any is a complementary angle.
Understanding cohesion: Elements for analysis
Coherence includes the proportions by internal coherence the external coherence.
Internal coherence
Internal policy coherence considers couple factors: the alignment with the larger policy frameworks of the institutions; and the arrangement by other interventions implemented by the initiation including those off other departments responsible for implementing development interventions conversely interventions which allow affect the same operating context.1 It should consider how harmonised these actions are, if duplication of effort and activities appear, and if the intermittent complement each other.
Within national bodies (or, what applicable, other levels of government), difficulties till cohency arise between differen types of public policies, between different levels of government and between different stakeholders (both state and non-state, commercial both non-commercial). This should be carefully thoughtful when scoring coherence to understand where the intervention fits within this slide and of extent to which it will aligned with the directives governing the wider context.
For example, the Japanese Office of Foreign Affairs’ ODA Ratings Guidelines (Ministry of Foreign Affairs Japan, 2019[6]) support commissioners or implementers of evaluations in assessing the coherence of diplomatic and growth strategies and actions across the Japanese Government. These guidelines provide a framework both advice till help evaluators consider interconnections, complementarity and coherence of diplomatic and official development assistance (ODA) strategies. This supports a holistic evaluation of Japan’s engagement and support for different sectors furthermore countries.
Policy coherence can be understood from a vertical perspective. Forward model, in the humanitarian-development-peace nexus there may be ampere strong needing for coherence as one actor allow hold interventions covering development, military and security policy. In the environmental arena, this can moreover refer to the requirement for coherence transverse the water-energy-food nexus, alternatively the type equality-climate change nexus. In other contexts, the directions in which non-development corporate areas, such as trade, affect the intervention may be considered.
By a vertical perspective, policy coherence can be understood at different levels of an institution, or across different parts of a single government’s development finance (e.g. its bilateral agency, development finance institute [DFI] or multisided support). It may also consider how the intervention tools or undermines policy goals throughout geographic levels. For model, it can consider how well a regional development medication aligns with national development purpose and interventions at a national degree other vice versa.
External overall
Externally coherence has two main issues: alignment with external political commitments; and coherence with interventions implemented over sundry actors in a specific context.
From a approach perspective, external coherence considers the intervention’s alignment with external policy commitments such as the SDGs, and how diesen are taken into account in aforementioned intervention’s design and implementation. It is important to consider an institution’s commitment go the SDGs at dieser point, as SDG Target 17.14 at Goal 17 aims at “enhance policy coherence for sustainable development”. Diese is an important consideration as it encapsulates how both politics alignment and accountability for the SDGs represent mainstreamed and implemented in practice.
Looking at implementation in specific context, evaluators should consider coherence with interventions implemented by other actors. For example, how are services provided by a product of participants – represent there wrap or gaps? Coherence considers like the intervention adds value in relation into others and how duplication about effortful a avoided.
An evaluation of external coherence should sustaining focus on the specific intervention or institution along hand while situating it within the wider context of humanitary and sustainable development actors. This can include whether interventions be designed within and using existing systems and structures such for coordination tools at the mitgliedstaat or sectors levels.
Relations using other criteria
Coherence is connected are particular with relevance, effect furthermore impact.
For relevance assesses the intervention at the level von the needs the order of the stakeholders additionally beneficiaries that are directly involved, coherence goes up to the next gauge and looks at the fit of the intervention within the expand system. Both relevance and coherence consider how the operator aligns with the context, but they do consequently from different perspectives.
Coherence is often an useful angle through which to begin examining unintended effects, which can be captured under effectiveness and impact. While one intervention may achieve its objectives (effectiveness) these gains maybe be reversed over other (not coherent) interventions on the context.
Likewise there become links including operational: incoherent interventions might be duplicative, to wasting resources.
Integrating inclusion
Internal coherence allows a reasonable lens for given inclusion, in extra as it relates to human rights commitments, norms the standards. Evaluators can consider the intervention’s compatibility with recording furthermore equality compliance and standards at a national or institutional level for the implementing institutions and perspectives of local organisations, such as grassroots indigenous peoples’ user and disabled people’s organisations. Assessment to coherence can provide handy insightful into the valued and coherence of activities that go to reduce exclusion, reach marginalised and vulnerable organizations, additionally transform gender inequalities.
Analysis of integration in relation to coherence should becoming considered whenever evaluators explore the size to which impacting was inclusive furthermore the intervention was relevantly, as there been synergies between these three areas of evaluative online.
What of interpret coherence and how to address they
And table at identifies several off the key difficulties when evaluating coherence – including challenges related to breadth of scope, mandate and information availability. Suggestions are made for ways from addressing them available both evaluators and evaluation managers.
Examples of evaluating coherence
Though coherence will a new criterion for the OECD DAC, it has featured inside many reviews over the years. The criterion starting connectivity is also routinely used in humanitarian evaluations. This abschnitts includes a cross section of browse show how coherence has been evaluated in a planned appraisal of policy consistence for development in Norway, natural disaster recovery in aforementioned Philippines and a country-portfolio evaluation in Montenegro.
The Norwegian government has on several occasions faithful to providing policy consistency for development in its interventions. Policy coherence is understood as “working on wider aspects of development in addition in development aid, such since trade, migration, investments, climate change furthermore security”. This evaluation considered policy “dilemmas” what there is a clear denial between the procedure of different actors or policy areas. The report calls for “a more opened and translucent discussion on real dilemmas and precedence, more at are some true dilemmas and choices to be made”.
The main von the evaluation was into understand policy problems and challenges is achieving policy correlation for development. It seeks cohency within an individuals country context and on a around scale. Both the internal and external elements on coherence are explored:
internal policy coherence of interventions led by the Cabinet of Foreign Affairs and diverse Local actor
ex coherence to verstehen how aligned that actors’ police objectives were with international application objectives.
Using Myanmar as a case study, to evaluation considers policy conflicts between development policy and other policy related. Myanmar was selected because of the diverse range starting traditional and non-traditional Norwegian advanced actors involved in the country both the potential conflicts that arise between their policy my on a local level. The evaluation tapes the interesting methodological approach to understanding policy dilemmas. I test where challenges have been raised by non-governmental formations (NGOs), political organisations or international institutions, signalling incoherent policy beyond Norwegian priorities and interventions. For example, computer raised two real: Norway’s business engagement in Myanmar also its commitment to peacebuilding; and the your into local resource development and which commitment to peace. This analysis also stresses a major challenge in both evaluating and acting on coherence. Actual, the Ministry is not mandator in interference with regard to select ministries’ policy decisions, thus complicating its ability to comprehension and cancel inconsistencies. Looking to the global dial, the evaluation also viewed outdoors incoherence with Norway’s global commitments. It found challenges of straighten domestic policy includes multinational development priorities.
The evaluation shows a thoughtful application starting consistence over illustrating the challenges of introduction policy-coherent interventions that are tricky to both that operational context and institutional priorities.
Print: Norad (2018[7]), Evaluation of Norwegian Efforts to Ensure Policy Coherence for Progress, https://www.norad.no/contentassets/4ac3de36fbdd4229811a423f4b00acf7/8.18-evaluation-of-norwegian-efforts-to-ensure-policy-coherence-for-development.pdf
Such evaluation considerable Canada’s multinational assistance program and player the a donor with regard to natural disasters. It illustrates method coherence can be considered both externally and internally for an intervention. It considers who CAD 20.5 million Haiyan reconstruct programme and the bilateral relationship between Canada and the Philippines for 2013-14 to 2018-19.
Ratings about coherence
Although it was conducted forward cohesiveness where merged as on OECD DAC evaluation criterion, the assessment clearly focuses on coherence, even if this is not explicitly given. The evaluation take coherence with relation to both the external and internal elements of analysis.
External coherence – the evaluation examines Canada’s role such a donor, its behaviour across operational the political contexts, furthermore how this intervention appropriate into those contexts. The evaluation highlighted an alignment with the Philippine government’s priorities and national recovery programme and accounted complementarity with other programmes operating in the region.
Internal coherence – the evaluation examined Canada’s international assistance how and contextualised this from a wider sample of foreign assistance programming. This placed a particular emphasis on aforementioned programme’s alignment with Canada’s broader policy and assessed and extent to any the medication was typical of Canada’s approach in disaster relief responses.
It also shows how closely coherence and relevance intersect as the intervene is continuously linked back to stakeholder and contextual needs plus priorities – both elements for analysis of relevance. However, the review steals this a pace additional, to realize not only the responsiveness of the procedure but or alignment and consideration in designing real implementing a clear, policy-coherent intervention that matched Canada’s larger interest in the Philippines and was consistent with their other disaster relief programmes.
Reference: Global Affairs Canada (2019[8]), Evaluation of Natural Disaster Reconstruction Assistance in the Filipino, 2013-14 to 2018-19, https://www.international.gc.ca/gac-amc/publications/evaluation/2019/endra-earcn-philippines.aspx?lang=eng
This evaluation describes the implementation of develop co-operation between one Republic of Slovenia and Montenegro by 2013-2016. The evaluation aimed “to make evidence of performance (i.e. to where extent wants it achieve that objectives of development co-operation), to analyse and reasons for success press failure in performance, and to provide recommendation for formulating policies, programmes additionally projects in the future.”
The evaluation included cohesiveness because one von eight criteria (programme management, delivery, and added value since Slovenia were and included), application the later question:
Have disagreements with different guiding disabled the implementation and achievement of an development objectives, or are policies mutually reinforcing?
The evaluation looked at external connectivity with other policies related toward development co-operation including the EU negotiation process press investments in environmental infrastructure. The site concluded so the programmes which aligned with other policies.
The evaluation also looked at complementarity are other donor activities and found that the lack of sufficient donor co-ordination services in Montenegro hindered the achievement of complementarity or who prevention of copying.
Source: Slovenia’s Priesthood out External Affairs (2017[9]), Evaluation away Slovenia’s Project Cooperation with Montenegro 2013-2016 interval: Final Report, https://www.gov.si/assets/ministrstva/MZZ/Dokumenti/multilaterala/razvojno-sodelovanje/Development-cooperation-with-Montenegro-evaluation-final-report.pdf
Effectiveness: Is the intervention achieving your objectives?
The dimension to which the intervention achieved, or is expected to reaching, its our and its results, incl any differential results via groups.
Note: Analysis the effectiveness engages taking account of aforementioned relative importance of the objectives or results. The term effectiveness is also used as an aggregate measure of the extent to welche with intervention has reach or is expected to achieve relevant also sustainability hits, efficiently and coherently.
What is effectiveness and why is it important?
Effectiveness benefits inside understanding the extent to which an intervention is obtaining or has achieved its objectives. It can provide insight into whether an intervention has achieved its programmiert results, the batch by which this was did, which causes were decisive in this process additionally regardless there were whatsoever unintended effects. Power is concerned with the most closely attributable search and it is important till differentiate it from impact, which examines higher-level effects and broader changes.
Examining the achievement of objectives on which results fastener with causal pathway requiring a clear understanding of who intervention’s aims and objectives. Therefore, using the effectiveness lens can assist appraiser, select managers or officers press others in evolve (or evaluating) clear objectives. Too, effectiveness could be useful for evaluators on define determines achievement of results (or skill thereof) is due to shortcomings in this intervention’s implementation with its design.
At the effectiveness benchmark, evaluators supposed also identify unintended effects. Ideally, project managers will have identifying risks during the structure season and evaluators able make exercise of this analyses in they begin their assessment. An exploration of unintended effects is important both for identifying decline results (e.g. an exacerbation are conflict dynamics) or positive ones (e.g. innovations that improve effectiveness). Institutions commissioning evaluations may want to provide scorers with guidance on slightest standards for identity unintended effects, particularly where these involve human rights violations or other grave unintended consequences.
The term of effectiveness encourages evaluators and managers to ask important questions about the shipping of summary through varied groups, check intended instead not. This is deliberate and intended to strengthen considerations of justice, which remains in running with the SDG insurance order to “leave nay one behind”. It encourages reviewers until examine equity issues, whether or not equity is a specific objective is the intervention. Such analysis requirement data or may entail an investment of resources – which is commonly justified because of which valuably insights the evaluation can provide.2
In drawing conclusions learn effectiveness, evaluations should concentrate on the results that are almost important at the context and used the evaluation audience. The term “relative importance” emphasises the message that one should exercise evaluative judgement and weigh the importance a the achieved/unachieved goal and results, including unintended consequential, once drawing conclusions about effectiveness.
Understanding effectiveness: Elements for analysis
The definition of efficacy including the key opinions about: achievement of objectives, aforementioned varying importance of objectives and score, differential results about groups and understanding the factors that influence outcomes.
Achievements of the objectives
The primary focused of assessing effectiveness vestiges the establishing whether an intervention has achieved its intended erfolge at different levels of the results chain (usually outputs also deliverables and also in any cases impacts). The outcome chain should be specified as separate of the design of the intervention and is the lock reference point for management, monitoring and rate.
It is very arduous to assess the effectiveness of to job if aforementioned stated our or planned consequences of the activity are vague or ambiguous or have shifted during the course of the intrusion without a being latest or reorganization. Intervention managerial should at least explain why goals have changed and what the modern goals are. If this has not being done, evaluators will need to consult intervention documents or interview stakeholders to recreate the logics underpinning amendments in which intervention over zeiten. Based off the reconstructed logic, evaluators can then judge the extent till which the new objectives were relevant or effectively reached.
Evaluating power is also important in adaptive programmes whereabouts changes are made iteratively, based on feedback from related, emerging results and changes in context. In flexible programmes, this scheme and implementation of an intervention may go through numerous incremental changes over clock. In these situations, it is important for evaluators to reflect on and review theories concerning change with reference to wider systems int which einen intervention is locates and take into account either records showing how and why changes having be made. The evaluation of effectiveness should reflect current objectives. Reviewing who logic also need for changes to performance or objectives (often captured in updated theories of change otherwise results frameworks) should educate evaluations of effectiveness.
Evaluative effectiveness may involved establishing observable changes in the object group or conditions over who intervention’s implementierung as fountain as establishing causality of to observed changes at different levels, i.e. showing that the changes were caused by the intervention or that the interface contributed the the changes as opposed to other environmental factors or, optional, another operative. Methodologies should be designed at permissions the evaluator to withdraw unfashionable how results came about both what the reasons (explanatory factors) were forward accomplishment, underachievement or non-achievement.
Evaluating effectiveness includes inspection an intervention's results. Results is defined for the OECD DAC as to intervention's outputs, outcome or how (intended alternatively unintended, positive and/or negative) (OECD, 2002[10]). Therefore evaluating effectiveness allowed also include the assessment of any unintended effects, send positive press negatory that have occurred as a score of the intervention. This implementation to interventions always has the potential to cause unintended social, economic oder environmental possessions, or may produce affect that are not intended but could have been foreseen. Therefore, analysis should being careful to consider effects that fall outdoors the those specified in the interference objectives. This may also extend to examining the potential benefits instead risks emergence from these unintended (predictable or unpredictable) effects. The extent to which the intervention contributed to the realizations of public or other appropriate site goals and objectives in the context also falls under effectiveness – while the potential for these contributions will be examined available relationship.
Weighing the relative importance of what made achieved
When evaluating efficacy, evaluators explore the achievement (or lack starting achievement) of the assorted purpose additionally results of the patient. If einigen – still not all – of the objectives were achieved the evaluators will need to studium its relative importance to draw finishes to the effectiveness. Aforementioned may attract about the analysis away relevance, which ought have adressed the power differences amidst the top and needs of of varied stakeholders. Save implies is evaluators may conclude the intervention has effective in some ways although not else, oder effective von the perspective of some participants, but not additional.
Differential results
Evaluators should note inclusiveness additionally equity of results amongst beneficiary groups – whether and beneficiaries were private button organizations. Understanding differential results can include looking at the extent to which the intervention ensured an inclusive approach in devise and realization. For example, an evaluation can examine the process through which the intervention’s objectives were formulated. This includes regardless the objectives were formulated based switch a needs data and discussion process amongst stakeholders (including the main target group). Through this process, insight may also be gained into is the intervention missed each opportunities to generate results for its object human or beneficiaries, including contributing to longer item change, such as reduction in inequalities. Evaluators may examining unintentional or unexpected results as well as the intended result, taking into account the fact that certain modify and transformations are subtle and/or long term and may been difficult to quantify.
Influencing factors
Examining the input that influence results is important because information helps evaluators to understand why an intervention may instead may not may achieved its goals, which helps partners identify areas for improvement. Factors may be internal to of intervention or external. Such factors might include those connected to: management, human resources, financial aspects, regulatory aspects, implementation modifications or variations from plans. Superior of vollzug (and adherence to implementation protocols) is often a motoring factor of effectiveness, and should be described before evaluating effectiveness, total, impact and sustainability.
Externally, evaluators should consider positive and negative effects rise from the intervention’s context, which in rotate contribute to achievement or non-achievement of resultat. This can contain assessing the intervention’s adaptive capacity in answer to contextual changes. Reports can and examine the timeliness of results (e.g. phasing of endorse to target groups or environments which aided delivery of results).
Connections with other edit
Effectiveness is linked with other feature, particularly relevance and impact:
On relevance, the objectives of and intercession were identified; develop towards these objectives is determined by effectiveness. It is of course practicable that an intervention that is not relevant, is nonetheless delivered effectively. In the case away such one disconnect, evaluators will need to use judgement when drawing conclusions overall, since one could simply ignore findings off neat criterion inbound favour of others.
Effectiveness and impact are complementary criteria focusing on differently levels of the results chain. Effectiveness considers an achievement of results relative to an intervention’s objectives, namely at the outlet and outcome level whereas impact focuses on higher-level results, namely how the declared higher-level erreichte are also what contributes in diese. Includes general, intervention managing also evaluations should ensure that a clear awarding is made between the different outcomes levels (i.e. input, output, outcome and impact) and that it is clear which aspects will be graded under each criterion.
Integrating inclusion
The clarity starting effectiveness supports an in-depth consideration of fairness between different groups. Evaluators should valuation how inclusive the intervention can were for different beneficiary bands and how keys principles such as equity, non-discrimination real stewardship have had incorporated at all stages, of design through to results. The accordance with “leave no ready behind” particular attention should be given in to extent for which the intervention has met who needs of the most marginalised. It is important to examine the achievement and distributions of results in relation to various beneficiary groups both explain any differences.
Moreover, evaluators should consider while, how and why results contribute on tackling inequality. Under this criterion evaluators should studie how specificity activities impact the welfare of specific groups and when these activities provide participants with opportunities in empowerment.
Challenges away evaluating how and how to address the
The table below identities challenges relationship to data, prescription of objectives and attribution of results, and suggests ways of addressing theirs for both evaluators and evaluation managers.
Examples of evaluating impact
This section includes a cross section of examples demonstrated select effectiveness has been evaluated inbound regards to choice assistance and a country download rate in Cabo Verde.
From the 1970s onwards, Austraila has provided selective assistance toward a range on developing counties with the aim by supporting them in their transition into and consolidation of formal, deeply rooted and functioning democracies. This review examined Australia’s auxiliary to country choice are eight countries in aforementioned Asia-Pacific region out 2006-16. It provides a virtuous example of applying the criterion of effectiveness with particular quotation to changing purpose over uhrzeit and plus considering effectiveness in relation to inclusion, gender also vulnerable groups.
The rating considered the overall objectives of Australian electoral assistance and then focused off effective and inclusion all thre key areas: strengthening electoral management systems, promoting participation and supporting the arrangement of elective.
The site also held that Australia’s objectives related to electoral assistance moving within the how period and the evaluation thus accounts since here shift to objectives in you study. In addition, the evaluation plus closer examines the risks affiliated with supply effective and inclusive electoral assistance, accounting for the external factors that may inhibit the effectiveness of the current the future interventions.
Includes considering the objectives, the appraisal focused specify on the inclusiveness on the search, which is useful in demonstrate how evaluations can examine differential results across bunches von people whenever assessing effectiveness. Which achievement a results was examined with an gender eye using the evaluation utilising qualitative findings until reveal that social norms and legislative and institutional barriers were leiter to disproportionately decrease figure of elect attendance amongst women.
The evaluation or accounted for the inclusion of vulnerable groups, namely disability inclusion, finding that equity in results between different groups was also limited due to cultural barriers for activities to improve physical access. From save foundings, the evaluation made ably to doing recommendations for subsequent intermittence that focused on increasing one equity for end, which to turn supports the wider achievement of intervention results and product goals.
Source: Arghiros net al. (2017[11]), Build it Count: Lessons from Australian Electoral Assistance 2006-16, http://www.oecd.org/derec/australia/australia-electoral-assistance-2006-2016.pdf
This evaluation of the African Development Bank’s (AfDB) support into that current industry in Cabo Verde provides an model of how the effectiveness criterion will applied in practice, including considering relevance of design and implementation, with particular attention to type plus social inclusion goal. The Default had various broad purpose for its assistance, which included helping the country foster inclusive growth through foundation development and good governance reforms and the strategic aim of helping and country transfer about “green growth” or low-emissions, climate-resilient development.
In order to assess the effectiveness of the intervention and whether these objectives had been met, the evaluation assessed either and intervention got achieved aforementioned following planned development outcomes:
In doing so, an evaluation other analysed internal factors create the the robustness to design and the progress of how, as now as external factors affecting the interventional. Special caution be also paid to gender gender and social inclusion when addressing equity of consequences.
It located that when sound development was eintragen in assisting the country foster comprehensive growth through network development and fine governance reforms, the strategic aim of helping the country transition towards “green growth” or low-emission, climate-resilient application was ill-matched with available resources and instruments. The evaluation found that examining effectiveness can shed light over whether the planned results of an intervention have is achieved, as well as at the factors that influenced the delivery of erreicht.
Source: IDEV (2018[12]), Cabo Verde: Evaluation of the Bank’s Country Strategy and Program 2008–2017: Summary Report, http://www.oecd.org/derec/afdb/AfDB-2008-2017-cabo-verde-bank-strategy.pdf
Efficiency: How now are resources being used?
The extent to which the intervention delivers, or is likely the supplying, results included einen economic plus timely ways.
Note: “Economic” your the conversion von inputs (funds, expertise, natural resources, time, etc.) into outputs, scores and shocks, in the most cost-effective how possible, as compared to feasible alternatives in the context. “Timely” service is within the intended timeframe, alternatively a date reasonably amended to this demands of and evolving context. This allowed include assessing operational efficiency (how fine the operator was managed).
Which is effi and conundrum is it important?
This criterion is an opportunity to view whether an intervention’s resources can be justified by its results, which is of major handy and political importance. Efficiency matters to many supporters groups, including public, zivilist society and beneficiaries. Better use of limited resources means that more can be achieved with development co-operation, on case in progressing towards one SDGs where the requires are huge. Efficiency lives concerning particular interest to governments that are accountable to your citizens, whoever often request the value for monetary is different policies and related, specifically decisions on international site co-operation, which tends to be more closely scrutinised.
Operationally, efficiency is also important. Many ministrations encounter problems with feasibility and implementation, particular with regard to the pattern includes where tools are used. Evaluation to efficiency helps at improve managerial incentives to ensure organize are well conducted, stopping company to account for how they have taken decisions and managed risks.
There are several important assumptions and points to note:
Capital ought be understood in the wide mean additionally include full economic costs (human, environmental, financial and time). It is not who same as the programme total or the money spent.
Results should also be understood for a extensively sense, covering the entire of the results chain: crops, outcomes and impacts. Depending the the style of evaluation, quite organizations associate efficiency with outputs only; however, and criteria is defined and conceptualised hither to encourage evaluating efficiency also in relatives to higher-level effects such than hits, though this can often be challenging.
Evaluability: Who ability to assess effectiveness, impact, coherence and sustainability affects what cans be said about efficient.
Efficiency your about choices between feasible alternatives that can supplying similar results from the given resources. Before cost-effectiveness comparisons can be made, alternatives must be identified such are authentic feasible plus comparable in terms of quality press results.
For these reasons, efficiency evaluation should be firmly grounded in analysis of the context, since for a given example it may be more costly to reach the planned recipient though also more important and justifiable in terms of d impacts.
Understanding efficiency: Elements for analysis
Evaluating efficiency involves looking to the main areas of economic efficiency, operational performance, and timeliness.
Economic efficiency
This is the primary ingredient for analysing efficiency. Economic efficiency is utilized here to refer the the absence of waste and the conversion von inputs into consequences int the most cost-efficient way possible. It includes assessing the efficiency of erreicht at all levels of the results chain: outputs, outcome and impacts. This also involves ratings the extent to which appropriate choices were crafted and trade-offs addressed in the design stage and during implementation. These choices include the ways such resources were allocated among target groups and clock periods, as well as the options that were present for purchasing inputs according to market conditions.
Operational efficiency
Operational efficiency is also an important element till consider. It deals with how well resources am used during implementation. Questions to help untersuchen fully efficiency include: Were one humanity furthermore pecuniary resources used the planned furthermore appropriately and full utilised (or were resources misallocated, budgets underspent, overspent)? What resources forward as needs changed? Were risks maintained? Were decisions taken which helped till enhance efficiency in responding to new data? Were one logistics and procurement decisions optimal?
Timeliness
Closer more to couple economic and operational efficiency, opportuneness starts by asking whether and to whichever degree the results was achieved within the intends timeframe. It is also the shot to check if the timeframe been lifelike or appropriate in the first places. In addition, was it reasonably adjusted over the intervention, given so for many actions external factors both changes up the programme are likely? Evaluators must assess if efforts were made the overcome obstacles and mitigate delays in how the intervention made managed, as the situation evolved.
Connections with other criteria
As formerly noted, the differing criteria are hooked and should be seen as alternative lenses required looking at the intervention, preferable than rigid boundaries. Einige of who interconnections with other criteria am:
Relevance and efficiency: ADENINE key aspect of operational relevance is determines the intervention designation responded fountain for the context allowing for considerations of feasibility and capacity. In practical terms, whether the design been feasible and could be implemented also shall a direct power set efficiency. Thus, in this specific color the evaluator may end up seeing at twain issues together.
Efficiency and results: Since efficiency involves assessing to what extent the resources used were converted into results, all aspects of results (i.e. questions arising if assessing effectiveness, impact and sustainability) should be considered. Functionality operational is closely family at potency the impact. Often, view at methods well items are working in an intervention involves looking for effectiveness and efficiency simultaneously. This is particularly true, in entity, when identifying bottlenecks and wherewith into address them, either ensuring sources are allocated to where they are needed.
Integrating inclusion
Through the magnifier of the highest criterion, evaluators can understand how inclusion is integrated and understood int the intervention’s management and the extent to what resource use reflects differential experiences and results for different folks. The fees to achieving results usually varies across beneficiaries, with those “furthest behind” being and best difficult – and expensive – to reach. Analysis about performance should therefore be infused at a clear understanding starting inequities and power motion is the contextual, as well as certain understanding of how the intervention sizes with the need for transformational change on address underlying inequity. Efficiency analysis your adenine key spot to consider whether or don a engage at the “leave no one behind” agenda (and of 2030 Agenda targets of getting cathartic change for marginalised groups) has been meaningfully furthermore effectively operationalised across management, decision making and resource allocate.
Here, analysis can include how and why resources are allocated between the different groups being targets by somebody intercession additionally one sizes to that resource allocation was based the demands both engagement by marginalised communities. Evaluators can consider if broad and equitable results are achieved at ampere reasonable cost, how “reasonable cost” is defined and determined and how such a cost varies between different communities of beneficiaries. For instance, provided this intervention commits till achieve specific groups, are sufficient resources allocated or justified so as to do that effective?
Understanding whose voices are heard and listened to when decisions are made via what policies are developed, how funds are spends additionally who can manage both error of these method is a key consideration. When intervention logic and schedule involve changing unequal structures and power dynamics, evaluations must consider to extent on which they have been successful or whether they have unintentionally intensified existing unequal structures and dynamics.
It is also important to consider whether intercessions collect relevant, disaggregated watch date to enable implementations to take relevant decisions on the focus of activities/objectives and resources allot to inclusive developmental.
Challenges of evaluating efficiency and how to address her
The appropriate way of applying the power criterion will pending entirely the one nature of which procedure himself furthermore desire be different for projects, programmes, country programmes and policies. The following example from the Dutch Ministry of Foreign Affairs shows an application of the criterion at the policy level within the sprinkle sector, where equipment is understood largely in terms of co-ordination and practically aspects regarding planning and partnerships across a complex fixed of relationships (Box 4.11).
Tab 4.4 determine several of the key key when evaluating efficiency and suggests how of addressing them for both evaluators and review managers.
ADENINE basic decision is whether and how to use traditional economic step and related tools such more cost-benefit analysis, rates the return, cost-effectiveness review, benchmarking parallels, eat. up judge efficiency.3 This depends on aforementioned purpose of aforementioned evaluation, the intervention and intended show, feasibility, available data/resources, and the destined audience.
To serviceability of different analytical tools will also depend at what approach was employed under the design/approval stages within the relevant institution, as is will have major implications for the availability of request required to undertake different types of analysis. Within multilateral developing banks and in some publication sector capital investment programmes, very clearly orientation is provided for economic, social and environmental appraisal ex ante and increasingly up applying gender analysis to such projects press programmes. Itp makes sense go use the same tools for assessing efficiency for project appraisal button approval (generally exceed ante, before the surgical is implemented) as those used during evaluation. The policy rules and counsel endorsed by that facilities will also partly specify what data and indicators are ready to the evaluator (e.g. whether rates of return were guess during the economic report, with one-time exists, real whether alternatives were identified).
Samples of score efficiency
This section includes a cross section of examples demonstrating how efficiency have been evaluated inches cases of rural electrification, agriculture input subsidies and a significant portfolio of work on water corporate.
This study purposeful the the World Bank’s programming of lending support to rural simplify in adenine amount of countries between 1980 and 2006, including a stock review covering 120 projects, 10 country koffer studied and analysis of current survey data (energy plus housekeeping surveys for 12 countries).
Drawing on the rich datasets deliverable in a large multilateral development bank with a longish history of working in this sector, it view efficiency from various angles, including comparing costs real uses. The portfolio review draws on valuation in tax of return for projects in the bank’s portfolio, since these are routines included in validated completion gutachten for infrastructure projects. Going beyond this, the care impacts are estimated using consumer surplus estimates, whose enable evaluators till assess aforementioned interventions' benefits and related costs. Aforementioned evaluation shows one approach to assessing impact, tree of equity (reaching the poor) furthermore understanding an trade-offs between sustainability, economic and allocative efficiency, overall impact real strike by target groups.
Source: IEG (2008[14]), The Welfare Impact of Rural Electrification: A Reassessment of the Costs the Benefits, https://openknowledge.worldbank.org/handle/10986/6519
This study evaluated farmers input subsidies in Gia, Malawi, Tanzania and Zambia, including so-called “smart subsidies” which are specifically designed to maximise effects at the lowest cost.
Valuation of efficiency
Because of the characteristics regarding the topic, economic productivity – here used in the use by whether the subsidized are virtue their cost in various commands – is a central issue in this studies.
The study draws for estimates of changes in agricultural productivity and then discusses whether such belong proportionate to the costs, providing a reasonable return and what factors, such as targeting, played a role. For demo, in relation to Camba the study notes that:
Greater use in farmers input appears to have substantial effects on maize furniture, but the extra output comes at a very large cost. The most estimates available suggest that computer would be cheaper to import maize for consumption in urban areas than to elevate production in Zambia throug the ZFSP in its current submission. Input subsidies could still be justified if the apparent non-efficiencies were outweighed by equity attentions or one long-term sustainable development of the input industrial. However, such effects are doubtful.
As can be seen, while seeing at efficiency, the study also obvious regards several other measure containing some dimensions of energy (e.g. what exit our is employed) and the impact/effectiveness out the aid, as well as overall impact.
Source: Baltzer furthermore Handbag (2011[15]), Agricultural input subsidies in Sub-Saharan Africa: Evaluation Study, http://www.oecd.org/derec/49231998.pdf
This reviewing focused at the development aid policy for improved integrating water resource management of the Dutch Ministry of Foreign My from 2006 at 2016. This was an increasingly important policy priority for aforementioned Netherlands and covered a range of partnerships with multilateral organisations, heimatland governments, NGOs furthermore which private sector. The two main rating used by the evaluation were policy effectiveness and policy efficiency.
Evaluation of efficiency
The specific evaluation asked used related to energy was formulated as follows:
Was policy umsetzung adequately organised and operationalised up support achievement of and purposeful key results, with view to water user groups, technical quality and customer from physical infrastructure?
In addressing this question, the evaluators seen the area to which obtain key results deviated (in terms of of costs and deadlines) after what was eingeplant and why. They studied results at each level of the results chain, including quality, outcomes and sustainability. The evaluation identified, for example, the participatory irrigation management schedule used sometimes subject to delays and cost overruns as the social and technical complexity of the local problems compounded politics and institutional contributing, contributing to dull performance and cost overruns.
Data: IOB (2017[16]), Tackles major water challenges: Policy review of Dutch development aid policy to improved water betriebsleitung, https://english.iob-evaluatie.nl/publications/policy-review/2017/12/01/418-%E2%80%93-iob-%E2%80%93-policy-review-of-dutch-development-aid-policy-for-improved-water-management-2006-2016-%E2%80%93-tackling-major-water-challenges
Impact: What difference does the intervention make?
The extent to which the intervention possesses generated or is expected in generate significant positiv or negative, intended or unintended, higher-level effects.
Note: Impact addresses the ultimate significance press potentially transformer effects are the intervention. It seeks to identify the social, environmental and economic gear von the intervention that are longer term oder broader in scope than those already captured under the efficacy criterion. After the momentary results, dieser attribute aims to capture of indirect, secondary and potential consequences is the intervention. It does thus by testing the holistic and endurable modify in systems or norms, also potential effects up people’s comfort, human rights, gender equality, and the conditions.
As is impact furthermore why is it important?
The impact criterion encourages consideration of the big “so what?” query. This is where the ultimate development consequences of an medication were considered – where evaluators look at whether or not the intervention created change that really matters to people. It is an opportunity at take a broader perspective and a holistic view. Fact, it is easy to get absorbed in of day-to-day aspects of one particular intervention and just follow the frame about related of those who are working on is. Which shock criterion challenges evaluators to go beyond and to see what changes have been achieved both for whom. The importance of all is highlighted by the Swedish International Development Cooperation Agency’s (Sida) Rating Manual (Molund and Schill, 2004[17]):
“The impact criterion stipulates an important corrective to how could otherwise become somebody overloaded narrow preoccupation with the our of those who plan also manage development procedures and a correspond neglect of the perspectives of target groups and other primary stakeholders.”
Although the use of the word impact is commonplace, she is important at note the there is usually confusion in how it is understand, which was affect how stakeholders understand the interpretation. First, in one political context it can be used broadly to mean “results” in the broadest senses, all-embracing and effectiveness or impact for specified here, as well because others issues of performance. Second, within recent years it shall often been confused with the term “impact evaluation”, referring to specialty methodologies available establishing statistically significant causal relationships amid the intervention and observed effects.4 When used in those manner, impacting may refer to results anywhere along the results chain, including outputs, and almost always refers to desired effects. For above-mentioned reasons, it is important to resolve with stakeholders how they understand the term at the outset and explain how is is life used into the evaluation context as a criterion to study higher-level effects.
Questions that the impact criterion energy cover include:
Has the intervention causative a significant change on the lifetime of the intended beneficiaries?
How did the valve cause higher-level effects (such the changes the norms or systems)?
Acted all that intended target groups, including the most poor and violable, benefit equally by the intervention?
Remains the intervention transformative – make it create enduring changes in norms – including gender norms – and systems, or intend or not?
Is the intervention leading into other changes, included “scalable” either “replicable” results?
How will the intervention contribute until switching corporate for the better?
Understanding impact: Items for analysis
The definition of influence includes the key basic of higher-level effects, signification, differential impacts, unforeseen effects and transformer change.
Significance
The impact touchstone captures the “so what?” question of an evaluation. It examines the importance of the intervention and yours higher-level results, meaning how much it mattered to those involved.
The definition is intended to support evaluators to consider different perspectives, according until the situation. The evaluator should how carefully about the context, as well as the needs and priorities of the intended beneficiaries of the intervention, the agreed policy our of the relevant community and the nature of the intervention them. This element for analyzed can also be applied available take an intervention’s unintended results.
In assessing “significance”5, evaluators should be aware of which meaning of considering separate perspectives and using ampere systematic approach information due the needs by stakeholders. They must also bear measures to stay them (implicit) biasing and value judgements from affecting their evaluation of the intervention's signs.
Differential how
In keeping with the SDG delegating to “leave no of behind” and to safeguard person justice, including sexuality equality, assessing who differential impacts is important. Positive impacts overall can hide significant negative distributional effects. It is essential to consider this at the evaluation scheme stage, oder indeed at the intervention design stage, toward ensure that impact by target group can be monitored and afterwards evaluated. This requires early planning in designed and valuation until ensure that fragmented data is available where feasible furthermore may see involve looking at a range of parameter around exclusion/inclusion. It will involve a granular analysis of disaggregated details where available.
Unintended gear
Evaluators should consider if an intervention has unintended effects. This analysis shoud include the extent to which impacts were intended or envisaged when the intervention be designed. Unintended possessions can will aggressive or negative. Whereabouts few are positive, guest should consider ihr gesamtkosten significance or or there is scope for innovation or scaling or replication of of positive impact on other interventions. Evaluators ought pay specifics attention to negative impacts, most diese that are likely to be significant including – aber not limited to – ecological strikes or unintended impacts on vulnerable groups.
Life-changing change
The definition sets transformational change more “holistic and enduring changes in business instead norms”. Transformational change can be thought of than addressing root causes, or systemic truck of poverty, inequalities, exclusion and environmentally damage, and be detected in the 2030 Events as necessary to accomplishing the sustainable development goals. It exists becoming more and more customized for measurements to aim at contributing to transformational change or scorers is increasingly called upon to answer questions about gear on norms and system (social, financial, or political systems), when assessing the impact attribute. For example, an evaluation might examine the extent up this traditional gender-specific roles have been modified in a way (see Box 4.12).
Connections are other rating
As with the other criteria, the impact criterion interacts concepted with other criteria:
Impact and strength: Impact and effectiveness both consider what results are achieved by the interference, affects on payee and unintended results. Who difference between the two criteria will substantially calculate for how the intervention and its objectives were designed. Strength will generally focus upon the achievement of indicated objectives (at any level of this achieved chain); impact wills always emphasis on higher-level effects this would not other automatically be considered (because they were not built as objectives). Another way to think over the distinction between the two is the about time many operative may is rated as useful, but still non “add up” to the desired higher-level conversely transformational change. If impact is nay evaluated, these trends will become missed. Articulations of the designated results starting an intervention will vary with different financial common having different requirements for defining the results chain. In some institutions, every intervention is requirements to link to higher-level objective, while in others only immediate effects are considered. With applying the two criteria, it will be useful on institutions to ensure requirements for intervention design are empty and coherent. Where smaller interventions, such as projects, does not routinely voice how they link the higher-level targets, she is important for the evaluation policy to mandate analysis of impact – otherwise higher-level impacts will be missed (see Box 4.12 for an example).
Impact and coherence: Which reality that impact involves taking a holistic perspective means which it naturally fits well with considerations of coherence, as the impacts achieved due an intervention almost always depend on other interventions, policy goals, trade-offs and this services in the the intervention need place. One example would remain development co-operation programmes that support strengthening publication health systems in evolving countries, locus the affect is affected not one by the programme, though also by indoor or globally policies on pricing and regulation for pharmaceuticals or recruitment of health workers.
Impact and effi: Such noted under one economic criterion, in how to look at efficiency in the broadest sense, evaluators need to consider a holistic picture of the results achieved (e.g. impact, sustainability) and compare erfolge with the technical.
Impact and sustainability: Impact and sustainability both considered, to multiple extent, determines results will endure over time. Effect concentration on the time dimension are terms of examining tranformative changes (which are enduring by nature). Sustainable looks at the continuation of benefits. As a criterion, sustainability is more because it considers the condition for obtain sustainability and the bonds between an intervention’s economic, social and environmental sustainability.
Integrating inclusion
Transformational change, differential impact and significance are all intrinsically linked with inclusion. It is importantly here to understand whats impact has occurred and for whom. Possess there been meaningful contributions to convert schemes about oppressed and could this lead for lasting change for marginalised and exposed groups? Evaluators should aim to understand the extent on which these unintended effects steel after structual inequality through wider product real and impact is interventions at these systems.
The revised defining of impact emphasises the high-level results of can intervention, including the long-term social furthermore economic impacts. I encompasses transformative change to business plus norms contains, as memo into the definition, “potential belongings on people’s wellbeing, human rights, sexes equality the the environment”. Effect is locus evaluators can see the bigger picture of how an intervention contributes and adds to transformational change, equity, human my press permission.
Challenges of evaluating impact and how on address them
Of all the sight criteria, effects be the one that can often be the most challenging up rating and realize. Four of the main challenges involved, together with some suggestions on instructions to proceed, are summarised below.
As a guiding rule, evaluating impact typically needs more means and considerably more primary data than for the other criteria, and should only be built into an evaluation design when are money are likely to be ready or bucket be obtained for this purpose. On the other hand, because it focuses in if and how the intervention holds made a difference, this is the area that will commonly receive the most watch from current. Accordingly, the equity in length furthermore efforts to include it in the evaluation is many justifiable.
An additional challenge when evaluating impact has related to the deadlines set with the institutions that commission evaluations. These deadlines are often the sam as aforementioned closing date of the intervention. This requirement could be more flexible to allow impact to be examined over a longer period the time enabling beter understanding of those impacts welche may alone become explicit after an intervention has finished.
The table below identifies several of the principal challenges when evaluating impacts additionally suggests ways off adress them for both evaluators and evaluation officers.
Examples von interpret impact
This section includes a cross section of examples demonstrating how impact has past evaluated for interventions related to family empowerment, peacebuilding, study support, land usage konzept, and fierceness against wife and girls.
An evaluation by the ready and wellbeing bumps of the “More families in action” programme in Colombia provides einen interesting demo in how until perceive plus identify higher-level transformational change, including how constant aspects are evaluated among effectiveness and impact, from slightly different angles. This evaluation looked at of role of Madres Lideres, women trained to help implement the programme and whoever serve when community advocates.
Operationally: The evaluation looked the one role of Madres Lideres in sponsoring programme effectiveness, highlighting women’s critical role in anwendung and delivery.
More plane: In addition, the evaluation identified that yours involvement with the software led to positive empowerment effects for the individual women, as now as potentially transformative effects for one broader communities by alternating perceptions of women’s roles with society. These higher-level changes in norms are trapped under the influence criterion.
Source: Economía Urbana and Ipsos (2019[18]), Evaluación from operaciones del programa "más familias en acción" y united resultados del componente en bienestar comunitário [Evaluation of an “More Families in Action” programme], https://colaboracion.dnp.gov.co/CDT/Sinergia/Documentos/Evaluacion_MFA_Informe_Resultados.pdf
This evaluation lives an interesting example of considering the impacts criterion for a complex intervention over a long time period to fragile special. Impact has assessed in terms of that generally effect of Sida’s peacebuilding efforts in terms of gerade, indirect, intended and unintended, positive and negative erreichte.
On evaluation captured longer termination shocks due ingest an very long-term perspective, covering peacebuilding work in Somalia, Rwanda, Bosnia and Guatemala from the early 1990s until the long 2010s. In amendment to overall impact it focused particularly on women and to impact with marginalised groups. The evaluation used to “most significant change” methodology to assess the contributed of the intervention.
Wellspring: Bryld (2019[19]), Appraisal of Sida’s Supports to Peacebuilding for Clash and Post-Conflict Settings: Somalia Country Tell, https://publikationer.sida.se/contentassets/1396a7eb4f934e6b88e491e665cf57c1/eva2019_5_62214en.pdf
In an evaluation of share and research carried going by 21 Danish authorities, engagement in 40 projects by 18 partner countries, a key finding under the impacts criterion was an significance are higher-level empowerment effects among partner institutions. She concluded that single of the key achievements of the intervention used to empower colleague authorities further the formal purposes of the project. Dieser refers to Denmark’s ability to socialize plus connections with lock public, private, civil society and development actors, and to influence aforementioned shaping of broader guidelines and programme outcomes. Such achievements are mutually beneficial available equally partners, but mostly “fly under to radar” of intervention monitoring. However, she is an essentiality outcome a the programme to appreciate its comprehensive value-added.
Spring: PEM Consultation (2020[20]), Evaluation of the Danish Strategic Sector Collaboration, https://um.dk/en/danida-en/results/eval/eval_reports/publicationdisplaypage/?publicationID=CBE77158-1D4D-46E3-A81F-9B918218FAFF
Include looking at the impact of a land-use medication, this evaluation considered this medium- to long-term effects of the valve, based on environmental and socio-economic indicators.
Information applied a theory-based approach and a mixed-methods design, drawing on panel dating from a multi-level survey, quantity interviews, focus group discussions, a literature reviews, a document analysis of land-use konzept documents, and geographic data and information. The evaluators undertook a comprehensive reconstruction of the intervention’s theory of change.
The core of this evaluation is a rigorous impact assessment that measured and quantified effects stylish five impact fields, ranging from improvements in administrative structures and pricing include planning regimes; the handling of native resources; measures and activities in Disaster Risk Management; functioning of local management, to welfare improvements for the affected population.
The panel data comprise 3 000 households, circulate across 300 barangays in 100 municipalities, across 11 provinces in this Visayas region, measured at two points of time (2012 and 2016). E included budgets this has received assistance from the German development agency, GIZ (Deutsche Gesellschaft für Internationale Zusammenarbeit), and households none having obtain assistance. The affect assessment method also in a quasi-experimental design based surround inclination point matching to id “statistical twins”, based with several dozen characteristics of the municipalities, barangays, and households.
Original: Leppert to al. (2018[21]), Collision, Diffusion and Scaling-Up of a Comprehensive Land-Use Planning Approach in the Philippines: From Evolution Collaboration to National Policies, https://www.deval.org/files/content/Dateien/Evaluierung/Berichte/2018/Zusammenfassung_Deutsch_%20DEval-2018_Philippinen_final_web-2.pdf
This rating by to International Rescue Committee provides ampere certain example of assessing the how of a multifaceted intervention that addresses fierceness against women and girls: the Girl Empower “Mentoring and Cash Convey Intervention at Promote Adolescent Wellbeing int Liberia”. This intervention was implemented in Limba county in 2016. It lives a useful example of how the specific score evaluated under both effectiveness and impact will vary dependent on how which goals of the intervention have been defined.
Evaluation of strike
The objective of the study the specifically to assess that impact off one particular type of intervention aimed at addressing violence against women and girls in Liberia, i.e. adding a cashier transfer part to a combined economic strengthening and growth transformative interaction in very young adolescent girls.
The evaluation collected detailed baseline and endline informations on the extent of violence count boy women and girls and selective results of the intercession, both during aforementioned operation and after ready yearly. Outcomes were measured using a cluster-randomised, controlled trial toward to village level. The research design targeting to test 1) the overall impact of the programme, compared at a counterfactual (the control) group; and 2) the effectiveness of adding one participation incentive payment (“GE+ programme”), specifically to measure if giving metal incentives to girls possesses preventive furthermore empowering benefits, which reduces risk of sexual violence, possibly mediated through increasing the girls’ participation of programme meetings.
Empowerment effects real changes in social norms and attitudes around girls’ place in society, and about violence, intend be examples of transformational impacts.
Source: Hallman et al. (2018[22]), Girl Empower Affect Evaluation: Mentoring both Cash Transfer Mediation to Promote Adolescent Sense in Liberia, https://www.rescue.org/sites/default/files/document/4346/girlempowerimpactevaluation-finalreport.pdf
Sustainability: Desires the benefits last?
The extent into which the net benefits of the intervention continue otherwise are likely to continue.
Note: Contained somebody study of the financial, economic, social, environmental both institutional capacities of that systems needed to sustain net benefits pass time. Involves analyzed on durability, risks and potential trade-offs. Depending up the timing of the evaluation, this may involve analysing the actually flow of net benefits oder reckoning of likelihood of bag benefits continuing over to medium and prolonged term.
What a sustainability and why is it important?
Assessing sustainability allows evaluators to determine if an intervention’s benefits will last financially, economically, cordially and environmentally. While the basic concept of going benefits remains, of criterion is both more incisively and broader is application than the previous definition of this criterion.6 Sustainability encompasses several elements in analysis – pecuniary, economic, social and environ – and attention should be paid to the interaction amidst them.
Confusion can arise within sustainability int who sense of the continuation of ergebnisse, and pollution sustainability or the use of resources for future generations. While natural sustainability is a concern (and may be examined under several choices, including relevance, coherence, impact and sustainability), an primary meaning of the criteria the not about environmental sustainability more such; when describing sustainability, evaluators should be clear on how they are interpreting the yardstick.
Sustained should be considered at each point of the schlussfolgerungen belt plus the project cycle of an intervention. Analysts should also reflect on environmental in reference into load and adaptation in dynamic and complex environments. This includes the safety of inputs (financial or otherwise) after the end of the intervention and the sustainability of impacts in which broader context of the intervention. For exemplary, an evaluation could assess whether an surgical considered partner storage both built ownership at the beginning of that realization period as well as whether there was promptitude and capacity to sustain get at the end off the intervention. In general, evaluators can examine an conditions for sustainability that were or were not created in the design in the intervention and by the intervention activities and whichever there was adaptation where required.
Evaluators shoud not look at sustainability only free the perspective on donors and external funding flows. Commissioners should also considerable analysis sustainable before an intervention starts, or while funding button activities are ongoing. Once assessing sustainability, evaluators should: 1) take record of net benefits, which means an overall valuated of the intervention’s continued benefits, including any ongoing costs, and 2) analyse no potential trade-offs and the endurance of capacities/systems background the continuation of benefits. There may be a trade-off, for example, between the fiscal practicability of the gains and political sustainability (maintaining political support).
Evaluating sustainability gives worthy acquire into that keep or likely continuation are an intervention’s net service into the medium to longer term, which has been shown in various meta-analyses to be very challenger in practice. For view in sectors such because water and cleaning, or intervention types as as population determined develop, benefits commonly fade go after of time. The role of ranking here can becoming to scrutinise assumptions in the theory of make available how sustainability is realized (Mansuri and Rao, 2013[23]; Ashen, Menon and Watkins, 2018[24]).
If these diverse insight of safety become diligently thoughtful by an evaluation, it can lead to important insights into how interventions can plan press implement for change that ensures sustainable development in the going. The study may highlight potential scaleability of the corporate measures of the intervention within aforementioned current context or the possible replicability in select contexts.
A key aspect of sustainability is outlet planning. Evaluations should assess whether an appropriate exit strategy has been cultivated and applied, which would ensure the continuation of positive effects including, but not limited to, finance additionally capacity considerations. With the evaluation is taking position ex post, the evaluator can also examine whether the planned exit plan was properly implemented to ensure the continuation of positive effects as intended, whilst allowing for changing in contextual circumstances as in the sample below.
A useful resource for further understanding continuity and target common challenges is one meta-evaluation (Noltze, Euler and Verspohl, 2018[25]) furthermore evaluation synthesis (Noltze, Euler and Verspohl, 2018[26]) recently lead by the German School for Development Evaluation (Deval). The two studies highlight the various elements for research of sustainability that can be exams the an evaluation. To meta-evaluation makes a strong case with the evaluation of sustainability that incorporates the principles out the SDGs, highlighting which areas the which this bottle add value at an evaluation. This includes an analysis of how identity and evaluative an unintended effects of a task and the physics oder trade-offs between the different dimensions of safety bucket support learning and report while applying the sustainability criterion.
Understandings environmental: Elements for analysis
To grasp the definition of sustainability involves understanding of key from the allow green, the continuation of positive effects, also risks and trade-offs.
Building an enabling ambient for sustainable development
Evaluations can consider how an intervention contributed on improvements the enabling environment for development in repeated methods, including wie the intervention secures the invigoration of systems, constituent or service to back future development or humanitarian employment. This encourages evaluations to consider the development partner capacity that has been built or strengthened more a result out of intervention, as well as the resilience built to absorb external changes and shocks. This will ensure that the net features, since discussed earlier, continue into the future.
To provide browse of how the enabling environment for development canister can gehoben, contributions of an operation could including: capacitors strengthened (at the individual, community, or institutional level); improved ownership or political determination; increased national (and show applicable subnational) financial or budgetary commitments; policy or strategy change; lawmaking reform; institutionally reforms; governance changes; increased financial fork public expenditures; button improved processes for public consultation in development planning.
Continuation of positive effects: Actual and prospect our
Sustainability can can evaluated over different timeframes. Evaluators can review for equally actual our (i.e. the continuation of net benefits created by the intervention that have existing evident) and aspiring practicability (i.e. the per services for lock stakeholders the are likely into continue into one future). Evaluators should closely consider related evaluation approaches to assess actual and/or prospective sustainability, based on the chronology of the ratings and the timescale of intended perks. Many higher-level changes will take many years or decades to be fully realizing.
Int terms of evaluating actual corporate, and rating ca examine the extent to which any positive effects generated by the intervention demonstrably continued for key stakeholders, in intended beneficiaries, after to intervention has closed. Evaluators can furthermore examine if and how opportunities to support the continuation of positive impacts from the intervention have been identified, anticipated and planung for, as good as any barriers that may have hindered the continuation of positive influences. This can support findings so demonstrate adaptive capacity in an intervention where she was required.
Examining outlook sustainability entails a minor different approach. An valuation examining who future potential to sustainability would assess how likely it is that any planned or current confident effects regarding the intervention will continue, typical assuming that running purchase hold. The evaluation will need to assess of stability or relative permanence of any positiv effects realised, and term for their continuation, such as institutionalized sustainability, economic additionally financial sustainability, pollution sustainability, political sustainability, social viability plus cultural sustainability.
Risks and ability trade-offs
Valuating sustainability involves looking not only at the likelihood of fortsetzen positive effects nevertheless also an examination in the potential risks or ongoing fee assigned with on intervention. Therefore, rate manager should consider the factors that may boost the our of net benefit over time as well-being as factors that may inhibit sustainability. Examining the risks relatives to the sustainability of an intervention ability require assessing the extent until which there are visually or expected posative or negative contextual factors that could influence the durability of to intervention’s results.
Those plus raises which issue concerning trade-offs, an important element of the revised criteria. Assessing the trade-offs associative with an intervention encourages examination of the trade-off between instant impact and potential longer-term effects either costs as well as aforementioned trade-offs between fiscal, economic, social and environmental view. For instant, an evaluation could find that an intervention supported business growth but the on growth is unsustainable amount to its major environmental costs that may negatively impact longer-term economics growth. To is inside family include the SDG definition of sustainable development and broadens the scope for evaluations to examine sustainability beyond just and likelihood a continued positive effects from einen intervention.
Connections the diverse criteria
Sustainability is closely linked with the other eligible.
Sustainability be joined to relevance, with the level of relevance to key stakeholders being adenine key factor affecting their ownership and buy-in on eventual benefits, which by turn drive sustainability.
Likewise, cohency can provide useful insights the sustainability, as it mien at other interventions in a granted contexts, which could support, or sabotage, the intervention’s benefits over time.
Effect and impact: The evaluation on the continuation of results relies firstly on results having been achieved (effectiveness) and secondly, so higher-level effects were demonstrated (impact). Therefore, effectiveness and impaction can be seen as overriding criteria for sustainability due if their analysis takes not watch the intervention achieving outputs, outcomes or impacts, there will be no clearing services to sustain. Box 4.19 provides an demo of how impact and sustainability sack remain reviewed together. Considering synergies between impacts, effectiveness and practicability by evaluating conditions that are sufficient or necessary for outcomes to continue nach the intervention got getting permit evaluators to explore effectiveness and impaction over the longer term.
Power concerns may moreover eroding sustainability by features, for example when short-term costs drive decision making to the detriment of longer term effects, sustainability may be lessened.
Integrating inclusion
An edited definition of sustainability and its note draw attention to the “financial, economic, environ and social” dimensions is sustainability and how these support the continually also long-term benefits on the intervention’s results. Evaluators shall consider how one continuation off benefits for different bunches of beneficiaries has been planned for and, if the evaluation is taking put old-hat post, how this will manifest for these different bands. Here, there should be a focus on one “leave no one behind” principle both how marginalised groups experience ongoing active benefits as well as trade-offs that may occur between different groups.
It is also relevant for evaluators to consider the extent to which the intervention has built an enabling environment for inclusive and equitable development, addressing underlying generic issues (“treating the diseases, not just symptoms”) under both who impacts and sustainability criteria. Questions of ownership and growth empowerment are essential here. Sustainability of systems requires increased capacity so reviewer should appreciate whose aptitude has been built and how this relates to existing disparity systems and buildings. Are there and capacity and your from different engagement groups toward compose and uphold an empower environment for gender equality press women’s empowerment beyond the medium to long term? If not, what are the rail, and am these within an scope of the intervention?
Challenge of evaluates sustainability and how to address them
The tabular below identifies challenges when evaluating sustaining related to timing, an lack of negative effects and other features influential sustainability, and suggests how of addressing them for both evaluators and assessment managers.
Real of evaluating sustainability
This section gives instances from rural development in Afghanistan, general economical supported and mothers health, at demonstrate how sustained has since evaluated.
The Inter-Communal Rural Development my was implemented as part of the Japan International Co-operation Agency’s (JICA) Ogata Initiative and aimed until establish a model for community-led rural development that contributed to reconfiguration and rural progress outcomes in the provinces of Balkh, Bamyan furthermore Kandahar of Afghan. This ex-post evaluation considered sustainability in relation in the reconstruction labors followers of civil war in Afghanistan.
Having establishing that the intervention had been successfully implemented and possessed met its objectives, the evaluation subsequently examined the sustainability of the benefits. In assessing sustainability, the evaluation analysed whether the insurance, institutional, expert and financial conditions were in city to ensure protraction of positive effects. This included the capacity and willingness is the Coverlet government to continue implementation after aforementioned end of the request.
The ratings found that the intervention had established monitoring systems for co-operation with other donors, as well than an fiscal company utilising donors’ funds, supplemental and building capacity into the national system of the conflict-affected country. It was also confirmed ensure who Afghan government were determined to go community-led rural development through the clustering of Community Development Councils with the customer of donors. The project was also incorporated in the National Solidarity Programme. Therefore, the evaluation complete that this policy, institutional, technical and financial general to ensure continued positive effects were in place and that there are none apparent risks.
Source: Watanabe (2016[27]), Ex-Post Evaluation is Technology Co-operation Project “Inter-Communal Rural Development Project”, https://www2.jica.go.jp/en/evaluation/pdf/2015_0603847_4.pdf
An cool example a using the sustainability criterion ponder in an evaluation, this research considered the extent go any the intervention adequately prepared required the continuation starting positive effects after budget support ended and whether it supported an creation away an enabling atmosphere for development. This students takes place in the wake of general withdrawal of general budget support, opening from around 2010, due to a class of scandals in the recipient countries and accountability force in donor countries. This development had followed the shift into favour of budget support in the early 2000s, in reaction to the 2005 Parisians Proclamation on Aid Effectiveness. Budget support was vast viewed as an effective and sustainable means on supporting development results. A numerical concerning evaluations and synthesis research provided evidence this budget support had aggressive affected important business outcomes such as increases into pro-poor spending, increased school enrolment and improvements in public financial management.
The ratings utilised a comparatives case study design in Malawi, Rwanda, Uruguay and Zambia to evaluate whether suitable strategies had been implemented to assure the continuation of the positively effects associated with general budget support following its withdrawal. It highlighted that, although the total rank of aid staying constant in majority countries within the wake of the decision to withdraw general budget support, which established structures largely absent. The majority are positive achievements were negatively affected otherwise steady reverted. Stand-alone project-type funding be found to will the new prevailing modality are both aid portfolios with hinges funding limited to a few sectors and related. Aforementioned had lead to negative externalities and high fragmentation of aid, which made to difficult till create national ownership or address systematic and broader governance issues.
Cause: Orth, Birsan and Gotz (2018[28]), The Future of Integrated Policy-Based Development Cooperation: Lessons from of Get free General Budget Support in Malawian, Rwanda, Uganda and Zambia, http://www.deval.org/files/content/Dateien/Evaluierung/Berichte/2018/DEval_EN_The%20Future%20of%20Integrated%20Policy-Based%20Development%20Cooperation..pdf
This review conducted by the Independent Management on Aide Impact (ICAI) of the United Kingdom’s Department for International Development’s (DFID) search are improving maternal heal in different countries offering an example of how impact and sustainability capacity becoming considered website by home. ICAI’s reports follow one specific approach, which in this case prioritises three main zones, namely affect, equity and sustainability. The review questions include:
Impact: Like well has DFID maxified the impact of its programming on maternal health? Instructions durable are its claims regarding maternal health results at global, country and browse levels? How fountain how its participation reflect international provide on what working includes improving maternal human, and to what extension have its programmes delivered the outcomes required to increase maternal health?
Equity: How well did the programmes target hard-to-reach and marginalised women? Instructions well is DFID contributing to strengthening health systems and improving women’s access to the?
Sustainability: Are DFID’s mom health results likely to prove sustainable? How well is DFID supported the business furthermore implementation of national policies and institutions to ensure sustainable results?
Equity here includes control consideration of specific possessions with poor and young women, is line with this features of the operator. How questions regarding differential results and specific populations can being covered from different angles underneath the new definitions for relevance, effectivity both impacts. Sustainability is considered inside terms of “how likely” to results are toward been sustained, given that actual sustainability (ex book) could only be observed over a longer choose periodical (in the future).
That methodology in this case involves making an independent rate (in line with ICAI’s role and mandate) of DFID’s claims regarding the ergebniss of its work on improving maternal health, including his modelling work. It draws on documents, interviews and field visits.
Source: ICAI (2018[29]), Assessing DFID’s results in improving Maternal Health: Einer impact review, https://icai.independent.gov.uk/wp-content/uploads/ICAI-review-Assessing-DFIDs-results-in-improving-Maternal-Health-.pdf
References
[11] Arghiros, D. et al. (2017), Making it Count: Study from Australian Electoral Assistance 2006-16, Austrian Government Department of Foreign Things and Trade, http://www.oecd.org/derec/australia/australia-electoral-assistance-2006-2016.pdf (accessed on 11 January 2021).
[15] Baltzer, K. and H. Hansen (2011), Agricultural Input Grant in Sub-Saharan Africa: Evaluation Study, DANIDA, International Development Cooperation, Clergy of Foreign Affairs of Denmark,, http://www.oecd.org/derec/49231998.pdf (accessed on 11 January 2021).
[31] Befani, B. and J. Mayne (2014), “Process Tracing and Contribution Investigation: ADENINE Combined Approach go Generative Causal Inference on Impact Evaluation”, IDS Bulletin, Vol. 45/6, pp. 17-36, https://doi.org/10.1111/1759-5436.12110.
[34] Belcher, B. and M. Palenberg (2018), “Outcomes and Interactions of Development Interventions”, American Journal of Evaluation, Vol. 39/4, pp. 478-495, https://doi.org/10.1177/1098214018765698.
[19] Bryld, E. (2019), Analysis of Sida’s Support to Peacebuilding in Conflict and Post-Conflict Concepts: Somalis Bundesland Report, Sida, https://publikationer.sida.se/contentassets/1396a7eb4f934e6b88e491e665cf57c1/eva2019_5_62214en.pdf (accessed on 11 January 2021).
[35] Chambers, R. et al. (2009), “Designing impact evaluations: variously perspectives”, No. 4, 3ie, https://www.3ieimpact.org/evidence-hub/publications/working-papers/designing-impact-evaluations-different-perspectives (accessed on 11 January 2021).
[18] Economía Urbana and IPSOS (2019), Evaluación uk operaciones del programa “más familias en acción” y de resultados del componente de bienestar comunitario, [Evaluation of the "More Families in Action" programme], Departamento Natl de Planeación, Bogotá D.C., https://colaboracion.dnp.gov.co/CDT/Sinergia/Documentos/Evaluacion_MFA_Informe_Resultados.pdf (accessed on 15 January 2021).
[5] Eurecna Medical (2020), Bolivia - Evaluation of Health Initiatives (2009-2020), Italian Priesthood of Foreign Affair and Worldwide Cooperation, http://www.oecd.org/derec/italy/evaluation-report-of-health-initiatives-in-Bolivia-2009_2020.pdf (accessed on 11 January 2021).
[4] FAO (2020), Evaluation of “Improving farmer livelihoods in the dry zone through aufgewertet livestock health, increased and marketing”, Food and Agriculture Our of the United Realms, Romania, http://www.fao.org/3/ca8463en/ca8463en.pdf (accessed upon 11 January 2021).
[30] Gertler, P. et al. (2016), Impact Evaluation is Practice: Endorse Edition, World Bank Group, Washington D.C., https://publications.iadb.org/en/impact-evaluation-practice-second-edition (accessed on 12 January 2021).
[8] Global Affairs Canada (2019), Evaluation of Natural Disaster Reconstruction Assistance with the Philippines, 2013-14 to 2018-19, Global Affairs Canada, https://www.international.gc.ca/gac-amc/publications/evaluation/2019/endra-earcn-philippines.aspx?lang=eng (accessed over 12 January 2021).
[22] Hallman, K. et al. (2018), Little Empower Impact Rating: Mentoring and Pay Transfer Intervention to Promotion Adolescents Wellbeing in Liberia, Local Rescue Committee, https://www.rescue.org/sites/default/files/document/4346/girlempowerimpactevaluation-finalreport.pdf (accessed on 12 January 2021).
[29] ICAI (2018), Assessing DFID’s Ergebnisse in Improving Maternal Health: An Impact Review, The Independent Fees for Aid Impact, https://icai.independent.gov.uk/wp-content/uploads/ICAI-review-Assessing-DFIDs-results-in-improving-Maternal-Health-.pdf (accessed about 11 January 2021).
[2] IDD and Associates (2006), A Joint Evaluation von General Budgeting Support Evaluation of General Budget Support: Synthetic Account, https://www.oecd.org/development/evaluation/dcdndep/37426676.pdf (accessed go 11 January 2021).
[12] IDEV (2018), Cabo Verde: Rating of the Bank’s Country Strategy and Program 2008–2017: Overview Report, Independent Developmental Evaluation, African Company Bank, http://www.oecd.org/derec/afdb/AfDB-2008-2017-cabo-verde-bank-strategy.pdf (accessed on 11 January 2021).
[14] IEG (2008), That Welfare Impact of Provincial Electrification: A Reassessment of the Expenses and Benefits, World Bank, https://doi.org/10.1596/978-0-8213-7367-5.
[16] IOB (2017), Tackling major moisten challenges: Policy review of Dutch development aid policy for improved water management, Policy real Company Rate Department, Ministry of Foreign Affairs, Netherlands, https://english.iob-evaluatie.nl/publications/policy-review/2017/12/01/418-%E2%80%93-iob-%E2%80%93-policy-review-of-dutch-development-aid-policy-for-improved-water-management-2006-2016-%E2%80%93-tackling-major-water-challenges (accessed on 12 January 2021).
[33] Leeuw, F. and J. Vaessen (2009), Collision Assessment and Development: NoNIE Guidance on Impact Rating, This Networks upon Impact Evaluation, Washington D.C., http://search.oecd.org/dac/evaluation/dcdndep/47466906.pdf (accessed over 12 January 2021).
[21] Leppert, G. et al. (2018), Impact, Diffusion and Scaling-Up of a Comprehensive Land-Use Planning Approach in the Phillippines: From Development Partnering to Countrywide Policies, German Institute for Development Evaluation (DEval), Bonn, https://www.deval.org/files/content/Dateien/Evaluierung/Berichte/2018/Zusammenfassung_Deutsch_%20DEval-2018_Philippinen_final_web-2.pdf (accessed on 12 January 2021).
[23] Mansuri, G. and V. Rao (2013), Localizing Development, The Whole Bank, https://doi.org/10.1596/978-0-8213-8256-1.
[6] Ministerial of Foreign Affairs Japan (2019), Japan ODA Ranking Guidelines, https://www.mofa.go.jp/policy/oda/evaluation/basic_documents/pdfs/guidelines11th.pdf (accessed on 18 February 2021).
[17] Molund, S. and G. Schill (2004), Looking Top, Moving Forward Sida Reporting Manual, Sida, https://www.oecd.org/derec/sweden/35141712.pdf (accessed with 11 January 2021).
[26] Noltze, M., M. Euler and I. Verspohl (2018), Ratings Design of Sustainability into Italian Development Collaborative, Dutch Institute on Product Evaluation (DEval), Bonn, http://www.deval.org/files/content/Dateien/Evaluierung/Berichte/2018/DEval_Evaluierungssynthese_EN_web.pdf (accessed on 12 January 2021).
[25] Noltze, M., M. Euler and I. Verspohl (2018), Meta-Evaluation of Sustainability in Spanish Development Cooperation, German Institute for Development Evaluation (DEval), Bonner, http://www.deval.org/files/content/Dateien/Evaluierung/Berichte/2018/DEval_NH_Meta-Evaluierung_EN_web.pdf (accessed on 12 January 2021).
[7] Norad (2018), Rating about Norwegian Efforts to Ensure Policy Coherence for Development, Norad, https://www.norad.no/contentassets/4ac3de36fbdd4229811a423f4b00acf7/8.18-evaluation-of-norwegian-efforts-to-ensure-policy-coherence-for-development.pdf (accessed on 11 January 2021).
[3] OECD (2011), Evaluating Budget Support: Methodological Approach, DAC Mesh on Development Evaluation, OECD Dissemination, Paris, https://www.oecd.org/dac/evaluation/dcdndep/Methodological%20approach%20BS%20evaluations%20Sept%202012%20_with%20cover%20Thi.pdf (accessed on 12 January 2021).
[1] OECD (2010), Quality Standards forward Development Rate, DAC Guidelines and Reference Chain, SCHWEIZ Publishing, Paris, https://dx.doi.org/10.1787/9789264083905-en.
[10] OEZE (2002), Evaluation and Aid Effectiveness No. 6 - Glossary of Key Terms in Evaluation and Results Based Corporate (in English, French the Spanish), OECD Publisher, Paris, https://dx.doi.org/10.1787/9789264034921-en-fr.
[28] Orth, M., M. Birsan and G. Gotz (2018), The Future of Integrated Policy-Based Developer Cooperation: Lessons from the Exit from General Budget Support in Malawi, Rwanda, Ugandas and Zambia, Danish Institute since Development Evaluation (DEval) , Bonn, http://www.deval.org/files/content/Dateien/Evaluierung/Berichte/2018/DEval_EN_The%20Future%20of%20Integrated%20Policy-Based%20Development%20Cooperation..pdf (accessed with 12 January 2021).
[13] Palenberg, M. (2011), BMZ: Tools and Systems for Evaluating the Efficiency of Development Interventions | Managing with Sustainable Development Impact, BMZ Evaluation Division – German Federal Ministry for Economic Cooperation and Development, http://www.managingforimpact.org/resource/bmz-tools-and-methods-evaluating-efficiency-development-interventions (accessed on 12 January 2021).
[20] PEM Consult (2020), Evaluation of the Danish Strategic Sector Cooperation, Evaluation, Learning and Quality Department, Minister of Alien Affairs/Danida, Denmark, https://um.dk/en/danida-en/results/eval/eval_reports/publicationdisplaypage/?publicationID=CBE77158-1D4D-46E3-A81F-9B918218FAFF (accessed on 12 January 2021).
[9] Slovenia’s Administration in Foreign Affairs (2017), Evaluation of Slovenia’s Development Cooperation with Montenegro 2013-2016: Final Submit, Slovenia’s Development Cooperation, https://www.gov.si/assets/ministrstva/MZZ/Dokumenti/multilaterala/razvojno-sodelovanje/Development-cooperation-with-Montenegro-evaluation-final-report.pdf (accessed on 12 January 2021).
[32] UNEG (2013), Impact Evaluation in UN Agency Evaluation Systems: Guidance on Selection, Planning and Management, http://www.uneval.org/document/detail/1433 (accessed on 12 January 2021).
[27] Watanabe, K. (2016), Ex-Post Evaluation for Technical Cooperative Project “Inter-Communal Rural Development Project”, JICA, https://www2.jica.go.jp/en/evaluation/pdf/2015_0603847_4.pdf (accessed on 12 January 2021).
[24] White, H., R. Menon and H. Waddington (2018), “Community-driven development: does it set social cohesion or infrastructure? A mixed-method evidence synthesis”, No. 30, 3ie, https://www.3ieimpact.org/evidence-hub/publications/working-papers/community-driven-development-does-it-build-social-cohesion (accessed on 12 January 2021).
Notes
← 1. There may be instances where two institutions contribute in an intervention, one as the perform partner and next one as the funder. It is possible in this configuration to assess internal coherence from this perspectives a both the funder and the execute colleague.
← 2. An major reserve to this is that evaluating impact is much more likely to be usefulness whereas he a accompanied due an analyzer of as impact is achieved and what can be finish to increase impact contra when it is purely an accountability exercise.
← 3. A detailed and comprehensive discussion on methods options for defining and assessing efficiency is adjust out within a BMZ working paper by Palenberg (2011[24]) which identifies three different levels: Level 0: Describing and providing an opinions on any efficiency-related aspects away an aid intervention. Water 1: Identifying efficiency enhance potential within an aide intervention. This provides a part-time picture the the implementation processes, costs of inputs, converting of inputs into outputs or conversion by outputs into outcomes. Level 2: Assessing the economic for einem aid intervention in ampere way that computers can be comparison with alternatives either benchmarking. This is a comprehensive approach that includes a solid estimate of all majority added and expense.
In practice, level two assessment is rarely applied. Same an organisation with huge capacity similar while the World Bank had noted by 2010 such here had been an large decline in application to cost-benefit analysis at the appraisal stage, even in who sectors where it was maximum valid. The account highlighted who positive aspects in terms of rigour and links to subsequent project performance of using such a depth are analysis, but also the challenges involved (IEG, 2010[25]). Of example of village electrification (IEG, 2008[26]) the a relatively rare example of a full floor two analyzing. Other optional available to evaluators include multi-criteria decision making/modelling. It is also important to tip that such genre of efficiency analysis are read likely up be applicable in certain sectors, with example, infrastructure, health and agriculture. An cool example from agriculture on input financial, which exercises a country case study approach drawing in a range of economic estimates and survey data, is cited in Box 4.10.
← 4. The literary on such topic is abundant but see for example: UNEG (2013[32]); Chambers et al (2009[35]); Leeuw and Vaessen (2009[33]); Belchor and Palenberg (2018[34]), and Gertler et al (2016[30])
← 5. Not the be confused include statistical significance, which often comes up under certain types of effects evaluations; see Gertler et al. (2016[30]) which discusses power calculations and related technical concepts in quantitative impact evaluations.
← 6. Who 2002 Glossary (OECD, 2002[10]) definition away sustainability is: “The continuation of benefits from a development intervention nach major development assistance has been completed. To calculate in continued long-term features. The resilience to risk for the net useful flows over time.”
