November 2007

News from the GEF Evaluation Office

The November Council Session

The GEF Evaluation Office is presenting the Joint Evaluation of the Small Grants Programme and the GEF Annual Report on Impact 2007 to the November Council Session. The Council will also review Terms of Reference for the Mid-term Review of the RAF and the Progress Report from the Evaluation Director. All relevant documents, including management responses and case studies, are available on the GEF website: www.thegef.org.

GEF Annual Report on Impact 2007

The GEF portfolio has matured to a stage where one can access the long-term results of its interventions. In this context, the evaluation office has this year launched its Annual Report on Impact to be presented to the Council in November 2007. The report focuses on protected areas, which have been central to the GEF strategy in the biodiversity focal area. Given the challenges of conducting impact evaluations, varied methodological approaches were explored, and the report applies two broad evaluation strategies.

The first approach, a theory-based evaluation, consists of detailed case studies of three protected area projects in East Africa:
• Bwindi Impenetrable National Park and Mgahinga Gorilla National Park Conservation Project, Uganda (World Bank)
• Lewa Wildlife Conservancy, Kenya (World Bank)
• Reducing Biodiversity Loss at Cross-Border Sites in East Africa, Regional (Kenya, Tanzania, Uganda), (UNDP)

The report concludes that the projects have contributed to reduced threats and safeguarded global environment benefits, particularly to mountain gorillas and black rhinos in Bwindi and Lewa, respectively. The report highlights that impact was achieved in two of the three protected area projects because an explicit plan for institutional continuity was built into the project from the start.

The second approach is a quasi-experimental analysis of protected areas in Costa Rica, and comprises a statistical analysis of existing time series data on deforestation and protected areas in the country.

The detailed first Annual Report on Impact is available as an information document for the GEF Council. It contains an analysis of the case studies and the quasi-experimental review. The case studies and other supporting documents, such as the approach paper, are available on the GEF Evaluation Office’s website under Ongoing Evaluations

Findings and Recommendations of the GEF-UNDP Joint Evaluation of the Small Grants Programme

The GEF and UNDP together assessed the results of the Small Grants Programme (SGP), the factors that affect these results, and the monitoring and evaluation systems. They also traced the evolution of the SGP, the changes that have taken place, and the drivers of these changes. In addition, they collected quantitative and qualitative data for this evaluation. Datasets for analysis were generated from a variety of sources, such as literature review, country program case studies, project sample survey, interviews, and an online survey.

Here are some of the findings:
• 93 percent of grants from OP3 have been rated in the satisfactory range of the overall outcome
• The 22 reviewed country programs have produced, or are contributing to processes that are likely to result in, global environmental benefits in the GEF focal areas
• Although monitoring and evaluation has improved significantly, there is scope for further improvements
• Compared to the earlier phases, during OP3, management costs were reduced, except for small island developing states
• The current management model of SGP will most likely need to be changed.

• The automatic graduation of country programs older than eight years may reduce the cost effectiveness of the overall GEF portfolio.

Here are some of the recommendations:
• The intended SGP country program graduation policy needs to be revised for GEF-5
• The level of management costs should be established on the basis of services rendered and cost-efficiency
• A process needs to start to change SGP’s central management system suitable for the new phase of growth

These findings and recommendations will be presented to the Council in November 2007.

For more information and a copy of the full document, please visit the GEF Evaluation Office website www.thegef.org . Please send any comments to SGPevaluation@thegef.org

Ongoing Activities

Evaluation of GEF Capacity Development Activities

In November 2006, the Evaluation Office began work on the Evaluation of GEF Capacity Development Activities. To date the evaluation team has completed its approach paper, literature reviews and country case studies of the Philippines and Vietnam. The country case studies examined the nature and results of the national, regional and global interventions and related these to capacity development targets at the policy, institutional, and individual level in each country. These case studies will be made available to the Council as information documents.

The country case studies found that the GEF portfolios in the Philippines and Vietnam include considerable capacity development activities. The results are generally positive and in some areas significant. Although there are many improvements to capacity at the individual, institutional and systemic levels, there are doubts about the sustainability of a number of capacity development outcomes. In particular, the evaluation found that there has been no systematic monitoring or evaluation of overall capacity development performance at the country level, which could promote improvements to approaches.

After reflecting on the issues identified by the country case studies, the Evaluation Office decided that further work is needed to apply the analytical framework developed by the country studies across the GEF portfolio. The evaluation team will now conduct a meta-evaluation of capacity development findings based on a review of a sample of Terminal Evaluations and previous Evaluation Office reports, to explore the prevalence of the key issues identified by the country case studies. This and other additional work will help develop a set of tools to enable forthcoming Annual Performance Reports, Country Program Evaluations, and OPS4—in order to evaluate the achievements of capacity development activities on a broader scale.

For more information and the country case studies, please visit the GEF Evaluation website www.thegef.org in the Ongoing Evaluation section; or contact David Todd at dtodd@theGEF.org.

GEF Country Portfolio Evaluation for Africa

The GEF Evaluation Office launched the GEF Country Portfolio Evaluation (CPE) for Africa with the initial missions between August and October 2007. The CPEs for Africa are part of an ongoing evaluation of GEF support at the country level. The evaluations explore three key areas: 1) relevance of GEF’s support to the national sustainable development agenda and environmental priorities and the GEF mandate and objectives; 2) efficiency of the preparation, implementation and evaluation of the GEF support; and, 3) results of the GEF support by GEF focal area (biodiversity, climate change, international waters, land degradation and persistent organic pollutants) at the national level. Four countries were selected in the financial year 2008: South Africa, Cameroon, Benin, and Madagascar.

The first mission sought to introduce the evaluation to major GEF stakeholders involved, receive feedback on the terms of reference, identify issues to be addressed by the evaluation, visit projects to talk to beneficiaries and project implementers, and form the evaluation team. GEF EO staff evaluation managers are working with a team of local and international consultants.

Terms of Reference for each country will be finalized in the next few weeks. The draft evaluation reports will be presented and discussed in each country at consultative workshops next year. Council will discuss these evaluations at its April 2008 meeting.

For more information on the Country Portfolio Evaluations, please visit the GEF Evaluation website www.thegef.org in the Ongoing Evaluation section; or contact Timothy Ranja at tranja@thegef.org.

GEF Evaluation Office Prepares Terms of Reference for Resource Allocation Framework Mid-Term Review

As part of the Resource Allocation Framework (RAF) decision of 2005, the GEF Evaluation Office was requested to prepare a mid-term review after two years of implementation to help with decisionmaking on a GEF-wide RAF.

In July 2007, GEF Evaluation Office began preparing the Mid-Term Review of the RAF by circulating the review approach paper to Council members, focal points, conventions, and NGOs for comments. The approach paper sketched out key issues that have emerged during the first year of RAF implementation: the formulation of the benefits and performance indices; cofinancing, particularly in less developed countries; legal validity of the RAF; and implementation and organizational aspects. GEF family stakeholders provided much guidance and comments on these key issues, which were fed into preparation of terms of reference. In early September, the terms of reference were circulated to stakeholders.

The terms of reference will be submitted and discussed at the November 2007 Council meeting. For more information and the approach paper, please visit the GEF Evaluation Office website under RAF Mid term review; or contact Lee Risby at lrisby@thegef.org.