MONITORING AND EVALUATION GUIDELINES -- FOR BETTER MANAGEMENT OF DEVELOPMENT PROJECT -- VERSION 1.0 NOVEMBER 2005 PREPARED UNDER THE TECHNICAL COOPERATION PROJECT OF JICA/PDMO DEVELOPING THE CAPACITY OF THE GOVERNMENT TO POST-EVALUATE EXTERNALLY FUNDED PROJECTS MONITORING AND EVALUATION GUIDELINES Table of Contents I. PROJECT CYCLE AND EX-ANTE EVALUATION 1. The Project Cycle Management. Logical Framework Approach of Development Project.
Integrating Ex-ante Evaluation into LP-MIS. MONITORING OF ON-GOING PROJECT 1. Objective of PDMO’s Monitoring of On-going Projects. Monitoring Steps of Implementation Progress.
Content of Monitoring. Performance Management and Reporting. Management Information System. Organizational Relations for Monitoring.
III-6 Sample Project Status Report IV. EX-POST EVALUATION 1. Framework for Ex-post Evaluation 1. Definition and Principles of Evaluation.
Five Evaluation Criteria. Procedure for Ex-post Evaluation 1. Preparation and Implementation of Field Survey. Conclusion of Evaluation.
IV-19 Sample Questionnaire Sample Evaluation Summary Sample Evaluation Report V. PERFORMANCE MONITORING INDICATORS 1. Performance Indicators in Development Project Evaluation V-1 2. Types of Indicators - World Bank, JBIC and PDMO.
Indicators of Major Sectors. V-5 Sample Indicators VI. Rating by JBIC I. Purpose of the Guidelines The Project Team 1 is pleased to print the first edition of Monitoring and Evaluation Guidelines for PDMO as part of our project activities.
The objective of the Guidelines is to present to mainly for PDMO staff information on various existing tools aimed at facilitating evaluation at the project design, implementation or monitoring, and project completion, and operation stages, including ex-post evaluation of completed projects. Throughout the document, the word "evaluation" is used generically, that is, as the analyses conducted at all phases of the development process. But it may sometimes indicate “ex-post evaluation” in some context. Evaluation is a useful tool to enhance project performance.
This handbook is designed to provide users with guidelines in support of evaluation activities at all stages of the project cycle, in order to assist in improving project performance. Sound project evaluation conditions must be established at the project design level, so that projects can be successfully monitored and evaluated. Unfortunately, this is not the current practice in development projects, not only in Thailand but also in other countries too. Only a few of the loan proposals contained the basic elements required to monitor result-based project progress, or to determine at a later point in time whether or not the development objectives were achieved.
A tool can be used to improve this situation at the project design level. The logical framework can be incorporated in project designs in order to establish a clear project purpose at the design level. Assessment is also important to test if projects submitted for approval have the necessary elements to enable result-based evaluation at a later point in time. The means for incorporating these tools are described in this handbook.
Successful projects do not solely depend on valid project designs -- projects also require effective implementation. Many of the loan projects are experiencing difficulties in implementation. JBIC’s ex-post evaluations2 show that of the 74 1 The team members consist of JICA expert and OSU manager/staff of PDMO, working on the Technical Cooperation Project on Development the Capacity of the Government to Post Evaluate the Externally Funded Project, JICA/PDMO 2 Country Review Report Thailand: Meta Analysis of Ex-post Evaluation Reports, JBIC 2003 I-1 projects evaluated by JBIC, 15% of them were delayed for more than three years and 45% of them were completed with one-to-three year delay. Similar situations are observed in the World Bank funded projects and ADB funded projects, too.
This reinforces the need to improve project design and provide the means for effective monitoring of project implementation. At the completion stage of a project, or some years later, most evaluations are conducted. If conducted carefully, ex-post evaluations can generate knowledge that is indispensable to the success of future projects. On the foundation of well-conducted ex-post evaluations, subsequent project designs and execution strategies can benefit from the collective experience.
Effective ex-post evaluations and impact evaluations, described in this guideline, are also important instruments for improving overall performance over a long-term period. The Project has produced this handbook as a guide that underscores the importance of evaluation as a learning tool of the project, from design to execution and monitoring, by developing standards for effective evaluation processes and products. This handbook would not be the quality product, but would be pleased to invite your thoughts and comments for continuing effort to provide the best evaluation products possible. The Guidelines is a document designed to evolve and change as we learn from its use.
Structure of Guidelines The Guidelines is divided into following four chapters. Chapter II is a general overview of evaluation, and project cycle and ex-ante evaluation, how the application of evaluation is essential to the design of the project at the project preparation phase. Chapters III-IV describe, the monitoring of the project at the project implementation phase, and the project's impact at (or after) the project completion phase. Chapter V discusses performance monitoring indicators, which is one of the most important elements of present monitoring and evaluation.
Quick Reference Table I-1 Evaluation Terminology Term Definition Effectiveness The extent to which the project produced its expected outputs and thereby achieved its purpose and contributed to its goal. Efficiency The extent to which project inputs were supplied and managed and activities organized in the most appropriate manner at the least cost to produce the necessary outputs. Evaluability The extent to which a project has been defined in such a way as to enable evaluation. Evaluation The parameters that define the evaluation and how it is to be design undertaken, including the evaluation questions, methodology, data collection plan, methods of analysis.
Evaluation An assessment -- as systematic and objective as possible -- of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, developmental efficiency, effectiveness, impact and sustainability of development. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipient and donors. (OECD/DAC, 1991) Ex-ante An appraisal or needs assessment.
Also used for the "ex ante evaluation phase" of the evaluation cycle, which includes feasibility studies, identification of project objectives and other such functions done before the project begins. Ex-post An evaluation conducted upon the completion of project evaluation execution. In JBIC, ex-post evaluation is generally conducted 2 years after the conclusion of the project. Impacts and As defined in the Logical Framework Analysis approach, Effects impacts and effects refer to the planned and unplanned consequences of the project; effects generally relate to its purpose and impacts relate to its goal.
Indicator The quantitative and qualitative specification for an objective, used for measuring progress toward attaining the objective. I-3 Lesson A lesson learned is a general hypothesis based on the findings Learned of one or more evaluations, but which is presumed to relate to a general principle that may apply more generally. Monitoring Monitoring is a procedure for checking the effectiveness and efficiency of implementing a project by identifying strengths and shortcomings and recommending corrective measures to optimize the intended outcomes. Monitoring in this context is a quality check or appraisal of activities of a whole system while it is actively in operation.
Project Cycle Management activities and decision-making procedures used Management during the life-cycle of a project (PCM) Project Project performance means achieving expected results within Performance planned timeframes and resource limits. Qualitative Data that use non-numeric information for description. data Generally words, but may include photographs and film, audio recordings, etc. Quantitative Information expressed in the form of numbers.
May be ordinal Data or ratio. Reliability The quality of a measurement process that would produce similar results on (1) repeated observations of the same condition or event, or (2) multiple observations of the same condition or event by different means. Validity The extent to which the conclusions of an evaluation are justified by the data presented. PROJECT CYCLE AND EX-ANTE EVALUATION 1.
The Project Cycle Management The project operations usually follow a sequence of procedures. They are project identification, formation, appraisal (ex-ante evaluation), project implementation and supervision, project completion, ex-post evaluation and monitoring after completion of the project. The lessons learned from ex-post evaluation and monitoring after project completion provide useful information, and are fed back to the preparation, appraisal and implementation of future projects. Thus the whole series of procedures forming the circle is called the project cycle.
The cycle of operations1 for managing development projects by PDMO has four phases, as shown in Figure II-1 below. Approval Implementation Ex-ante Implementation Evaluation Monitoring Project Cycle for PDMO Ex-post Ex-post Evaluation Monitoring Operation Completion Figure II-1 PDMO’s Project Cycle 1 Project cycle normally has an identification/formulation stage. PDMO, however, is not usually involved in this stage. Therefore, it is dropped from the chart.
II-1 As another example, each step of the project cycle corresponding to the JBIC loan process is shown in the Figure II-2. Figure II-2 Project Cycle and JBIC Loan Procedure Project Cycle Management (PCM) is a term used to describe the management II-2 activities and decision-making procedures used during the life-cycle of a project PCM helps to ensure that: ¾ Projects are relevant to a strategy and to the real problems of target groups/beneficiaries; ¾ Projects are feasible, meaning that objectives can be realistically achieved within the constraints of the operating environment and capabilities of the implementing agencies; and ¾ Benefits generated by projects are likely to be sustainable. To support the achievement of these aims, PCM: ¾ requires the active participation of key stakeholders and aims to promote ownership; ¾ uses the Logical Framework Approach (as well as other tools) to support a number of key assessments/analyses (including stakeholders, problems, objectives and strategies); ¾ incorporates key quality assessment criteria into each stage of the project cycle; and ¾ requires the production of good-quality key document(s) in each phase (with commonly understood concepts and definitions, such as Logframe, Progress monitoring report, Project completion report, Ex-post evaluation report, etc.), to support well-informed decision-making. Logical Framework Approach of Development Project The logical framework (also known as “Logframe”) is a conceptual and analytical tool for undertaking sector analysis, project planning, and project management.
The logical framework process is distinct from the logical framework matrix, which you may be familiar with. The logical framework process starts with the analysis of a sector and ends with the design of a project or program using the logical framework. Although the PDMO is not involved in the entire process of logical framework exercise, it is useful to know the process through which projects/programs are formulated. A brief overview of this logical process is presented below, step by step.
Step 1: Assess Sector Performance Sector performance is assessed by using performance indicators that reflect the contribution of the sector to the larger economy and to the quality of life of II-3 people. Examples of sector indicators are “mortality” for the health sector, “productivity” for the industrial sector, and “traffic flows” for the transport sector. Each sector has its own set of indicators which, taken as a whole, reflect the performance of the sector. Step 2: Identify Sector Performance Problems/ Opportunities Problems or opportunities are identified as issues of concern.
Such problems or opportunities are identified in relation to a specific sector performance indicator. Examples would be “rising mortality rates” or “deteriorating productivity” or “increasing traffic congestion”. Step 3: Cause-Effect Analysis of Problems/ Opportunities A core problem or opportunity is selected to improve sector performance. It is analyzed to identify the causative factors as well as consequent effects.
It is usually diagrammatically presented in the form of a cause-effect tree. The effects of the problem indicate its wider dimensions and impacts on the economy. The causative factors identify the variables influencing the problem/opportunity and provide the basis for solution.