EL BLOG DE LA FIIAPP. Cooperación Española

When approaching a public policy evaluation, one of the first commonly accepted methodological steps is to analyse the logic of the intervention: How was the policy designed? Does a clear relationship exist between your objectives (explicit or implicit) and your results, activities and outcomes, and among all the interlinking elements?

To address this complex situation, governments have adopted extremely different measures that affect very diverse sectors of society: health, the labour market, housing, mobility, taxation, and so on.

But what are their specific, general, medium, short, long-term objectives? In principle, we might think that the main objective would be to save as many human lives as possible, to prevent the collapse of the health system, to enable the population to keep its rights and purchasing power. However, to make a rigorous, exhaustive assessment, we need to establish clear goals against which to measure progress, achievements and failures. Against which criteria can the adopted measures be evaluated to determine their effectiveness, that is to say, against their capacity to achieve the proposed goals?

“Against which criteria can the adopted measures be evaluated to determine their effectiveness?”

 Of course, this leads to complex and politically and socially sensitive questions: What is the acceptable goal in terms of loss of human life? How many (non-temporary) unemployed people are acceptable? How many small and medium-sized companies can be expected to disappear? These are just a few examples.

In order to carry out an evaluation of the effectiveness of the adopted measures, we therefore need out governments to:

-Clearly define existing and potential social problems.

– Based on these, establish a Contingency Plan, ideally, inserted into the Decree on the Declaration of the State of Alarm (so that it forms part of the Legal System), with a set of approved measures that address the objectives in a clear manner, also defining process and result indicators for monitoring their evolution.

-Establish new measures to build a coherent master plan to achieve the objectives defined.

-Design a thorough monitoring system with flexibility to modify measures based on the results obtained and the evidence collected.

A mature society, the vast majority of which is behaving impeccably and following the recommendations deserves to know the parameters used by institutions to manage the situation for subsequent evaluation.

If we consider an impact analysis in the strict sense, the difficulty lies in the absence of a counterfactual argument: What would have happened if no measures had been adopted rather than taking this course of action? What would have happened if other measures had been taken or had been taken earlier? Which of the possible approaches would be most successful to deal with the crisis?

Every day and every hour that goes by, data and figures are being generated for drawing graphs and compiling statistics that evolve rapidly and are ever changing as they are analysed by country, by region and by continent. They provide valuable information and reveal developments on which to build hypotheses to test.

In view of the data and the context, countries have adopted different solutions, but the existence of too many variables that are difficult to control (strength of health systems, baseline citizen health, population pyramid, cultural habits, life expectancy, etc.), are preventing a comparative analysis with the necessary rigour to draw robust conclusions.  However, we do have similar previous experiences that can give us some analysis guidelines.

“The existence of too many variables that are difficult to control is preventing a comparative analysis with the rigour necessary to draw robust conclusions”

 Experts in the field have mentioned that during the SARS (2003) and influenza A (2009) crises, there was a breakthrough in research and the fight against these diseases. It is also evident that as the outbreaks were controlled and the emergency ended, the funds to continue research and development of different drugs either dried up or were drastically reduced, and these lines of work stopped. The work was unfinished. Unfortunately, we did not use those earlier crisis to slow the current pandemic earlier and better.

One of the benefits of evaluating public policies can be the lessons learned. Hopefully an ex post evaluation of the current crisis will teach us lessons that were previously lacking and we will use them to build strengths that, without being aware of it, we are all building these days.

The EVALÚA Project is attending to the new priorities of partner institutions to create a baseline that will serve as a reference point for  subsequent evaluation of current actions, with a comparative approach at the regional level.

When it comes to preparing the National Evaluation Agendas, different goals come into play: accountability, policy improvement or programme evaluation (its design or management), transparency promotion as a democratic tool, and ultimately institutional or managerial learning. 

In terms of Disaster Risk Management, this evaluation is relevant to all of these matters. It is also a particularly sensitive matter: in addition to exposing the major vulnerabilities that afflict a large part of the world’s population, there are known cases of regrettable deficiencies in fund management for emergency and reconstruction. The unanimous opinion of specialists, also included in the Sendai Framework, is that in terms of disaster risk having a solid prevention system helps prevent the loss of human lives, as well as material loss and the loss of basic goods for population survival. It will also make a difference when facing subsequent reconstruction. 

On April 16, 2016, an earthquake of magnitude 7.8 (Mw)3 was recorded on the north-east coast of Ecuador.  671 people died and 6,277 were injured. The damage affected four provinces, and fourteen cantons were declared to be in a state of emergency.  

After assisting the first moments of the emergency, the Ecuadorian government approved the 2016 Post-Earthquake Reconstruction Plan, framed  within its Risk Management regulations and in the National Decentralized Risk Management System (SNDGR). The Reconstruction Plan’s aim is territorial recovery, canalise the reconstruction and recovery processes of post-earthquake livelihoods under the criteria of resilience and sustainability through intersectoral and multilevel coordinated interventions. 

What has emergency and reconstruction assistance coverage been like for the population in the affected areas? How many families benefited from this asisstance and for how long? How and in what way were the shelter, rental and food aid distributed? Were the most vulnerable people included? To what degree was infrastructure rehabilitated? How many public health facilities were rebuilt and rehabilitated? What is the degree of citizen satisfaction with regard to medical care and services? Is Ecuador’s National Decentralized Risk Management System working as well as it could to prevent and manage disasters of this nature? What improvements should be made to minimize the consequences of possible future disasters? 

These are just some of the questions that the evaluation will answer: Transparency, improvement, learning. 

The consulting team hired by EVALÚA has already completed the field work. In the next few weeks we will have the answers. 

What is being assessed?

Janet López: In Uruguay, we are now tackling the Casavalle Plan, which is a comprehensive policy and a demand that has to do with policies of inclusion. It is a project being developed by Montevideo City Council that has had input from various ministries, which have submitted different policies.

A comprehensive policy consists of input from a variety of stakeholders, from our Ministry of Social Development and Montevideo City Council: in regard to infrastructure, the housing sectors, how overall development is applied to sport and culture as an essential part of the development of a society. These different lines make this a comprehensive policy.

We are now gathering new data to check how this policy has progressed.

Teodora Recalde: In Paraguay, we started a process of performance budgeting in 2011. We began by introducing three types of performance indicators, public programme assessments and a management balance sheet. We are working very hard on the institutional assessment and starting to evaluate the designs and results of the different programmes, such as the agriculture programmes, judicial and prison programmes, health programmes and other institutions that have specific programmes.

Why is the assessment necessary? What is the objective?

J.L: The assessment is part of a long-term strategy. We are working on the entire public management cycle: planning, budgeting, monitoring and evaluation. Assessment was the final stage that we included.

The unit that I manage was created 10 years ago and maybe, a little over 10 years ago, no one even talked about assessment. So, as part of the entire public management cycle (planning, budgeting, monitoring and evaluation) the assessment unit was created in 2011. We started doing internal training and now we are tackling a variety of interventions in public assessment.

The fundamental objective is to improve management. This is not a punitive assessment; this must be made clear to the parties involved. It must really contribute to improving management and, in the end, to the use made of public resources

T.R: The purpose of the assessment, from the viewpoint of the budget and the Ministry of Finance, is to design programmes that will reach the public, and for the programmes that are implemented through the budget to have an impact and to look at where the investment is going.

What does EVALÚA offer as a project?

J.L. Casavalle is not the first project on which we have worked with EVALÚA. We have been doing this for some time and I think that EVALÚA contributes technical expertise, from support for reviewing all the terms of reference to the support that the administration of FIIAPP is providing us with. In turn, the exchange that EVALÚA makes possible with other countries in Latin America involved in the project also adds value. Exchanging the different viewpoints that we all have adds value.

T.R. For us, the EVALÚA project is a mainstay; there are comparative advantages in the complementarity between countries. They have created a new work scheme by stimulating the relationships between professionals in different departments, whether the planning department or the financial department, like ours, and an instrumental and methodological contribution to assessment.

What is it like working with EVALÚA?

J.L. To explain what it is like working with EVALÚA I could give examples of many activities. I could emphasise the exchanges and the formation of a common working agenda; the review of assessments that were made in another country and the expertise of the various technicians also add value. That is one aspect and we in particular are working on a monitoring and assessment skill development plan that, we understand, is part of having a nationwide and international language in common with the countries that are involved.

T.R. There are assessments that have already been made with other countries. In particular, in Costa Rica, we worked on the terms of reference for starting assessment and adopting these types of assessment models in that country. For us, this is an advanced process since we are just starting it.

What are the benefits for the public?

J.L There is going to be a direct benefit because what we want to do is to evaluate the real impact of the different policies and approaches that have existed in the region. And we are also going to work on consultative workshops with civil organisations. This will also bring about a result, feedback so that we can see where things must be improved or what to focus on.

T.R. The public will benefit as these programmes will be well designed and the services properly delivered, there will be a direct impact on the public.

In the assessment, viewed from a financial and budgetary point of view, the debate is where to cut resources. One never knows where to cut, so an evaluation is necessary. As you evaluate, you know where to invest. Because, if you invest well, the budget really reaches the public.