The role of impact evaluation on service delivery within the public sector organizations

In spite of the potency of impact evaluations in the performance of most organisations, particularly in the public sectors, there are still scepticisms about its efficacy in terms of service delivery. However, these scepticisms cannot overshadow the relevance of impact evaluation on service delivery within the public sector organisations. The paper aims to show the role of impact evaluation on service delivery, particularly in the public service organisations. The paper has utilised a qualitative method, which is based on conceptual analysis, theory building and literature reviews. It considers the role of impact evaluation on service delivery in the public sector organisations from an “emic” perspective. It is proposed in this paper that impact evaluation will assist organisations to determine and adopt the best practices that will create the enabling environment for improved service delivery.


INTRODUCTION
Recently, there is an increasing demand to evaluate the impact of programs and policies in the public service.The reason for this trend revolves around the need for these organizations to deliver efficient services.Organizations, including the ones in the public sector are faced with uncertainty in a constantly changing environment and need to learn from previous examples, what is most suitable to improve service delivery, because public service organizations are constantly under pressure from the public to provide quality services.
The establishment of the Department of Performance Monitoring and Evaluation within the Presidency in 2010 was a clear demonstration of the South African government's commitment to ensure that its performance makes a meaningful impact on the lives of citizens.This department adopted a result-based approach focussing on improved service delivery through performance monitoring and evaluation across all spheres and organs *Corresponding author.E-mail: wiukpere@uj.ac.za. of the state.According to Engela and Ajim (2010: 13 to 14), the aim is to introduce a specific focus on performance and to have an early warning system in place that will allow the South African government to recognize bottlenecks on delivery times.This article will explore the importance of impact evaluation in public service organizations by using case studies of international public service organizations.Advocates and critiques of impact evaluation will provide the framework for assessing the value of impact evaluation and enables one to understand the advantages and possible disadvantages of this process.One of the authors also briefly reflected on his own research on impact evaluation in the South African Police Service (SAPS) by illustrating the possible disadvantages in situations where the impact of programs is not evaluated.

Problem statement
In spite of the powerful role of impact evaluations in the performance of most organisations, particularly in the public sectors, there are still scepticisms about its efficacy in terms of service delivery.However, these scepticisms cannot overshadow the relevance of impact evaluation on service delivery within the public sector organisations.

Research questions
The following research questions will be answered in this paper: 1. How do these organizations determine the effectiveness of their services delivery?2. What is the role of impact evaluation as far as service delivery is concerned in the public sector?

Research objectives
The objectives of the research are as follows: 1. To evaluate how organizations determine the effectiveness of their service delivery.2. To establish the role of impact evaluation in the delivery of services by public sector organizations.

RESEARCH METHODS
The paper is a qualitative study that is based on con-ceptual analysis and theory building.It considers the role of impact fact on service delivery in the public sector organisations from an "emic" perspective (author's viewpoint).The analysis included a review of literature of the international and local case studies relating to impact factors.Previous research by the author on impact factor was also brought to the fore and analysed.Savedoff et al. (2006: 13 to 14) are of the view that the value of impact evaluation is best understood as part of a wide organized activity of learning, in which evidence is built over time and across different contexts, thereby forming the basis for better policymaking and program design.This type of knowledge is in part a public good, in the sense that once the knowledge is produced and circulated, anyone can benefit from it without exhausting it or excluding others from its benefits.Investments in building knowledge can thus have incredible and unpredictable returns.For example, the discovery in the late 19th century that cholera was transmitted through contaminated water has saved untold lives around the world.When a country learns what works (or not), that knowledge can be invaluable to others facing similar problems.The duplication of impact evaluations in similar programs in different places builds an increasing foundation of evidence and is the most systematic way of improving public policy and efficient utilization of public fund.

Graan and Ukpere 10459
World leaders have agreed to support the establishment of an independent body to sponsor comprehensive impact evaluations of social programs in developing countries.Consensus was reached among participants that such studies were crucial to learning about what works in development, and there was a call for developing country governments to play a central role in the new body.Participants agreed that this newly created body should be established to direct funds to high-quality, independent impact evaluations around key questions that confront policymakers in donor agencies and developing countries' governments (Centre for Global Development, 2006).
There is a growing appreciation within the development community that an important aspect of public sector management is the existence of a performance orientation in government.Such an orientation of evaluation culture is considered to be one possibility for improving the performance of a government, in terms of the quality, quantity and targeting of the goods and services which is produced by the state.In support of this objective, a number of countries are working to ensure results orientation through building or strengthening their monitoring and evaluation systems.Evaluation specialists have a strong advocacy that monitoring and evaluation systems are something good to have because of their inherent merits (Mackay, 2006: 1).
If impact evaluations did not exist today, it will then be difficult to have access to the information needed for evidence-base decisions.Attention to the measurement of results makes it imperative to lay down the foundations today so that people can learn about the effects of their actions in future.Any investment takes time to deliver benefits.Hence, building the kind of knowledge generated by impact evaluations is one of the best investments that have been made today (Savedoff et al., 2006: 26).
The United Kingdom (UK) government, for example, uses impact evaluation extensively, to ensure that policies, programs and public services are planned and delivered as effectively and efficiently as possible, and is at the heart of their performance management system.Governments need to know at the start of policy development, and after policies have been implemented, their expected and achieved impacts, in terms of both the positive and negative outcomes (Davies, 2004: 2 to 4).The need for sound evaluation to be at the heart of policymaking has been recognized in a number of UK government's publications (Owens, 2003: 2).Similarly, Pollit (as quoted by Sandberg et al., 2002: 44) argues that one of the most distinctive characteristics in European state steering during the past decade has been the overwhelming interest in new public management ideas, resulting inter alia, in an evaluative state.
The importance of impact evaluations was further emphasised in "Assessing results using impact evaluations ( 2007)".This paper advocates that impact evaluation is essential to making assistance more effective.This paper also argues that impact evaluations contribute to a culture of accountability and helps to foster a culture of accountability within governments of developing countries.Credible impact evaluations should, however, gradually become a central input to the design of policies.Impact evaluations are essential for creating a knowledge bank.
The number and quality of impact evaluations are gradually increasing, but research also needs to move on to bigger questions, which include understanding why the same intervention can be successful in some circumstances and settings, but problematic in others, and how to make policies more effective.Credible, rigorous impact evaluations have large returns, because they improve our knowledge about the effects of policies, which enables policy makers to design interventions, and help to ensure the political sustainability of successful programs.
The South African Public Service has also realized the significance of impact evaluation.Until recently, no coherent system of systematic policy monitoring and evaluation existed in the South African Public Service.However, this state of affairs has changed in 2005 with the introduction of the Government-wide Monitoring and Evaluation system (GWM and ES).This program is intended to establish a uniform system of monitoring and evaluation across all spheres of government (Cloete, 2009: 293).Bamberger (as quoted by Cloete 2009: 294) acknowledges the significance of evidence-based policy analysis and assessment to improve the quality of policy decisions that is fast emerging as the prevailing international paradigm for policy analysis and management.

CRITIQUES VERSUS ADVOCATES OF IMPACT EVALUATION
In spite of the great value of information from impact evaluation, there is at times scepticism about its utilization.Sometimes, it is met with outright rejection.Nevertheless, these objections of impact evaluation are not supported with facts (Savedoff et al., 2006).
Some critics are of the view that good impact evaluations may not be appropriate in answering certain questions of importance to decision-makers.Nevertheless, advocates of impact evaluation noted that although it may be difficult to design a high quality impact evaluations that can answer policy questions such as, under what circumstances should a country have a fixed exchange rate?The variety of questions that can be dealt with and answered in a fairly simple way by welldesigned impact evaluations is much wider than is normally recognized.For example, do vitamin A supplements reduce infant mortality?Impact evaluation can therefore provide answers to questions such as: do cash transfers to poor families that are restricted from attending school and utilization of preventative health care services, improve children's health and schooling?Even questions that might be considered quite difficult to answer can be rigorously studied.For example, do quotas for women's participation in political decisionmaking improve allocations of public funds?(Savedoff et al., 2006: 22) to mention a few.
In addition, some critics contend that impact evaluations are too costly.However, according to the advocates, this argument is often made by comparing the cost of an evaluation to the program that is its subject.Therefore, the appropriate measure is not the program cost, but the value of the knowledge it would produce.For example, evaluations of demonstration training programs in the United States and Latin America have sometimes exceeded a third of the initial program costs, but the evaluation results affected decisions regarding the implementation of much larger national programs.The principal cost of a study is the cost of data collection, and the cost of collecting data for a bad study is just as expensive as collecting data for a good study.Hence, critics should understand that the cost or difficulty of good impact evaluations is not a general fact, but rather, one that has to be judged for particular questions and contexts (Savedoff et al., 2006: 23).
Moreover, some critics argued that impact evaluations do not provide important information about how programs function, they are too complex and also do not influence policymaking.Nonetheless, according to the advocates of impact evaluation, good impact evaluations are relatively easy to present to policy makers, with less effort.For example, the Manpower Demonstration Research Corporation (MDRC) that conducted some trials of numerous state welfare programs in the United States and were readily able to convey their findings to policymakers.MDRC's studies had a significant impact on United States welfare reform legislation in the mid-1980s.
In Argentina, matching techniques were used to estimate the impact of a program that provided support to families that had become unemployed during the severe economic crisis of 2002.The evaluation showed that the program reduced overall unemployment and extreme poverty.This impact evaluation has helped governments to develop appropriate policies during macroeconomic upset.The comprehensive nature of the Programmea de Educación, Salud y Alimentación (PROGRESA) evaluation, a conditional cash transfer program in Mexico, was one of the reasons why the administration of President Fox supported a program inherited from the previous administration, which is an unusual outcome.This program has continued in the administration of President Calderon (Assessing results using impact evaluations, 2007).Some critics commented that impact evaluation produce results too late to be used by decision-makers.However, advocates of impact evaluation remarked that the time taken to produce results greatly depends on the questions being studied.Some rigorous impact evaluations produce results within a matter of months.Others take longer, but are still available on time to influence important policy decisions.For example, the initial findings of Mexico's impact evaluations of its national conditional cash transfer program were available on time to convince a new administration to preserve it.It is also possible to design impact evaluations that generate useful feedback during implementation.
For instance, a multi-year study of the impact of HIV education in Kenya was designed to not only to monitor the long-term impact, but also to evaluate the intermediate outcomes, such as the accuracy of knowledge about HIV transmission.Measuring the success or failure of reaching such intermediate goals can help program managers to make necessary changes to improve implementation.It is more important to have accurate information about what programs work, even if it takes some years to acquire, than to have inaccurate information that is generated quickly (Savedoff et al., 2006: 24).
Additionally, critics contend that impact evaluation cannot be ethically implemented.However, advocates counteract this view by stating that impact evaluations that rely on collecting data from control groups are unethical, because they exclude people from program benefits.This criticism only applies when resources are available to serve everyone as soon as the program begins.Whenever one has a reasonable doubt of a program's effectiveness, or concerns with unexpected negative effects, it is not only ethical, but a crucial responsibility to monitor and evaluate the impact (Savedoff et al., 2006: 23).
Finally, critics have argued that impact evaluations only inform whether or not something has an impact, without saying why and how.As opposed to this argument, advocate contends that a good impact evaluation can provide reliable evidence about the method through which results were obtained, when it simultaneously collects information on processes and intermediate outcomes (Savedoff et al., 2006: 25).Impact evaluation is about understanding what works in development, under what circumstances, and why.
They are, however, not about only measuring inputs, such as, how many school meals were delivered in a school feeding program.Instead, they are about assessing impacts on key development objectives, such as children's nutritional status by determining who benefits from a program and by how much the impact evaluations are significant to making aid more effective (Assessing results using impact evaluations, 2007).

INSTITUTIONALISATION OF PROGRAM EVALUATION IN PUBLIC SERVICE ORGANISATIONS: INTERNATIONAL EXPERIENCES
International experiences of countries' efforts to establish Graan and Ukpere 10461 impact evaluation in public service organizations demonstrate the link between evaluation and the larger mandate of improving public service management.These examples illustrate the realization of the international community's efforts to improve public service management through impact evaluation.Blomquist (2003: 24), Dubois (2002) and White (2006) identified the significance of impact evaluation in the public service citing a few examples: In Sweden, evaluations were first undertaken in the 1950s by public commissions preparing policy decisions.The commission system combined the views of stakeholders, and introduced research findings into decision-making.From the 1960s onward, evaluation began to be viewed as an activity that could continuously provide decision-makers with information.Specialized research bodies and agencies were then established with evaluation as their main task.Evaluation became an important part of Australia's public management reform in the mid-1980s, when the government's evaluation strategy was launched.All public programs, or significant parts of it, are reviewed once every few years.All major new proposals include an evaluation strategy.Ministries are required to provide an annual evaluation plan, and results of major evaluations are required to be made public.
Chile developed a system of performance indicators for government programs.A political agreement was reached to evaluate all programs in Chile.In Québec evaluation has become the primary mechanism that include the development of targeted results, the achievement of concrete results and the measurement of results achieved and reporting of the outcome.Québec has proposed through legislation that the relevance, effects and performance of these programs be evaluated more extensively.

THE VALUE OF IMPACT EVALUATION WITH SPECIFIC REFERENCE TO THE POLICE ORGANISATIONS
For present-day policing to be relevant and to meet the expectations of the community, it must be managed strategically to have an outcome focus rather than a continuing dependence on output measures of performance.This outcome focus must include systems to generate useful knowledge (Coleman, 2008: 318).International examples of police organizations' impact evaluation processes illustrate the value of impact evaluation in the generation of useful knowledge to improve service delivery in these organizations.

COMMUNITY ORIENTED POLICING SERVICES (COPs) IN AMERICA
The Violent Crime Control and Law Enforcement Act of 1994 authorized the creation of an intergovernmental grant program within the United States Department of Justice to fund the hiring and redeployment of 100,000 additional community policing officers on America's streets to be administered by COPS.COPS funding were expected to reduce fear of crime, increase the quality of life, promote social control and boost citizen satisfaction with police services, while improving information sharing between citizens and the police.Two questions were asked: Do COPS funding stimulate local police department spending in large cities? And, do COPS funding discourage crime in large cities?The results of this study indicated that COPS funding awarded to large cities did not stimulate local spending and that the cities may have used the grants to compensate local police expenditures.Whether or not community policing is an effective crime reduction strategy, this evaluation found that federal funding for community policing was associated with marginal reductions in crime in large cities (Muhlhausen, 2006).

THE NATIONAL REASSURANCE POLICING PROGRAM IN ENGLAND
An impact evaluation of the national reassurance policing program (NRPP) in England was conducted between 2003 / 2004 and 2004 / 2005.Reassurance policing is a model of neighbourhood policing which seeks to improve public confidence in policing.The primary aim of the study was to fill a gap in the research evidence in England, on the impact of local policing activities.This evaluation is one of the most robust tests of a neighbourhood policing model ever completed, and a key step in building evidence.The basis of a neighbourhood policing model is to have dedicated police resources for local areas, and for police and their partners to work together with the public, in order to understand and tackle the problems that matter to them most.
The results illustrated that the NRPP delivered positive changes, such as reduction crimes and antisocial behaviour, feelings of safety and public confidence in the police.The findings also show that the public did not only notice increased police foot patrol, they also noticed the efforts that the police put into their work to address the problems associated with anti-social behaviours (Tuffin et al., 2006).

THE BOSTON GUN PROJECT AND OPERATION CEASEFIRE
The Boston gun project was a problem-oriented policing initiative specifically aimed to address serious, largescale crime problems such as homicide and victimization amongst young people in Boston.Boston experienced an accelerated youth homicides between the late 1980s and early 1990s that led to the 'operation ceasefire' inter-vention.Operation ceasefire included two main elements, namely a direct law enforcement counterattack on illicit firearms traffickers that are supplying youth with guns, and an attempt to generate a strong deterrent to gang violence.The yearly decrease in the number of youth homicide in Boston, certainly suggests that something noteworthy has happened after operation ceasefire was implemented in mid-1996.Boston has on averaged 44 youth homicides per year between 1991 and 1995.However, in 1996, this number decreased to 26, and then further decreased to 15 in 1997.
Evaluation suggested that operation ceasefire was associated with a large reduction in Boston youth homicide.In order to compare youth homicide trends in Boston to youth homicide trends in major U.S. and England cities various models were introduced.Results indicated that only Boston has experienced a significant reduction in the monthly count of youth homicides, coinciding with the implementation of operation ceasefire.The principal intervention, operation ceasefire, was likely responsible for a substantial reduction in youth homicide and youth gun violence in the city (Braga et al., 2000: 2 to 4).

RESTRUCTURING IN THE SOUTH AFRICAN POLICE SERVICE (SAPS)
In 2006, the SAPS implemented a restructuring process with the primary aim of improving service delivery.Consequently, specialized units in the SAPS including the family violence, child protection and sexual offences (FCS) unit were restructured into a decentralized structure.Neglecting to evaluate the impact of this process resulted in great public outcry.
Research conducted by Van Graan (2008) evaluated the impact of the restructuring process in the FCS, with the aim of identifying the effectiveness of this process regarding the functioning of the unit.One of the key findings of this impact evaluation revealed that the restructuring of the FCS had a detrimental effect on the service delivery of the unit, as well as on the client system it served.Had the impact of this process been evaluated the negative effect on service delivery of this unit could have been minimized and the necessary adjustments implemented to improve services delivered of the unit.
The research findings furthermore indicated that the rationale for the FCS restructuring was possibly based on incorrect perceptions, rendering the process ineffective.Had the restructuring not taken place, service delivery may not have been adversely affected the way it did.However, in 2010 the Minister of Police instructed the reinstatement of this unit back to its former centralized structure.

Conclusion
In conclusion, there is a universal consensus that public service organizations should provide efficient services to the public.However, how do these organizations determine what works and what does not work in order to provide satisfactory services?This paper has focused on numerous international examples, which illustrates the advantages of impact evaluation.Impact evaluation will enable organisations to determine and adopt best practices to improve services delivery.It is thus evident that impact evaluation is rapidly emerging as the prevailing international standard to evaluate policies and programs in public service organizations in order to enhance the delivery of services.The findings of previous research by one of the authors show the possible disadvantages if the impact of interventions are not evaluated.This harmonizes with other empirical evidences that illustrate the significance of impact evaluation as a management tool for improving service delivery within organizations, particularly in the public sector.