Planning and Evaluating Health Services

AIKESH SHRESTHA

Introduction

Evaluation in a planning process is defined as the systematic accumulation, analysis and reporting of data about a program so as to assist in decision making (Health System Intelligence Project 2008). Evaluation is not only the mere accumulation and summarization of data that are clearly relevant for decision making rather it determines the effectiveness, accuracy and efficiency of any purposed plan, and further contributes in the improvement of plan. Evaluation is the most valuable aspect of planning cycle. It is an intregal phase in the planning cycle or the method of convencing, developing, implementing, and modifying planning strategy or decision as shown in figure 1(Davis & Bridgman 2004).

Fig.1. The Planning cycle (Source Health System Intelligence Project 2008).

Importance of Evaluation

As shown in figure 1, evaluation plays a significant role in the improvement in the health service delivery along with provide guidence for the better allocaton of the resources. It acts as an important tool to help the planners to learn and understand wether the plan that has been purposed has achieved its goal or not. And further, evaluation acts as a tool to examine the success and failure of the plan, whether the plan had the favourable impacts and met the target set by the planners, and whether anything that can be introduced to for the betterment of the plan in future. Proper and planned execution of evaluation of plan can be beneficial for any organsation. High standard evaluation strengthen accountability and delivers a rigorous evidence based to inform health care service development and plan design (Agency for Clinical Innovative 2013).Some importance of well executed evaluation includes (ACT Government 2010),

• Accumulation of more information so as to undergo better decision making.
• Better and effective allocation of resources.
• Enhance the mentality to achieve goals and priorities the goals.
• Enhance the skills and knowledge to broaden the concept of plan.
• Assure whether the organisation is moving in the correct pathways and appropriate treatment need to be taken if required.
• Provide guidance to the individual involved in the plan along with the entire staff of the particular organisation

Steps for Evaluation of NSW Renal Dialysis Plan 2011.

The main purpose of this paper is to provide a strategy for evaluation of NSW Renal Dialysis plan 2011 that has been designed by the Statewide Services Development Branch with an aim to improve the renal dialysis services in future and consider all the elements of renal dialysis service delivery including issues and demands which are most likely to affect the delivery of services in coming years. Effective plan evaluation is a well planned and systemic procedure to understand the nature and outcome of plan purposed. For the effective evaluation of the plan, systematic and well managed steps are required. Inclusion of appropriate steps in evaluation strategy helps in the better understanding of the aim of the evaluation and other tools needed for the evaluation procedure. For the effective and well planned evaluation of NSW Renal Dialysis plan 2011, stages has been categorized into two basic criteria which is based on evaluation steps proposed by Wall 2005 and Health System Intelligence Project 2008,

1. Stages for preparation of evaluation
2. Stages for conduction of evaluation

1. Stages for preparation of evaluation

It is essential to understand and well organize the stages that have to be considered prior to conduction of evaluation. Following steps need to be considered for the proper evaluation of Renal Dialysis plan 2011,

1. The entire member or stakeholders whether internal or external, that have been involved in the planning of NSW Renal Dialysis 2011 has to be identified and engaged in evaluation procedure.
2. Set the main purpose of the evaluation.
3. Conducting an evaluability assessment to determine whether the plan evaluation is possible or not and to understand the pathways that stakeholders would consider for the information delivered by evaluation.

Evaluaility Assessment (EA)

Evaluability assessment is a pre-evaluation tool designed to maximize the possibility that any subsequent evaluation of plan, program or policies would result in beneficial and reliable information (Leviton et al. 2010). Evaluability assessment not only indicate whether the plan can be evaluated effectively but also indicate whether the evaluation would improve the effectiveness of plan. Similar to evaluation process, an important procedure of an EA is to compare the plan design to the plan in operation thus early detection of any variation by EA, then time and money will be saved (JJEC 2003). For the effective evaluability assessment following set of questions must be answered by NSW Renal Dialysis plan 2011 (JJEC 2003),

• Does the main objectives and principle of the plan have been clearly mentioned in MSW Renal Dialysis Plan 2011?
• Does this plan serve the population for whom it has been designed?
• Does this plan have the sufficient resources discussed in the plan design?
• Are the activities considered by the plan being executed as purposed?
• Does this plan have the potential to deliver data for an evaluation?

To understand whether the NSW Renal Dialysis plan can be evaluated or not following EA steps can be included (JJEC 2003),

• Study the plan , strategy, design and operation
• Observe the action of plan.
• Examine the plan capability for data collection, analysis and interpretation.
• Determine the possibility that the plan reaches proposed scope and objectives.
• Indicate the reason whether evaluation may or may not help the plan and its stakeholders.

4. Formulation of evaluation team from the stakeholders either internal or external depending upon the field of expertise. The main role of the team is to perform the planning, implementation, interpretation and reporting of the evaluation (Agency of Clinical Innovative 2013).
5. Establishment and confirmation of the evaluation method.

Type of evaluation implemented.

For the effective evaluation of NSW Renal Dialysis Plan following type of evaluation has been employed,

• Formative (Process) Evaluation
• Impact evaluation
• Summative (Outcome) evaluation

Type of Evaluation	Description	Achieved by
Process Evaluation	Examines whether 10 planning principles set by renal dialysis plan have been incorporated as proposed. Strategies proposed have reached the desired population. Stakeholders and target population are satisfied with the plan. Target like dialysis equipments and services purchase, data management and reporting, workforce training, development and innovation, state wise transportation, services networks and home dialysis has been achieved or not.	Routine survey among patient with renal disorder and patients undergone renal dialysis. Queries Interviews with staff as well as focused groups Feedbacks Data review
Impact Evaluation	Examines the immediate effect that NSW Renal Dialysis Plan on The focus group regarding the intervention. The work force regarding on training and education in renal dialysis. Home dialysis services. Transportation facilities for renal dialysis patients. Development of a consistent statewide database and information management.	In-depth interview Open-ended queries Survey Data collection (Routine as well as special) Focus group as well as staff discussion.
Outcome Evaluation	Examines The long term effects of renal dialysis intervention. Whether the desire scope and purpose of the renal dialysis intervention has been achieved. Effectiveness and reliability of intervention. Impact of the plan and target set by the plan to focus group, stakeholders and all the workers involved in intervention regarding renal dialysis.	All the data associated with renal dialysis. Queries and interviews with focus groups and stakeholders. Surveys Feedback mechanism. Annual meetings.

Table 1. Representing types of evaluation approached

6. Formulating evaluation questions which would mainly focus on plan implementation and outcomes.
7. Establishment of measurable indicators to measure the principles and issues mentioned described by renal dialysis plan.

2. Stages for conduction of evaluation

For the execution of evaluation plan following strategy has to be implemented (Health System Intelligence Project 2008)

1. Define the purpose and scope of evaluation.
2. Tools for data collection like survey, face to face interviews, observation, case study, multimedia (Photograph, videotapes, slides), document review, expert review, journals, annual reports,
3. Data collection (Quantitative as well as Qualitative)
4. Preparation of data for analysis
5. Analysis and interpretation of result so as to summaries the findings and observe for any trends or fluctuation as it is mandatory aspect o evaluation (Program Evaluation Unit 2013).
6. Recommendation or feedback for any improvement to be implemented.
7. Communication of the findings through broachers, network feedback processes, newsletters, conference papers and peer reviewed journals.
8. Evaluate the evaluation.

Cost associated with evaluation

Evaluation of plan is a costly process. A complete and adequate allocation of budget is mandatory for the smooth operation of evaluation as well as to assure that the evaluation procedure considered is fully funded and deliver the intended outcome (Centre for Disease Control and Prevention 2011). Evaluability assessment is an effective tool to control the cost associated with evaluation process. Moreover, evaluability assessment is a method or procedure that is cost effective under numerous circumstances to improve the theme of NSW Renal Dialysis plan and further reduces the cost associated with costly evaluation procedure. In addition, evaluability assessment is cost effective as it sharpens the aim of this plan by generating the underlying logic model and further investigating whether strategies implemented and resources are adequate and suitable for the desirable outcome (Leviton et al. 2010). Generally there are two types of cost; cost required throughout planning phase and cost required for planning. Cost required for planning NSW Renal Dialysis Plan has been well allocated by the planners.

Conclusion

Evaluation of a plan is essential to identify pros and cons of particular plan. On the other hand effective, organized and well planned evaluation procedure is crucial for the proper evaluation of plan. In addition understanding the expenses while performing an evaluation, it is mandatory to employ evaluability assessment of plan prior to actual evaluation of plan. This paper in general describes the importance of evaluation, evaluation steps to be included in evaluation cycle and method of evaluation along with cost associated with evaluation.

Reference:

Agency for Clinical Innovation 2013,’Understanding Program Evaluation. An ACI Framework’, Chatswood, NSW

ACT Government 2010,’Evaluation Policy and Guideline,’ Australian Canberra territory, Canberra.

Centers for Disease Control and Prevention 2011,’Developing and Effective Evaluation Plan, Atlanta, Georgia.

Davis, G. & Bridgman 2004,’Australian Policy Handbook,’ 3^rd edn, Allen & Unwin, Sydney.

Health System Intelligence Project 2008, The Health Planner’s Toolkit Module 6: Evaluation in Ministry of Health and Long-term care (edn.), Canadian Government, Ontario.

JJEC 2003,’Evaluability Assessment: Examining the Readinessof a Program for Evaluation’, Washington D.C.

Leviton, L.C., Khan, L.K., Rog, D., Dawkins, N. & Cotton, D. 2010, ‘Evaluability Assessment to Improve Public Health Policies, Programs, and Practices’, Annual Review of Public Health, vol. 31, pp. 213-33.

Program Evaluation Unit 2015,’Evaluation Guideline’, Department of Treasury,Government of Western Australia, Perth.

Wall, J.E. 2005,,Program Evaluation Mode 9-Step Proces’,Sage Solution.

Page | 1