Assessment Strategies: Paramedic Education

Paramedics have, historically, been algorithmically led through all clinical procedures and decisions which do not directly centre on clinical decision making. It has been indicated that this system of paramedic training concentrated on life threatening conditions with protocol driven practice, based on limited underpinning knowledge (Williams, 2002). They have undergone ‘front-loaded’, fit-for purpose courses which has suited the operational need of the Ambulance Service. Over the last ten years United Kingdom (UK) ambulance services have become increasingly aware that there is a need to evaluate their educational provision. It became evident that paramedics needed to move on from surface to deep thinking strategies and develop a greater thinking repertoire. The paramedic profession is currently at a major crossroads in its development with the transition from a training paradigm to one of higher education (British Paramedic Association, 2006). This evolution in paramedic education heralds a shift away from a training ethos of ‘surface learning’ and expository teaching, best described by Ausebel (1968) as the ‘presentation of the entire content of what is to be learned in its final form’.

This commentary will discuss the current assessment strategies used within paramedic education by higher education institutes (HEIs) including the assessments implemented in the clinical practicum. The various methods and delivery of assessment will be explored including, formative, summative and feedback. My own personal experiences and involvement with the assessment of paramedic students will also be discussed. This discussion will attempt to highlight the validity and reliability of certain assessment strategies such as objective structured clinical examinations (OSCE), portfolios, work-based assessment. The concept of role models for paramedic students during their time on clinical placement will also be explored.

In the UK, the Institute for Healthcare Development (IHCD) is the awarding body for paramedic qualifications and have been involved in prescribing the curriculum, content, and assessments for all paramedic training throughout the UK. IHCD produce multiple choice questionnaires, short answer and clinical assessment tools which have been seriously challenged by the British Paramedic Association (BPA) which is the professional body overseeing paramedic practice and education. The assessment strategies employed by IHCD have been criticised for their errors of format and content relevance (Cooper, 2005). They were also seen as a ‘tick-box’ exercise for the employers’ legal liability.

IHCD assessment processes were criticised for their validity, reliability, and feasibility, with the BPA highlighting a clear need to identify alternative methods of assessment such as objective structured clinical examinations with links to competency frameworks (British Paramedic Association, 2006). The use of a variety of different assessment methods has now become a characteristic of paramedic education within HEIs. Currently HEIs employ a plethora of assessment methods including simulation, standardised patients, written examinations, oral examinations (viva voce) and reflective portfolios. During their time in the clinical practicum paramedic students are also continually assessed with regard to their clinical competencies.

Pugsley and McCrorie (2007) state the need to have valid, reliable, fair and defensible assessments due to increased litigation from students are adding new dimensions to educator roles. With the ongoing change within paramedic education there has been a need to move away from standard assessment methods such as written examinations. Historically, educators have used the same assessment methods for all of the competencies of a paramedic, even when they were ill-suited to the task. For example, it is critical for a paramedic to be able to communicate effectively with patients but an assessment of this aspect of competence is not tested well by written examinations or a viva voce in which the student-patient encounter is unobserved. To correct this problem, several methods of assessments which are new to paramedic education are being implemented with ongoing development. These new methods focus on clinical skills, communication skills, procedural skills and professionalism. An important role of the teacher relates to assessing student’s competence. This hold great relevance within pre-hospital environment as life, health and welfare of patients is potentially at risk if students are not accurately assessed which may lead to future paramedics with low levels of clinical competence.

Stuart (2007) states that assessment can be formative and summative with Boud (2000), claiming that both forms of assessment influence learning. Formative assessment can guide future learning, promotes reflection and provide reassurance (Epstein, 2007) whereas summative assessment tends to take place at the end of module or program of study and is used to provide information about how much students have learned and to what extent learning outcomes have been met (Stuart, 2007). Many of the assessment strategies can be used as either a method of formative and summative assessment (see table 1).

Any particular method of assessment will have its strengths and its intrinsic flaws. The use of several different assessment methods may partially compensate for the intrinsic flaws in any one method (Epstein, 2007). Validity and reliability are deemed as critical for determining the usefulness of a particular method of assessment (Van der Vleuten, 1996). Validity is the extent to which the assessment measures what it was designed to measure (Quinn & Hughes, 2007). Reliability indicates the consistency with which an assessment method measures what it is designed to measure (Messick, 1989).

OSCE

The teaching and assessment of clinical skill proficiency is a major part of paramedic education. Paramedic degree programs throughout the UK are using and developing OSCEs as an approach for the assessment of clinical skill performance. OSCEs have been utilised in advanced life support, trauma, medical condition scenarios as well as individual clinical skills. Newble (2004) states that OSCEs are better suited to assessing clinical and practical skills, often with a high degree of fidelity. Stuart (2007) warns that OSCEs have limitations as a method of assessment due to the performance of students under laboratory conditions may not accurately reflect their real performance in the clinical practicum. Other limitations also exist such as students finding OSCEs highly stressful (Phillips, Schostak & Tyler, 2000; Brand & Schoonhein-Klein, 2009). An OSCE also tends to focus more on the assessment of basic skills rather than cognitive skills (Redfern, Norman, Calman, Watson & Murrels, 2002). A major limitation is the compartmentalisation of the clinical patient assessment process and students may not learn to holistically assess patients (Stuart, 2007).

There is conflicting evidence throughout the literature regarding the reliability and validity of OSCEs as a method of assessment. Goaverts, Schuwirth, Pin et al. (2001) have reported high reliability and validity for this assessment strategy, whereas Phillips, Schostak and Tyler (2000) claim that OSCEs are seriously flawed due to lack of inter-assessor and intra-assessor reliability. Hodges (2003) points out that OSCEs are a ‘social drama’ with students modifying their own behaviour in an attempt to convey the impression that the assessor desires. There are measures that have been reported to increase the validity and reliability of the use of OSCEs and these are listed in Appendix X.

I have had some experience of assisting with OSCE assessments and have found that almost no training has been provided for the assessors which in my opinion, made them unreliable. The assessors are given objective checklists but a personalised form of scoring was sometimes used due to the lack of assessor experience or understanding of the assessment strategy. One of the OSCEs required an assessment of a student’s ability to perform advanced life support (ALS) with the checklist containing a total of 111 points to be assessed (see appendix X). This created problem with the reliability of this particular OSCE as the assessors found it difficult to observe the student as well as check they were performing everything on the checklist. This OSCE lasted almost 30 minutes which not only caused fatigue to the student but also to the assessor. According to Reznick et al. (1992) the heavy workload on the assessor can affect their performance. A lengthy OSCE can also cause student fatigue which may affect their performance (Rutala, Witzke, Leko, Fulginiti & Taylor, 1990).

As part of my ongoing development as a teacher I have started to design an OSCE for the assessment of some basic clinical skills. Some of these have already been designed for other healthcare professions but none are paramedic education specific.

Simulation – ALS

Simulations are increasingly being used in paramedic education to ensure that students can demonstrate integration of prerequisite knowledge, skills, and affect in a realistic setting. For these aspects of competence, the use ‘Sim-Man’ which is a physical simulator that models the human body with a very high fidelity is currently used extensively at my employing university. This form of simulator is very realistic and can provide an excellent assessment of skills. These simulators are used in conjunction with observed structured clinical examinations (OSCE).

Standardised Patients

A standardised patient is a person trained to portray a patient with a particular medical or traumatic condition. They are used to assess a students’ ability to collect medical history and physical examination data (Tamblyn & Barrows, 1999). Following the encounter between the standardised patient and a student, both the standardised patient and an assessor make judgements about the students’ performance based on history taking, physical examination and communication skills. Checklists are developed for each patient scenario focusing on the students’ ability to collect the relevant data. Skills in summarising and interpreting the information collected are often measured following the encounter consisting of open-ended questions or short answers. I have found this to be the most realistic form of assessment outside of the clinical practicum and research has shown that reasonable levels of reliability and validity can be achieved (Norcini & McKinley, 2007). However, McKinley and Boulet (2004) warn that there is a drift in assessor stringency over time and that the standardised patients themselves become increasingly inconsistent portraying a patient. This assessment strategy is very expensive to use and has been implemented infrequently. As part of my role as an associate lecturer I have had the opportunity to take part in this form of assessment. The students were required to perform a cardiac assessment on a standardised patient. Prior to this summative assessment they were given a lecture on cardiac assessment, a demonstration of the assessment and many hours practice time with support from the lecturing team which included myself. A checklist (see appendix X), a viva-voce question sheet (see appendix X) and a short answer paper (see appendix X) were all designed. Following the patient encounter the standardised patient and I went through the checklist to ascertain the ability of the student. The viva-voce followed where open-ended questions (see appendix X) were asked to allow the student to discuss their findings and plans for treatment which was followed by the short answer paper (see appendix X).

Work-based Assessment

Work based assessment of learners in the clinical cenvironment

Assessment plays a major role in the process of medical education, in the lives of medical students, and in society by certifying competent physicians who can take care of the public. Society has the right to know that physicians who graduate from medical school and subsequent residency training programmes are competent and can practise their profession in a compassionate and skilful manner (Shumway & Harden 2003). Miller (1990) proposed his now famous pyramid for assessment of learners’ clinical competence (Figure 2).

Knowledge tested by written exams

Application of knowledge tested by clinical problem solving etc.

Demonstration of clinical skills, tested by OSCEs, clinical exams, competency testing (Competency)

Daily patient care assessed by direct observation in the clinical setting (Performance)

Adapted from Miller (1990)

At the lowest level of the pyramid is knowledge (knows), followed by competence (knows how), performance (shows how), and action (does) The clinical environment is the only venue where the highest level of the pyramid can be regularly assessed.

Assessment in the workplace is quite challenging as patient care takes top priority and teachers have to observe firsthand what the learners do in their interaction with patients and yet be vigilant that patient care is of the highest quality.

Paramedic students need to retain vast amounts of information, integrate critical thinking skills as well as having an ability to solve a range of complex clinical problems often under intense pressure. In an effort to enable this I have started to acknowledge the need for alternative teaching, learning and assessment strategies. Educational literature supports the use of concept mapping as a means to promote learning and is seen as a metacognitive strategy (Novak, 1990; Pinto & Zeitz, 1997; Irvine, 1995). I have explored the idea of concept mapping and have now started to use it within my teaching practice. I view the goal of concept mapping as a way to foster learning in a meaningful way, to enable feedback and to conduct learning evaluation and assessment. As a learning resource, concept maps can facilitate a students’ understanding of the integration and organisation of important clinical concepts (Pinto & Zeitz, 1997). A group of the paramedic students I currently teach found that concept mapping encouraged them to think independently, promoted self-confidence and provided them with a greater awareness of connecting across different areas of knowledge. From my own perspective I view concept mapping as a creative activity which assists the students in becoming active learners. It is also seen to allow students to organise theoretical knowledge in an integrative way as well as fostering reflection into the learners’ understanding (Harpaz et al., 2004). During and following the use of concept mapping sessions numerous opportunities arose to identify student misunderstandings allowing provision of relevant feedback and clarification of content. According to Kinchin and Hay (2000) the use of concept mapping highlights the learners ‘constructions of connections’ as well as a useful form of communication between the teacher and learner. As an assessment strategy concept maps can be used either formatively or summatively (West, Park, Pomeroy & Sandoval, 2002) by identifying the learners’ valid or invalid thoughts and links. However, Roberts (1999) warns that a suitable scoring method must be selected for each particular type of map due to their great variation in style.

I have used concept mapping with individual students and small groups of students and it has been effective when dealing with the differential diagnosis of certain conditions. For example, the causes of chest pain are numerous but it is important for a paramedic student to be able to understand and differentiate between the different causative factors. A lesson was planned, delivered with handouts provided (see appendix X). Over the next two weeks the students were formatively assessed individually by using concept maps to illustrate their understanding of the causes of chest pain. A decision was made to use a relaxed approach in an effort to overcome any anxiety so the assessment became an informal discussion. As the students progressed through the task I was able to gain insight into what each student knows and how they arrange knowledge in their own minds. I was able to give feedback on their misunderstandings, misconceptions and errors. Paramedic educators rarely use concept mapping as a teaching or learning method but I see them as valuable tool in revealing students’ misunderstanding of concepts which could in turn lead to the identification of potential clinical errors.

FEEDBACK

Effective feedback may be defined as feedback in which information about previous performance is used to promote positive and desirable development. Giving feedback, whether corrective or reinforcing, is complicated but is an essential part of paramedic education. Feedback plays a central part in the support of cognitive, technical and professional development (Archer, 2010). Cognitive theorists view feedback as a comparison between actual performance and the intended performance level (Locke & Latham, 1990). According to Kluger and DeNisi (1996) this will highlight the gap between a learner’s knowledge and the required knowledge and provide a learning catalyst. Behaviourists conceive feedback as a way of modification or reinforcement of behaviour (Thorndike, 1931). In the health care education literature including paramedic education, feedback seems to lack any form of theoretical basis (Colthart, Bagnall, Evans, Allbutt, Haig et al., 2008).

Paramedic education feedback is often problematic for both the provider and recipient. The diversity of the feedback settings creates multiple challenges for paramedic educators in the provision of effective feedback. Settings such as bedside teaching, practical skills training as well as the often ‘chaotic environment’ have been utilised for the provision of feedback by myself. The protection of the professional standards, the rights and safety of patients as well as the self-esteem of the paramedic student must be safeguarded. Added to this is the acknowledgement of the psychosocial needs of the paramedic students as well as ensuring that the feedback is accurate and honest (Molloy, 2009: Higgs, Richardson & Abrandt Dahlgren, 2004). These challenges are an ongoing issue within paramedic education but despite this feedback has been described as ‘the cornerstone of effective clinical teaching’ (Cantillon & Sargeant, 2008).

An important factor necessary for progress is the provision of feedback. Parsloe and Wray (2000) suggested that feedback is the fuel of improved performance, that it can provide motivation. However, they do warn that motivation can be reversed if you get the feedback wrong. Feedback should be viewed as a positive tool as even negative aspects of feedback actively encourage learners to improve practice; Bennett (2003) affirms that it can also highlight strengths and weaknesses which in turn can lead to enhanced practice.

There are many types of feedback used to support learners and these can have either a directive or facilitative function. Directive feedback can inform a learner of what requires some type of corrective measure whereas facilitative feedback can involve the provision of suggestions to facilitate learners in their own revision (Archer, 2010). Feedback specificity may also be variable. Specific feedback can sometimes be beneficial for an initial change in performance but it may have a negative effect on the learner’s ability to further explore their performance which can lead to an undermining of subsequent performance and learning in the long term (Goodman, Wood & Hendrix, 2004). Less specific feedback can lead to uncertainty for the learner that may subsequently lead to a reduction in learning (Kluger & DeNisi, 1996). Verification and elaborative feedback are structural variances of directive and facilitative types of feedback. Feedback can simply indicate that an answer is correct (verification) or it may facilitate the learner to arrive at the correct answer (elaboration). According to Bangert-Drowns, Kulik, Kulik and Morgan (1991) the guiding principles of feedback are that it should be specific, with verifying and elaborative feedback enhancing effectiveness.

I have attempted to utilise a number of types of feedback for my learners since the start of my current studies on the Post Graduate Certificate in Education (PGCE). I will now go on to discuss some of the methods of feedback utilised for the different learners I am currently supporting. I have utilised many of the types of feedback interchangeably and have slowly gained a conscious knowledge of their appropriateness. Throughout their placements paramedic students have regularly received constructive feedback which is usually delivered contemporaneously which is something that Price (2005) recommends. A large proportion of the feedback has been delivered informally and is often carried out in between emergency calls and has often been given as a running commentary during student/patient interaction.

Non-verbal communication is seen as a strong method of conveying meaning and is often utilised in providing feedback (Stuart, 2007). Non-verbal communication serves several functions which can be summarised as giving and receiving information, expressing emotions, communicating interpersonal attitudes (e.g. warmth, dominance and liking) and establishing relationships (Williams, 1997). Non-verbal communication requires acute observation by the receiver for accurate interpretation of the message (Golub, 1994). Non-verbal communication includes specific behaviours that include proximity, touch, eye-contact and eye gaze, facial expression, body posture and head movements. I have used this form of communication as a method of feedback during bedside teaching sessions and when the student is involved within patient care episodes. A ‘nod’ and a ‘look’ of approval are seen as positive feedback which all ads to the feedback process.

More formal feedback is given at the end of each shift. The timing and format of these feedback sessions are of great significance. For feedback to have maximal motivational impact on learning, it should take place while it is still relevant and points raised are therefore more meaningful and alive (Hays, 2006). During these feedback sessions the format is a blend of oral and written. A copy of the written element is given to the student to place in their portfolio. Fish and Twain (1997) believe written notes are essential in providing continuity in the monitoring of progress.

Initially, the students were asked to provide a self-assessment of how they felt the shift had gone. This was an addition to the feedback cycle and encouraged learners to be more self-aware and more self-critical but gives an insight into how the student feels about their progress (Mohanna, Wall & Chambers, 2004). Rather than a one-way process the feedback becomes collaborative. On further exploration of the literature I became aware that involving the learner in comparative self-assessment that places them at the centre of the feedback process will not always improve competence. Paramedic students are frequently required to self-assess their performance in an attempt to identify their own strengths and weaknesses in relation to other people’s views.

Written feedback is important for learners as well as teachers. As an on-line tutor for a local HEI I provide support and feedback to paramedic science students via email. These students send me there work in progress and constructive feedback is given (see appendix X). Any negative comments made are supported to produce positive outcomes which is something that Webb (1955) advocates. According to Archer (2010), lengthy and complex feedback maybe ignored with its main messages lost. In effort to counteract this problem some of my written feedback has been deliberately left concise.

In conjunction with verbal feedback students on clinical placement are provided with daily written feedback on patient encounters (see appendix X) which are supported by further written feedback monthly, quarterly and at the end of the clinical placement (see appendix X).

I consider the use of feedback on student progress generated from other sources such as other paramedics and assesors as highly important as part of a systematic approach. Feedback from a variety of sources is described as multi-source feedback (Archer, 2007). Paramedic students on clinical placement are sometimes rostered to work with other Clinical Supervisors or paramedics. Early in a paramedic student’s placement a critical development issue was identified by one of my colleagues who had completed a set of shifts with the student. The unacceptably slow pace of the student’s clinical practice including patient assessment, patient management and long turnaround times at hospital following incidents had been fedback to me. As a student’s practice placement progresses, evidence of care activities should be gathered showing increasing speed and dexterity with increasing clinical experience (Stuart, 2007). Early support is vital in preventing the student with a learning issue from experiencing the ‘cycle of failure’ (Cleland, Arnold & Chesser, 2005). Other key personnel who had observed the student in practice were consulted to ensure that the issue was as prevalent and problematic as had initially been assessed. Eliciting the opinions of other assessors also helps provide a more fair and unbiased assessment of a student’s abilities (Gomez, Lobodzinski & Hartwell West, 1998). The issue was then raised with the student whilst they were performing a task too slowly. Stuart (2007) highlighted the critical importance of the critical role of providing feedback in learning and support. Furthermore, providing feedback of a development issue whilst it is relevant and ‘alive’ is important for memory recall (Bailley, 1998: Gipps, 1994).

The key to support struggling students however is to identify specific reasons for poor performance (Cleland, Arnold & Chesser, 2005). Furthermore, major feedback which corrects a particular performance deficit should end with a plan of action (Branch & Paranjape, 2002). During a feedback session with the student, input regarding specific areas where time could be saved was requested by the student but a specific action plan had not been prepared. Providing poor quality feedback may make the student feel they are being personally judged and can lead to defensiveness or reactivity (Katz, 2006). In order to rectify this, the student was closely observed in practice for a set of four shifts in order to identify specific areas of slow practice (see appendix X). A three month plan was negotiated with the student to rectify the problem and a ten point action plan developed which identified ten areas where the student could save time on each call attended (see appendix X). Data was collected over a three month period of the student’s on scene times and hospital turnaround times. The student was not informed of this in order to avoid the ‘Hawthorne effect’ when working with other members of staff where a subject’s normal behaviour and/or study outcomes are altered as a result of the subject’s awareness of being under observation (Mangione-Smith. Elliott, McDonald & McGlynn, 2002). Branch and Paranjape (2002) state that evaluation should follow efforts at remedying the learners’ problem. The results of the data after the three month period were therefore presented in chart format to the student (see appendix X) which provided tangible proof that the student had demonstrated real evidence of progress.

Just as paramedic students are interested in their progress, so too should the paramedic educators be interested in the quality of their teaching. The measurement of teaching quality can be a very complex, multi-perspective task that may include many different approaches (Hays, 2006). One form of teaching quality assessment is student feedback. Collecting student feedback also sends messages to students that their views are welcome (Hays, 2006). With this is in mind teaching assessment tools were designed to elicit the student’s opinions on the teaching provided (see appendix X). Other feedback has been provided from peer observations (see appendix X).

Prehospital education is changing rapidly, and as demonstrated the way paramedic education is delivered at clinical practicum level is also evolving. In order to establish quality and stability in this rapidly changing delivery of education, eliciting feedback from students is critical (Jones, Higgs, de Angelis & Prideaux, 2001). A feedback questionnaire was designed to elicit the students’ opinions on the quality of the learning timetable and reference handbook after three months of the placement period. The students indicated a very high satisfaction rating for the timetable and particularly the reference handbook (see appendix X).

Feedback was also gained via a questionnaire regarding general teaching effectiveness in areas such as quality of feedback, supervisor/student relationship, goal management and mentoring activities (see appendix X). Whilst overall the feedback obtained regarding general teaching and mentoring effectiveness was very positive, one of the students indicated some dissatisfaction with the prioritisation of goals. This was discussed further with this student who expressed concerns that it was unclear in reference to the learning timetable which goals were of a higher priority than others as the learning objectives were set out in chronological order.

Patients have also been able to add to the feedback process as they can provide testimony to the student’s performance. This type of feedback is unique to healthcare professionals and is deemed influential in changing performance (Fidler, Lockyer, Toews, & Violato, 1999). However, Archer (2010) warns that feedback elicited from patients may lack validity with Crossley et al. (2008) suggesting that there is limited correlation with other sources of feedback. This form of feedback is often utilised when a student has been driving the ambulance and the patient is asked about the comfort of the journey to hospital, which is naturally carried out in earshot of the student.

Giving feedback

In the clinical environment it is vital to provide feedback to trainees as without feedback their strengths cannot be reinforced nor can their errors be corrected (Ende 1983). It is

a crucial step in the acquisition of clinical skills, but clinical teachers either omit to give feedback altogether or the quality of their feedback does not enlighten the trainees of their

strengths and weaknesses. Omission of feedback can result in adverse consequences, some of which can be long term especially relating to patient care. For effective feedback, teachers need to observe their trainees during their patient interactions and not base their words on hearsay. Feedback can be formal or informal, brief and immediate or long and

scheduled, formative during the course of the rotation or summative at the end of a rotation (Branch & Paranjape 2002).

Reflection on feedback

Role Modelling

Ambulance services in the UK are slowly becoming reliant on undergraduate paramedic education programs to meet their employment needs (JRCALC, 2000). The HEI’s and ambulance services have now forged strong links and work closely with one another in the education of student paramedics with the ambulance services providing the clinical practice placements. The clinical practicum is an important component of a student paramedic’s development. This practicum seeks to integrate theory and practice as well as enable the development and assessment of professional competencies. During this time in practice the student will have many opportunities to develop skills and to refine these skills based on performance feedback by clinical su