Assessment and process of medical education

Assessment plays an important role in the process of medical education as it is an effective tool which detect quality in students training to motivate and direct them to what they must learn(1). “Assessment drives learning” this statement focus on the essential role of assessment as well planned and implemented assessment has an important steering effect on learning because it transfers what is important to learn and motivate students for learning(2). Many people argued that as the curriculum should be the key which motivate learning while assessment should be designed to be sure that learning outcomes have occurred, So assessment tool must has clarity of the learning purpose and must be designed to drive educational intent and maximize learning(3).

Constructive alignment is an important influential idea in which the students construct meaning from related learning activities and teachers apply learning environment which support planned learning activities to achieve the intended learning outcomes(4). So constructive alignment makes the teaching systems consistent when curriculum, learning activities and assessment methods are aligned with intended learning outcomes(5) . Moreover, assessment may reveal learning outcome which isn’t expected but it is recognized as important outcome, so it must be integrated into the intended learning outcome as emergent outcome(6).

Formative assessment promotes deeper learning as it provides students with feedback to encourage them to know their strength and weakness which reinforce students internal motivation to learn and improve their knowledge and skills(7). Summative assessment is a final assessment which determine the rank-order students and decide grades(1). Wass et al(7) argued superficial learning which aim mainly on passing the examination and they emphasized on the importance of feedback on students assessment which encourage student reflection and deep learning. However, Epstein(8) showed that summative assessment influence learning even in the absence of feedback as students study what they expect to be tested on. Although formative and summative assessment are stark in contrast, they are both necessary and distinction between them should be made to detect which assessment is suitable only for formative use or have sufficient rigorous for summative use(7). Van der Vleuten and Schuwirth(9) emphasized that formative and summative assessment can be used with little difference with focusing on the development of comprehensive assessment programme in which both encourage learning and right decision about learners.

I will focus my writing on written assessment as I am involved in assessing written examination of MSc of Radiology scince 5 years. According to Miller pyramid we use written assessment to assess the domain of cognition, either factual recall of knowledge “knows” or application of knowledge and problem solving “knows how”. We use written assessment in the form of essays and multiple choice questions in formative assessment of the residents and in summative assessment of final exam. Our final written exam formed of two papers of essays, each one formed of four essay questions with three hours duration for each, and third paper of 20 multiple choice questions with one hour duration. When we prepare a written exam we identify the level of residents training to apply test which assess knowledge appropriate to students experience.

Essay questions are effective method for assessing cognitive skills as they can assess ability of students to form answer and measure their attitude and opinions, also they can give students effective feedback on their learning(10,11). But it has the disadvantage of being time-consuming test to grade and its test doesn’t cover a wide domain.

Newble and Cannon(11) stated that essay is either extended response questions which are useful in assessing higher cognitive skills like analysis, synthesis , problem solving, and restricted response questions used for testing knowledge of lower level but it has the advantage of being more reliable as scoring variation can decreased with it. Epstein(8) stated that well structured essay with clear framework can eliminate cueing and maintain more cognitive process with context rich answers. We usually used extended response questions by which we assess students’ higher level of knowledge, but I think for improving essay test utility, we must make mix of the two essays types with using clear words on constructing questions like using describe, criticize and compare instead of discuss to direct students to desired answer, as I find some poor structured essay questions in our exam, for example “discuss radiological imaging of breast mass” which I can change it to be “compare between ultrasound and mammography for differentiating breast mass”.

Van der vleuten(12) stated five criteria to assess assessment tool utility which are “reliability, validity, educational impact, acceptability and cost effectiveness”. Reliability measures consistency of the assessment test and it is often described as reliability per hour of testing time as time is a limiting factor during exam, so essays are low reliable than MCQ because it require longer time to answer(13). Schuwrith and Van der Vleuten(14) stated that inter-case correlation of different essays in certain test is low as the essays numbers which can be asked in a certain test is limited.

Chase(15) stated that essay scoring is a complex process as it has many variables which are essay content, writer, rater and other colleague variability with their significant writing effect. The most important type of reliability for rater-type assessment is inter-rater reliability, single inter-rater reliability (which mean correlation between two raters) ranges from 0.3 to 0.8 as this depend on topic of the essay, the essay length, the rater experience and the level of rater training(16). But Munro(17) et al stated that single inter-rater reliability can be regularly obtained as 0.7 if there is continuous extensive rater training. On agreement with those authors about increasing inter-rater reliability we already use double markers for assessing essays question and the mean of their score is calculated to be the end score.

Essays are poor objective test for assessing learning outcomes as there is variability in the assessment scores through different examiner with variation of perfect answer (18,19). Norman et al(20) stated that providing structured marking of the essay may improve its reliability but it may cause process trivialization. Schuwrith and Van der Vleuten(14) emphasized that using one marker for each essay for all students is more reliable than one marker for all essays of the same student. Davis(18) stated that using double marking for the same question is mandatory to reduce variation incidence between the markers. Beattie and James(21) suggested using checklist in marking essay to reduce subjectivity and improve objectivity of essay as it provides the examiner with key point of each item and its allocated marks. As mentioned before, double markers are applied in our radiology department for assessing each question but we don’t use checklist in marking the essay question, so I think this make our examination less reliable with poor objectivity and we have to use checklist with specific marks on each part of the question.

Validity is the ability of assessing method to measure what is purported to(19). The valid method will reflect what the students achieved from intended learning objectives of the course, so increasing the test item is essential for more valid test, therefore the validity of the essays is limited(6). Brown(22) advises using large numbers of short essays to improve its reliability and validity and to reduce sampling errors. However, Davis(18) argued that as this may cause more time consuming to mark. As we begin to apply a test blueprint to determine the main content of the test which must have high content validity to cover intended learning objectives, we have to use larger number of shorter essays to be eight to ten short essays instead of four long essays according to test blueprint. Van der Vleuten(23) stated that assessment methods should have content validity which must be designed and mapped on a blueprint.

Modified essay questions was initially produced by the Royal College of General Practitioners in London and are widely used now(11). Davis(18) stated the importance of using context rich scenario which will direct the students to answer with precise data and increase exam reality. Schuwirth and Van der Vleuten(14) showed that written case-simulation essay appeared to be more valid as its questions focus on history taking, diagnosis, investigation and examination findings which are closely related to real practice. However Swanson et al (24) argued that as these essays aren’t suitable for assessing problem solving questions. Newble and Cannon(11) showed that certain skills is needed for constructing modified essay questions to avoid giving idea about answers of a previous question or punshing the student on question constructing error. Also, Schuwirth and Van der Vleuten(13) emphasized that considerable structure of essay question is necessary but over-structuring may lead to limited increase in its reliability, As we use essays in both formative and summative assessment we have to use modified essays instead of traditional essays especially in resident formative assessment as we returned it to students with its model answer for discussing during the tutorial, as this will encourage student’ critical thinking and reflection, but also we must take training about constructing modified essay questions to avoid poor form which may cause assessment error.

Schuwirth and Van der Vleuten(13) advised using essays in limited occasions when objective tests are not suitable, Objective written tests like short answer question, matching exercise and multiple choice questions ( MCQ) have the advantage of being economic, rapidly scoring, high reliable and evaluate the student in large content(25).

There are two major format of MCQ which are True/False format and single best answer. True/False format can cover a broad amount of the topics and are easily marked but they mainly measure definitions and simple facts(26). Case and Swanson(27) explained why using True/False format is markedly reduced as it is not only difficult to construct but it mainly used to assess recalling of isolated fact to avoid ambiguous items, also they can’t detect if the student who identify correctly the false statement knows the right answer or not. Another disadvantage of true/false format is their high probability of guessing(28). To overcome guessing, negative marking was achieved in which there is deducing marks for the wrong answer, but these may produce negative psychometric results(25). We sometimes use true/false format instead of single best answer, as we think it covers a broad items in the curriculum and can measure complex outcomes but we don’t apply negative marking for MCQ correction as we think that is stressful to the students, also I have bad memory about using negative marking when I was medical student at 2nd year I got 19/50 in physiology MCQ test and this caused to me poor willingness to MCQ risk. When, I read carefully a previous exam of True/False format, unfortunately I find some unambiguous questions which may cause a critical failure for these questions. So I think we must limit using these types only for assessing definitions and facts identifications and apply other types of objective tests to avoid the guessing probability of true/false format. This is in agreement with Schuwrith et al(13) who stated that True/false questions are only suitable when the question purpose is to evaluate if the student is able to determine the correctness of hypothesis.

MCQ are able to evaluate broad range of learning outcomes within a short time and limited human intervention, also they have low guessing probability with free question of ambiguity(29). In the tutorial of decembrer 2010, there is a debate about effect of MCQ guessing on test reliability, but I learn from the discussion an interesting concept which emphasized that guessing doesn’t change test reliability as good student is a good guesser.

For constructing good MCQ items it is essential to have a good idea about the content, study the objective of the assessment and apply high quality form for items writing(27). MCQ consist of stem and several options, stem is formed of sentence or question and may be accompanied by diagrams or tables, while the correct option is defined as “keyed response” and the wrong options are called “distracters”(29,30).

Case and Swanson(27) stated that MCQ must be well structured to be simple, easily understood with using plausible distracters, also grammatical errors especially using negative and inaccurate words like ” never, sometimes, frequently and usually” should be avoided as they may lead to examinees confusions(31). Lowe(32) stated that the useful distracters should demonstrate a misconception between the students about the right option, so writing many plausible distracters is a difficult part for MCQ construction with more time consuming. The flaws of writing distracters which include using more than correct answer, using “all of the above” or “none of the above”, or making the right option is the longest one should be avoided(33). MCQ reliability increase with removing non plausible distraction(34,35). Although we choose MCQ from question banks or MCQ books to reduce the examination preparation time , unfortunately I find many drawbacks in our last MCQ exam, firstly one question contains double negatives, also in another question I find it was easily to eliminate some distracters, while other questions contain inaccurate words which are sometimes and always. So I think we must take care during choosing MCQ distracters which should appear to the students as a valid answer while it is incorrect, also we must avoid apparent incorrect or plain distracters. So, we need to take training courses for MCQ preparation and writing MCQ stems and distracters to avoid MCQ flaws and constructing good items.

Collins(30) showed that MCQ have the disadvantage of being test knowledge recognition rather than constructing answer. Mcaleer(31) argued that as MCQ are an objective test which doesn’t allow students the chance for giving additional information and doesn’t apply examiner to put judgment on student answer quality. I agree with Mcaleer(31) as we use MCQ as an objective test to assess understanding knowledge of a broad range of learning objectives within a short time.

Reliability is refered to reproducibility of the assessment score and it is expressed as a coefficient which range from 1 for perfect reliability and 0 for no reliability. MCQ are widely used due its high reliability which is attributed to its ability to assess broad amount of knowledge by providing large number of items which address areas of context specificity within a short time(7,30). Downing(36) stated that written test especially MCQ has high internal consistency reliability as the test score would be near the same if exam is repeated at later time. Van der vleuten and Schuwirth(9) showed that the predominant factor which affect reliability is domain as competence depend on context specificity. While McCoubrie(25) argued that and he stated that the assumption of MCQ as a reliable test is weak as they are only reliable because they maintain a time efficient test with wild sampling of topics. Van der vleuten and Schuwirth(9) stated that the reliability of MCQ test in one hour is 0.62 which is increased to 0.93 for four hours test due to using more items number. Wass etal(37) stated that for important exam in which stakes are high a high reliability of 0.8 or more is essential to determine pass-fail decision but for formative assessment lower reliability can be accepted. Our final MCQ exam contain 20 questions with examination time of one hour, s has low reliability due to small number of items within short time which miss many objectives of our curriculum, So I think we have to increase the question numbers to cover more knowledge of context specificity and consequentially increase the test time to improve the test reliability.

A criticism of MCQ validity as it measures the factual knowledge and doesn’t integrate skills, attitude and communication skills(25). Downing and Yudkowsky(38) emphasized that knowledge is the single best domain which determine expertise, so MCQ is a valid competence method which assess cognitive knowledge. Collins(30) stated that MCQ have a high validity if it represents a wide sample of content that serve the objective learning outcomes. However, Shumway and Harden(1) critic that as MCQ asses discrete superficial knowledge not deep understanding as they designed to detect what students know or don’t know.

Blooms taxonomy of educational objectives is a hierarchy of knowledge for different cognitive level which are “knowledge, comprehension, application, analysis, synthesis and evaluation”(39). While educators simplified Bloom’s taxonomy into three levels which are knowledge recalling, comprehension and analysis, and problem solving(11). Case and Swanson(27) and Mcaleer(31) showed that well-structured MCQ can assess the taxonomic higher cognitive process like interpretation, application and analysis rather than assessing recalling of facts. Peitzman et al(40) argued that as they stated using higher-order MCQ doesn’t improve MCQ validity but it makes them more real and acceptable to students and examiner. Also, Frederiksen(41) stated it is difficult to construct MCQ with rich context as item writers tend to escape from topics which can’t be easily asked. In agreement with Case and Swanson(27) and Mcaleer(31), we try to choose MCQ level with different cognitive level, and when I revise our MCQ tests I find some questions which can assess recalling of knowledge(Q*) and other assess problem solving(Q**) for the same topic, example of this is:

Q*:what is the effective measure which reduce radiation of CT chest?

a-120 mA

b-150 mA

c-200 mA

d-250 mA

Q**:what of the following will reduce dose of radiation for CT chest?

a-reducing mA from 250 to 150

b- reducing KVp from 160 to 120

c-reducing the pitch to be 1 instead of 2

d-reducing scanning time to be 1 instead of 2

Blueprint is an important powerful tool for integrated curriculum as it maintain assessing all its intended learning objectives.(42). Our faculty assessment centre members work in progress and they make many orientation about blueprint construction and its importance, also they asked all departments to finish their blueprint, but until now we evaluate our exams retrograde according to our ILOs, but unfortunately in some written exam we found that the items don’t cover all topics of the curriculum and missed many ILOs, also in other written exam we find a focus on certain system rather than other systems which may produce bias of examination results as the questions sample doesn’t represent a big domain of knowledge. So, I think we are urgently in need to use test blueprint which cover the learning objectives and assessing methods to identify the key topics which must be tested according to our objectives and determine the questions numbers according to their corresponding weight in the context. This is in agreement with Downing and Haladyna(43) who stated that blueprint reduce two validity threats which are under-sampling bias of the curriculum and constructing irrelevant items.

Consequential validity is referred to the real impact of an assessment method on learning which appropriately drive students’ learning(25). Wass et al(7) stated that consequential validity refers to the educational consequence of the test as it produce the desired educational outcomes, which means that students should study subject rather than studying the test. Although consequential validity is an important process, it is ignored by many examiners(44). I think our written exam has significant educational impact on how our students study, as from my experience students study what they need to pass rather than studying the whole integrated information. To improve this, we have to use different forms of written assessment which must cover the important content of the curriculum, and it should be mixed with continuous formative assessment and feedback to steer our students to determine what they study and how they learn. This is in agreement with Van der Vleuten(12) who stated that assessment can drive learning through four ways: assessment content and structure, the question which asked and the frequency of repeated examination.

Newble and cannon(11) advice using computerized optical mark reader to score and analyze MCQ tests as the computer programmed has the advantage of applying statistical data of the test which include reliability coefficient, standard deviation and test item analysis . In our exam we use a hand marking sheet of answers to correct MCQ. But recently our faculty bought a new computer machine for correcting the MCQ test, so we need to learn how to use it for interpreting the test information as these may help us to improve next exam.

Shumway and Harden(1) emphasized that practicability of an assessment method depends on resources, expertise availability and their costs. Resource intensiveness is determined by cost of constructing and correcting the test items(45). Cost includes beginning and continuing resources which are needed for test implantation(1). Essay questions appear to be easily constructed items but specific answer key is needed which may cause more time-consuming for preparation(18). MCQ seem to be easy to grade especially with using computer machine but for good structured items more time is needed for construction(30). Shumway and Harden(1) stated that it is important to consider the relation between the assessment method cost and its benefit. Van der Vleuten(12) critics that as he considered investment in an assessment methods is an investment in teaching and learning process. I think we must take care about the criteria of each method and balanced them against each other as the outcome may change according to the assessment context specificity. Also, In agreement with Van der Vleuten, I think we must use different assessment tools especially for summative assessment for high stakes exam to obtain more reliable and valid assessment.

Schuwirth et al(45) explained that students can answer correct MCQ by detecting the right answer but they aren’t able to answer it in the absence of MCQ options. Graber et al(46) explained the problematic effect of MCQ cueing which may cause diagnostic errors especially if diagnostic reasoning is assessed. Schuwirth et al(14) advise using extended matching items and short-answer question as they can reduce the cueing effect.

Extended matching questions (EMQs) are good authentic test as they use real clinical scenario which need sufficient clinical knowledge and can test a wide range of topics for knowledge application and problem-solving ability like diagnosis, investigation and management(47). Beullens et al(48) emphasized that EMQs are able to assess extended learning and minimize recognition effect rather than memorizing facts which is needed for MCQ solving. McAleer(31) critics that as EMQs with its many different items and long list of suitable answers are difficult to construct. However, Schuwrith and Van der Vleuten(13) advice using EMQs as they are good reliable test with short time scoring. We don’t have experience in EMQs, but after knowing its importance and its significant role for improving written assessment reliability, I think before applying this form we need training of how construct these questions and how practice them to avoid bad representation of some items.

Short answer questions is an important assessing tool because they are objectively scored test as they need clear sets of answer with little guessing incidence(3). McAleer(31) critics that as he stated, although short answer questions are easy constructed item, it is used only to measure recalling of information as they can’t measure complex learning outcomes like synthesis and information analysis. Epstein(8) stated that short answer questions can be used for summative and formative assessment but its reliability depend on mainly training the students how they answer these items. We don’t apply short answer questions in our exam, but I think we can use in certain situation when we want to cover broad area of content and be sure that the students are able to supply an answer rather than choosing it from many options.

Score determines the number of correct answers of an assessment but it doesn’t represent the quality of students performance(49). Norcini(50) stated standard setting is the process by which pass mark of exam is determined to distinguish competent from non-competent students as it allows for variation according to the level of test difficulty.

There are two types of standard setting: relative (Norm-refrenced) and absolute (criterion-refrenced) standard, in relative standard setting fixed number of students will pass the exam irrespective to their level of competence as it is related to peer performance and fixed percentage of success(50). In our faculty we use relative standard setting to select students with highest score for admission to postgraduate course when fixed number is determined. In the tutorial of , I gain a new information which is supposed from one of our peer who advice using relative standard setting for choosing lower achiever in formative assessment who need extra-training.

Absolute standard setting is more suitable for competence test as accurate standard should be determined below which the candidate wouldn’t be fit for particular purpose(7). Absolute standard setting may be test-centered method or examinee-centered method, in test-centered method (like Angoff method) the examiner evaluate every item to hypothetically determine how the candidate will get in each item(51). While in examinee-centered method (like contrast group method), panelists decide the pass score by detecting it on the score scale which should be most fit to the exam purpose(52). In our faculty we don’t use any forms of standard setting as we use 60% as an ideal setting for pass/fail decision for all test types, But as we recently apply assessing centre in our faculty, I think we must use standard setting in our assessment, in my opinion I prefer applying modified Angoff method as an example of absolute standard setting as it is widely used in medical assessment and it can be used for many assessment types. This is in agreement of Smee and Black(53) who stated that modified Angoff method reduce the difficulties of traditional Angoff method ,for examlple the difficulty of detecting hypothetical borderline candidates in Angoff method which is facilitated by supplying the examiners with real test scores of previous assessment of the candidates.

Norcini etal(50) stated that absolute standard setting is applied either as conjunctive or compensatory standard, In conjunctive standard the candidate must exceed each item separately to pass the total test, while in compensatory standard the test scoring permit the candidate to compensated poor performance in one item by high performance in another item. In our written assessment we use compensatory method in which the standard is achieved according to total test performance, but now I think we can use conjunctive method in assessing essay paper by which the candidates must pass each essay separately as this will improve their studying to pass in each item.

Case and Swanson(27) stated that many medical schools provide their faculty with item analysis of their test before test results are announced by which a useful information about the quality of each item separately and the whole test quality are obtained. Items analysis will be valuable when it maintains effective feedback to test writers as this will improve their skills in further test construction, also it would be helpful in discarding poor items and detecting certain areas of the content which may need more clarity(30). Item difficulty is detected from the proportion of students who answered each item correctly, Items are considered difficult if 50% of students or less answered them correctly and low difficulty if 85% or above of students answer the item, while moderate difficulty which have 60-80% discriminating index are the most discriminating items(30). In the tutorial of December 2010, I gain an important information about the value of applying difficult items in the exam as these will encourage students towards excellent and to study to get more marks, so I think we must apply certain percentage of difficult items in the exam to drive learning of our students.

Item discrimination is determined by the difference of the percentage of correct response between two students group (top third and lower third) with discrimination ratio lie between +1 and -1 and acceptable index is in the range of -0.5 to +0.5(27). Good item has discrimination index closer to +1 as it can distinguish good student from poor one but if poor student can answer more item correctly than good students, this indicate negative discriminating item which should be excluded (30). Downing(36) emphasized that items of MCQ test represented sample of all questions which could be tested, so for test with good internal consistency the test score should be an indicator for the student score on any other set with relevant items. Although our faculty recently develop assessment centre, we don’t apply item analysis to any exam, So I think before applying it, we are in need to orient our faculty members about the importance of item analysis and how we use its statistical data to detect causes of low discriminations , discard poor question, and identify gaps in curriculum.

Finally, we use written assessment to assess the major domain of cognition in its low level of knowledge recalling to its high level of knowledge application and problem solving, but as mentioned before, I think our written assessment has low reliability and validity as we use limited number of essay questions, and the percentage of essays marks are more than MCQ marks in our assessment, so we must apply using more objective tests of well structured MCQ, extended matching questions and short answer questions with more essays question especially modified essay, also we must determine the questions numbers according to their corresponding weight in the context and according to test blueprint, as these will facilitate sampling a broad range of relevant contents and constructs of our learning objectives.

Although I finish my essay about written assessment, During studying this course I was interested in OSCE assessment and how apply it in our department, but I can’t write about it as I don’t have experience on applying it because we don’t use it in clinical assessment and we use two long cases for applying report and ten short cases for radiological diagnosis for. Now I think we must apply using OSCE in our clinical assessment by using 10-20 stations, some of them are procedure stations like carrying ultrasound examination under observation and other pictorial stations on analyzing radiologic image like conventional, CT, MRI images, and answering context-rich questions related to images.

Refrences

Shumway JM, Harden RM. AMEE guide No 25: The assessment of learning outcomes for competent and reflective physician. Med Teach.2003;25:569-584.

Wass V, Van der Vleuten C,Shatzer J,Jones R. Assessment of clinical competence. Lancet.2001;357:945-949.

Dixon H.Candidates’ views of the MRCGP examination and its effect upon approaches to learning: a questionnaire stu