The MCQs and Continuous Professional Development

Objectives: To assess the appreciation of self assessment (SA) by multiple choice questions (MCQ) tests during continuous professional development (CPD) meetings and to evaluate the attitude towards the option of recertification.

Methods: A questionnaire was completed at a meeting of the Pakistan Society of Surgeons, Mirpurkhas Chapter (response rate 94.9% (223/235)).

Results: 60.9% of members found SA useful to assess surgical knowledge by MCQ tests; 71.3% thought it

motivating to receive an anonymous feedback; 47.2% wanted an additional individualized feedback; 70.8% asked for SA to be continued every 2 years during CPD sessions; 26.3% favored the option of recertification with identical standards to the Pakistan Society Of Surgeons. Physicians in private practice less often chose the option of recertification than physicians employed by hospitals. No correlations between the type and duration of specialization, personal reading time, frequency of attendance at CPD meetings, sex of the members, and the choices made were found.

Conclusions: SA by MCQ tests during CPD meetings is highly accepted by Surgeons of Mirpurkhas Chapter of Pakistan Society of Surgeons. This reliable, valid, and economic method of learning needs assessment enables individual physicians and scientific societies to plan educational with specific goals in mind.

______________________________________________________________________________________________

INTRODUCTION

Continuous professional development (CPD) (1,2) is characterised by lifelong, self directed learning rather than supervised training and aims at maintaining and further developing a broad range of competencies for knowledge, skills, and attitudes. Learning is more likely to lead to a change in practice when needs assessment has been conducted (3). CPD planning should be based on a thorough and accurate assessment of needs, opportunities, prospective responses to internal and external forces, and an understanding of, and respect for, the learners’ interests (4). Tools for self assessment (SA) should be developed to help doctors identify their personal learning needs (5). There are various formal or informal and planned or opportunistic ways of identifying learning needs for individuals or groups. These include a clinician’s own experiences in direct patient care, interactions with the clinical team, formal approaches to quality management, specific activities directed at needs assessment including objective tests of knowledge and skills, peer review, and non-clinical activities (3).

Since 1995 multiple choice question (MCQ) tests have been conducted during several annual scientific meetings of different Scientific Societies. The tests used 60 MCQs each covering all areas of surgery. The questions of three types were developed according to a blueprint by a group of surgeons working in private practices and public hospitals. Participation was anonymous, free of charge and lasted for 1-2 hours. Feedback was given about the state of knowledge in comparison with peers for surgical diseases as well as general issues. About 40-60% of the CPD meeting participants took part, with increasing numbers over the years inshaullah, representing about 30% of the members of the societies.

As time and money constraints have been increasing over the past few years the question of continuation of these tests has been raised. We therefore decided to conduct a survey about the attitudes of Mirpurkhas Surgeons with the following aims:

To assess the appreciation of SA by MCQ tests

To evaluate the attitude towards the option of recertification

To analyse the results of the survey for influences by specific factors.

METHODS

At the beginning of the general meeting in January 2010 the president of the Pakistan Society Of Surgeons, Mirpurkhas Chapter explained the aims of the survey and asked all members to fill in the anonymous questionnaire. The questionnaires were collected by one author (CB) during the meeting with the consent of each person.

Statistical analyses included the calculations with the help of SPSS Version 10.

RESULTS

Of 235 members attending the meeting, 223 returned the questionnaire (179 men, 41 women, 3 without specifications; response rate 94.9%). Of the participants of the survey, 180 were certified in General Surgery, whereas 14 were still in postgraduate training for General Surgery. The remaining 29 surgeons were specialised in Orthopaedics (20), Urology (5), Plastic Surgery (3), or gave no information (1). One hundred and nine had been certified for more than 10 years. One hundred and forty seven were working in private practice, 65 were employed by a hospital (34 non-teaching, 21 academic, 10 private hospital), and 11 gave no specifications.

Figure 1 shows the reading times and frequency of attendance at CPD meetings declared by the participants. Employed physicians more often read publications for >2 hours a week than physicians in private practice. Conversely, the frequency of visits to CPD meetings was similar in both groups. Physicians specialised for >10 years more often visited CPD meetings for ≥7 times a year than colleagues with a specialisation for ≤10 years. On the other hand, the likelihood of reading for >2 hours a week was similar in both groups. No significant differences between men and women were found.

Table 1 shows attitudes about concepts of CPD, SA by MCQ tests, and the option of recertification. Additional subanalyses for type of specialty, duration of specialisation, duration of reading time, and frequency of attendance at CPD meetings showed no differences. There was a trend for physicians in private practice to be more sceptical about SA and feedback by MCQ tests. However, physicians in private practice more often denied the option of a recertification. Physicians who rejected the offer of an individual, personalised feedback in addition to the anonymous participation in SA more often denied the option of recertification.

DISCUSSION

MCQ tests assess knowledge and, partly, problem solving in an objective and reliable way (6). The development of the questions is challenging and time consuming. On the other hand, evaluation of the tests and the questions is very quick if automated analysis is used. Owing to the high number of participants, MCQ testing becomes a very economical method of assessing the updating of knowledge in order to understand, diagnose, treat, and prevent diseases.

However, MCQ tests do not evaluate other competencies, including clinical reasoning and communication skills. Other, far more expensive methods, such as the objective structured clinical examinations (7) or the standardised physician assessments in medical practice, (8) are more suitable for assessing a wide range of competencies and skills necessary for the delivery of various professional tasks. Nevertheless, the results of MCQ tests are related to clinical competence, quality of practice, and patient outcomes (9,10).

Analyses of the test results enable individual physicians and scientific societies to plan educational experiences towards a specific goal. There is no certainty that physicians who complete SA programmes will participate in education sessions that relate to their self assessed needs. But the fact that more than two thirds of our participants, including those who had completed their postgraduate training more than a decade ago, feel motivated by this kind of feedback gives rise to optimism that many of them will construct their lifelong, self directed learning around their SA results and their individual practices after completion of such tests. Moreover, organising committees of scientific sessions can put more emphasis on areas where a high proportion of participants show deficits in topics relevant to their practice. In addition, CPD planners can change their role from providing professional education and instruction to facilitating professional learning to enhance competence and performance wherever and whenever this may occur (4)

The goals of recertification are to improve the care of patients, to set standards for practice of medicine, to encourage continued learning, and to reassure patients and the public that physicians remain competent throughout their areas (11). Our approach could be used as screening method13 and as basis of a credit system for monitoring and promoting excellence (13). When poor results are obtained, these MCQ tests could be followed by peer review of practice to assess performance in the work place and by tests of competence, including written and practical examinations, to assess knowledge and skills (12). Discussions of this kind have not yet been carried out within our societies.

Not surprisingly, only one quarter of the participants of this representative survey favour recertification. This illustrates the clear distinction between the formative and summative purpose of MCQ tests by our members. The majority favour feedback in order to improve their competence. In contrast, only a minority approves a final assessment, possibly because of fear of failing the examination and losing anonymity. The latter is supported by our finding that caution is more pronounced in members who reject individual, personalised feedback in addition to the anonymous participation in SA. Scepticism is less in employed physicians who are reading more according to their self declaration and work preferentially in secondary and tertiary referral centres. Similar attitudes can be expected in European countries. Nevertheless, the implementation of voluntary periodic recertification could become standard policy in the future (13).