Article Text

PDF

Portfolios, appraisal, revalidation, and all that: a user’s guide for consultants
  1. H Davies1,
  2. N Khera1,
  3. J Stroobant2
  1. 1Sheffield Children’s’ Hospital, UK
  2. 2Lewisham Hospital, UK
  1. Correspondence to:
    Dr H Davies
    Consultant in Medical Education and Late Effects, Sheffield Children’s Hospital, Western Bank, Sheffield S10 2TH, UK; h.daviesshef.ac.uk

Abstract

From April 2005 all doctors in the UK will be expected to be able to demonstrate their fitness to practice as part of the GMC revalidation procedures.1 The revalidation process is explicitly linked to the consultant appraisal process implemented in 2001.2 Central to both processes is the development of a folder (portfolio) of supporting evidence. Many consultants have no experience of developing portfolios and are unclear about how to meaningfully do so and what sort of evidence is suitable for revalidation. Furthermore, they are uncertain about whether there is any evidence to support their use for appraisal or assessment. This paper describes what a portfolio is, summarises the evidence for their use in appraisal and assessment, and provides guidance on the collection of evidence for revalidation purposes. In addition, it explores the distinction between appraisal and revalidation. Some evaluation data on perceived benefits and drawbacks by participants in appraisal in a paediatric setting is also included to inform conclusions and thoughts on planning for the future.

  • appraisal
  • portfolio
  • revalidation

Statistics from Altmetric.com

Portfolios have been used for many years in nursing and in professions outside medicine, such as teaching and architecture.3–6 More recently they have become increasingly advocated as means to support CPD in medicine and also as a possible means of assessment.7,8 Their potential for supporting personal development has been well described.8,9 Whether portfolios are suitable for use in (summative) assessment of doctors is however the subject of fierce debate.8,10–15

Wilkinson et al define a portfolio in a way that specifically relates to medical education and assessment as “a dossier of evidence collected over time that demonstrates a doctor’s education and practice achievements”.14 This definition can be used to define a portfolio developed at any stage of a doctor’s career, although the content will change over time. Wilkinson et al recognise that portfolios may take many forms and in the same issue of Medical Education, Webb et al discuss four models of portfolios identified in their case studies of nursing programmes.16 They recognise that the format as well as the content of a portfolio will depend on the purpose for which it is intended.

PORTFOLIOS AS A DEVELOPMENTAL TOOL

There is a large body of literature supporting the use of portfolios for personal development. Mathers et al compared portfolio based CME with traditional CME and found that the portfolio based learning group had a wider range of learning activities and topics.9 In addition, it promoted an increase in self-knowledge and confidence in relation to individual learning preferences and needs. A questionnaire based evaluation of the use of portfolios by paediatric specialist registrars in North Trent who had used portfolios to support their personal development for two years included the question “What do you think is useful about portfolios?” and identified a range of outcomes (fig 1). Their usefulness as a record of achievement was identified by 85%; and to facilitate the setting of objectives, by 61%. Nearly 50% felt that using portfolios facilitated reflection.

Figure 1

 Usefulness of portfolios for paediatric SpRs.

Only one SpR (from 21 respondents) identified using their portfolio for the RITA process as a useful outcome emphasising the developmental nature of the portfolio.

Portfolios developed for revalidation will be used for developmental purposes, but also explicitly to determine whether or not the doctor is fit to practice—in effect as an assessment tool.

PORTFOLIOS AS AN ASSESSMENT TOOL

There is general agreement that peer rating scales can achieve sufficient reliability and face validity to be acceptable for high stakes assessment. However, the role of portfolios in performance assessment is much more controversial.13,14,17 Possible benefits of portfolios include:

  • The potential to evaluate a doctor’s ability to reflect on practice and learn from experience

  • Improved patient care (based on the evidence that where properly used portfolios can enhance learning from experience and lead to improved patient care)

  • The flexibility to demonstrate professional development over time as well as contemporaneous satisfactory performance

  • Ability to be tailored to an individual’s practice profile and their contents determined by a doctor’s own learning/practice needs.

The usefulness of portfolios as a formative assessment tool is not disputed. What is not clear is whether they can be sufficiently robust in terms of reliability and validity to be used as part of a high stakes summative assessment such as recertification procedures, revalidation, and annual review for SpRs.13 Currently there is a paucity of evidence on their reliability as an assessment tool for summative purposes in the medical education setting.12,13,15,17 In fact, the largest body of evidence in relation to portfolios and assessment comes from the American elementary school system.13,18,19 The school studies suggest that reliability can be enhanced by utilising uniform content, preparing students adequately, and appropriate preparation and training of examiners who use clearly defined assessment criteria. Wilkinson et al acknowledge concerns around the use of portfolios for assessment purposes, but emphasise the potential benefits and provide guidance on constructing portfolios in a way that will provide a complete and accurate picture of an individual doctor’s practice.14 In spite of concerns about reliability, evidence for their effectiveness in enhancing learning from experience is a powerful incentive to include portfolios in an assessment strategy if possible.7,17,20,21 In addition, a portfolio would only be one of a range of assessment tools which made up an overall assessment programme and any assessment programme is more than the sum of the individual parts.22 A number of organisations worldwide are advocating the use of portfolios for assessment of doctors in practice including the ACGME and ABIM in the USA.23,24 In practice, whatever the current evidence on the use of portfolios in assessment, all consultants will be required to prepare a revalidation folder: What should go into it? How do we use a folder (portfolio) to demonstrate our fitness to practice?

COLLECTING EVIDENCE FOR APPRAISAL AND REVALIDATION

Collecting evidence to demonstrate our fitness to practice is essentially about providing an accurate picture of us as professionals. The latest guidance on revalidation describes two main routes to revalidation: the appraisal route and the independent route.1 The appraisal route is the route likely to be followed by the majority of practising doctors and requires the doctor to have worked in a managed environment, participated in an appraisal system, and completed the supporting documentation. The supporting documentation (evidence) will be collected in a folder. Good Medical Practice (GMP)25 is central to both appraisal and the linked revalidation process.1,2 The appraisal guidance states that:

“Appraisal should include data on clinical performance, training and education, audit, concerns raised and serious clinical complaints, application of relevant clinical guidelines, relationships with patients and colleagues, teaching and research activities, and personal and organisational effectiveness.”

For doctors in training, evidence collected to inform the RITA process will also need to fulfil the requirements for revalidation, an important consideration when planning assessment systems for doctors in training.26

In addition to reflecting the general content of GMP, the evidence collected should reflect the specifics of a given doctor’s individual practice. Norcini suggests a useful classification for work based assessment methods and suggests that three key aspects of doctors’ performance can be assessed: patients’ outcomes, process of care, and volume of practice27 (fig 2). There are advantages and disadvantages to all of these which are summarised in table 1.

Table 1

 Work based assessment

Figure 2

 Classification scheme for work based assessment (from Norcini, with permission).27

The consultant appraisal documentation requires the evidence considered for annual appraisal to be recorded in categories related to GMP (Form 3—Record of Reference Documentation) and discussion around the evidence to be summarised in the same categories in a separate form (Form 4).2 The paperwork required is somewhat cumbersome and involves duplication. In addition a number of consultants are unclear about what sort of evidence is suitable for each domain of GMP. Table 2 lists the domains of GMP and suggests possible sources of evidence for the reference documentation required for Form 3. A modified version of Form 3 with indicative lists of evidence for each section can be accessed on the ADC website. Table 2 and the web resource are intended as a list of possible sources of evidence from which doctors can sample. It is not anticipated that anyone would produce a folder containing all these sources of evidence! It is helpful to record whereabouts in the appraisal folder each source of evidence is located for ease of navigation, both for the appraiser and also potentially for revalidation purposes. One way of organising an appraisal/revalidation folder based on the original GMC guidelines is also provided on the ADC website. It should be emphasised that the majority of consultants will not be required to submit their entire appraisal folder, rather that a proportion will be randomly sampled.

Table 2

 Categories of evidence required for consultant appraisal and possible sources of evidence

It is important that some robust evidence that is defensible is included in the appraisal process, both to encourage individual doctors’ reflection on their own practice and to ensure that the evidence is sufficiently robust for revalidation purposes. Professional colleague surveys and patient satisfaction questionnaires are particularly suitable for this purpose. In addition they provide information that facilitates professional development planning and is linked to an individual doctor’s practice. They have been used elsewhere in the world, in particular by the American Board of Internal Medicine (ABIM) and Canada for certification and re-certification purposes, and will form part of paediatric training grade assessment processes.28–31 Professional colleague surveys (peer rating forms) for use in the UK should preferably be mapped to GMP (as the framework for good practice for UK doctors), and evidence for their reliability and validity should be provided. Similarly it is important that the measurement characteristics of patient satisfaction questionnaires are evaluated. Despite a large number of patient satisfaction tools being reported in the literature, very few have had their measurement characteristics adequately evaluated, a problem acknowledged by Sitzia in an excellent review of patient satisfaction tools.32 Professional colleague surveys and patient satisfaction tools33 have been developed for use in this context in the UK in a paediatric setting.34

WHY BOTHER; STICKS OR CARROTS?

It is already a requirement for all consultants to participate in annual appraisal using the Department of Health framework, and from April 2005 all consultants will be expected to be able to demonstrate their fitness to practice when required to do so. These then are the sticks, but what about the carrots?

The explicit link between appraisal and revalidation aims to minimise the time and effort required by consultants to fulfil the requirements of both processes. It does however have the significant disadvantage of having blurred the distinction between appraisal (which should be a largely confidential and supportive process), and revalidation, which is essentially a summative assessment process. This difficulty is highlighted by a number of authors.35,36 Careful thought, adequate support mechanisms, and input into the processes from individual practitioners are essential if the majority are to buy into the processes.

Importantly, if these processes are to be worthwhile for individual practitioners, there must be individual benefit arising from them. There is evidence that workplace based assessment can be used to inform and initiate change.37,38 In a Canadian setting peer ratings have been used as part of the Physician Achievement Review undertaken by the College of Physicians and Surgeons of Alberta.38–41 A recent evaluation of the process with a random sample of 252 surgeons from a range of surgical specialties38 found that three months after administration of the instrument, 71% of the surgeons had contemplated or initiated change on the basis of the multi-source feedback (based on self-reporting). A small study of the impact of providing structured feedback on outpatient letters demonstrated an improvement in letter writing after receiving feedback using a validated assessment tool.37 It is essential, however, to recognise that implementation of such processes can be threatening and should be handled sensitively. Furthermore, provision of adequate support mechanisms is essential, and the effectiveness of feedback will be greatly enhanced by discussion of the feedback with an appropriate appraiser.

The conflict between appraisal and assessment (revalidation) has compromised attitudes to both processes. Every attempt possible should be made to encourage acceptance of the processes by all the stakeholders involved. Important factors in promoting acceptance of performance assessment are summarised by Finucane and colleagues.42 Inevitably neither process is going to be universally accepted, but it is to be hoped that the enthusiasts can lead the way.

Evaluation of the process is essential if we are to improve it for subsequent cycles and optimise its perceived and actual effectiveness.

WHAT DO CONSULTANTS IN A PAEDIATRIC SETTING THINK ABOUT APPRAISAL?

Following the first year of consultant appraisal in a paediatric Trust, consultants were asked their views on the process by one of the authors as part of an evaluation of the process. Consultants were able to self select appraisers. There was the option to participate in a voluntary 360° feedback process. Forty two consultants returned questionnaires. Twenty two of these had elected to participate in the 360° process.

The three main benefits identified by participants’ responses to the question “What did you like best about the process?” were:

  • Opportunity to reflect on current and future practice

  • Supportive nature of the process

  • Opportunity for protected, confidential discussion focused on their own needs.

This is encouraging as these benefits are in keeping with a formative, supportive appraisal process focused on self reflection, characteristics likely to optimise the developmental potential of the process.35 Unsurprisingly however, there were inevitably also a number of problems with the process.

Key issues identified by participants’ responses to the question “What did you like least about the process?” were:

  • Time required to prepare for and undertake the process (without adequate resources)

  • Paperwork required, including the repetitive nature of the DoH documentation.

Other issues identified included the option of selecting friends to provide feedback, potentially limiting its usefulness, and lack of resources/action in response to the process.

In addition to the time required clearly being the major issue in relation to undertaking appraisal, it was also identified as the major barrier to achieving personal development plan objectives. Box 1 provides some indicative quotes.

Box 1: Some indicative quotes

“What did you like best about the process?”

Examples of responses:

  • “Challenging, made me reflect on practice”

  • “Time to think about me and where I am”

  • “Supportive and non-confrontational”

  • “Reflective opportunity”

  • “Opportunity to discuss issues with someone unconnected with the department”

  • “Good feedback—help clarify thinking about current work objectives”

“What did you like least about the process?”

Examples of responses:

  • “Massive amount of time and repetition spent”

  • “Time consuming—forms complex”

  • “Had to make time in a busy schedule—yet one more thing to fit in”

Of those who participated in the 360° feedback process, all except one said they would be willing to participate in the process again. Fifteen (68%) felt that the feedback had identified areas that needed addressing, although only nine had used it to inform their personal development planning.

PRACTICAL TIPS

Maintaining a portfolio (folder) is yet another thing for busy consultants to undertake. Putting evidence together the first time is inevitably the most time consuming. While most will maintain a paper based portfolio, some doctors will choose to maintain their portfolio as a (largely) electronic document. Dornan et al report experience on the use of an electronic portfolio for physicians for reflective CPD.43 There are no published studies of the use of portfolios in a paediatric setting from which we can draw specific paediatric guidance. Efficient maintenance of a portfolio is promoted by ensuring that a box file or similar receptacle is kept close to hand as a repository for all possible documents for your portfolio and then weeded and sorted into the portfolio periodically. Similarly maintaining a portfolio folder on your computer with sections equivalent to the paper version to store documents in prospectively will greatly facilitate preparation of your portfolio for annual appraisal purposes.

Participation in a validated peer or patient feedback process, if available, will provide evidence in line with GMC guidance on fitness to practice. Output from these processes along with other suitable evidence will provide reassurance to the GMC of a doctor’s continuing fitness to practice. Evidence will be collected in the folder and summarised for appraisal/revalidation purposes. The GMC are only likely to want to see the actual folder in a minority of cases. Box 2 summarises some practical tips on putting a portfolio together.

Box 2: Some practical tips for maintaining a portfolio

  • Keep appraisal folder where it is easily accessed

  • Collect all possible suitable material on an ongoing basis and then weed intermittently

  • Keep a portfolio folder on PC to “store” suitable electronic material in for later weeding in the same way

  • Ask appraiser to review portfolio formally

  • Consider asking a friend or mentor to review portfolio on an informal basis

  • Plan collection of peer rating data, patient satisfaction data, etc in advance to ensure that it is available for discussion at appraisal

  • Discuss with IT department what data they are able to provide you with, e.g. activity and outcome data

  • Organise portfolio so that it is easy to navigate—lots of lists and labels

CONCLUSIONS

Appraisal and assessment (revalidation) are important and largely desirable processes for senior doctors. Both potentially involve collection of evidence in a folder (portfolio) and discussion of this evidence with another individual. There should, however, be important distinctions—appraisal should be a confidential, supportive process largely used for personal feedback and development planning. In contrast, assessment as part of regulation necessarily involves the making of judgements. The blurring of the distinction between the two processes as currently planned does create difficulties which potentially makes management of both appraisal and revalidation more difficult. Nevertheless given the potential burden of both processes in terms of time and resources it is clearly desirable to minimise duplication as far as possible. This paper has attempted to clarify the two processes and in particular highlight the advantages and disadvantages of portfolios (folders) for assessment processes as well as provide some practical guidance on possible sources of evidence for portfolios.

Major issues that clearly require resolution are those of time and paperwork. In addition we should aim to build on the benefits of appraisal while learning from feedback on the process and minimising the drawbacks. Streamlining of the current appraisal paperwork would be desirable, and training of appraisers as well as allocation of protected time for preparation and participation in appraisal as part of job planning is essential.

Collection of evidence for revalidation should aim to collect evidence that doctors themselves are interested in, is defensible, and can be tailored to an individual doctor’s practice. Encouraging ownership of both processes for example by self-selection of appraisers, allowing doctors to choose from a range of suitable validated assessment tools for revalidation, and by modifying processes on the basis of feedback will enhance acceptance and facilitate the usefulness of both appraisal and revalidation. Certainly, while recognising the important stimulus regulatory processes provide for participation in both appraisal and revalidation we should be aiming to utilise carrots rather than rely on sticks.

Summary of key points

  • A portfolio is a collection of evidence that demonstrates a doctor’s education and practice achievements

  • Portfolios have been shown to promote reflection and encourage practice based learning

  • Reliability of portfolio assessment does not meet conventional reliability requirements for summative assessment purposes

  • Appraisal and revalidation are mandatory and important

  • Revalidation will be introduced for all doctors in 2005

  • Appraisal and revalidation both require collection of evidence in a folder

  • Peer and patient feedback are suitable for both appraisal and revalidation purposes

  • Maintenance of a folder is facilitated by setting up systems which encourage prospective collection of evidence

  • Evidence collected should cover the domains of Good Medical Practice


Embedded Image

Acknowledgments

Dr Khera was supported by grants from Lewisham Hospital and the South Yorkshire and South Humber Deanery.

REFERENCES

View Abstract
  • Form 3 is available as a downloadable PDF (printer friendly file).

    If you do not have Adobe Reader installed on your computer,
    you can download this free-of-charge, please Click here

     

    Files in this Data Supplement:

    • [View PDF] - Form 3 Record of reference documentation supporting the appraisal and report on development action in the past year

Footnotes

  • Competing interests: none declared

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.