Context: Well-designed assessments of performance are urgently required for training doctors, and to provide indicators of the quality of practice.
Objectives: To design an assessment process that uses routine outpatient letters, and to evaluate its validity, feasibility, reliability (reproducibility and discrimination) and potential educational impact.
Participants: All 26 paediatric registrars in North Trent attending annual assessment panel in 1999 participated.
Study design: An assessment instrument (SAIL) was developed from a consensus framework for good practice in written communication. It comprises an 18-point checklist and a global rating scale. Three judges applied the instrument to 260 letters from the routine clinical practice of the 26 participants. Results We achieved consensus on good practice in written communication. This was in keeping with the published literature. All participants completed the assessment. Scoring took 3-6 min per judge per letter. The reliability coefficient in this test situation is 0.72. Modelling predicts that a coefficient of 0.8 (the threshold for high-stakes judgements about performance) can be achieved with more judges or letters. The assessment results are well suited to formative feedback.
Conclusions: SAIL uses letters as a face valid indicator of written communication performance. The instrument is feasible to use, and produces reliable results when applied to paediatric registrars to inform the annual Record of In-Training Assessment (RITA). Feedback from the assessment should help doctors to improve their written communication. Its use may extend to other specialities and settings including revalidation.