Assuring validity of multisource feedback in a national programme
- 1Peninsula College of Medicine and Dentistry, University of Plymouth, Plymouth, UK
- 2University Hospitals Bristol NHS Foundation Trust, Department of Paediatric Nephrology, Bristol Royal Hospital for Children, Bristol BS2 8BJ, UK
- Correspondence to Dr Julian Archer, C408 Portland Square, Peninsula College of Medicine and Dentistry, University of Plymouth, Drake Circus, Plymouth PL4 8AA, UK;
Contributors MM was Vice-President for Assessment and Training. RCPCH established the national assessment programme with support from HD and JA. JA undertook the analysis. JA wrote the paper with assistance from the other authors.
- Accepted 8 February 2010
Objective To report the evidence for and challenges to the validity of Sheffield Peer Review Assessment Tool (SPRAT) with paediatric Specialist Registrars (SpRs) across the UK as part of Royal College of Paediatrics and Child Health workplace based assessment programme.
Design Quality assurance analysis, including generalisability, of a multisource feedback questionnaire study.
Setting All UK Deaneries between August 2005 and May 2006.
Participants 577 year 2 and 4 Paediatric SpRs.
Interventions Trainees were evaluated using SPRAT sent to clinical colleagues of their choosing. Data were analysed reporting totals, means and SD, and year groups were compared using independent t tests. A factor analysis was undertaken. Reliability was estimated using generalisability theory. Trainee and assessor demographic details were explored to try to explain variability in scores.
Main outcome measures 4770 SPRAT assessments were provided about 577 paediatric SpRs. The mean scores between years were significantly different (Year 2 mean=5.08, SD=0.34, Year 4 mean=5.18, SD=0.34). A factor analysis returned a two-factor solution, clinical care and psychosocial skills. The 95% CI showed that trainees scoring ≥4.3 with nine assessors can be seen as achieving satisfactory performance with statistical confidence. Consultants marked trainees significantly lower (t=−4.52) whereas Senior House Officers and Foundation doctors scored their SpRs significantly higher (SHO t=2.06, Foundation t=2.77).
Conclusions There is increasing evidence that multisource feedback (MSF) assesses two generic traits, clinical care and psychosocial skills. The validity of MSF is threatened by systematic bias, namely leniency bias and the seniority of assessors. Unregulated self-selection of assessors needs to end.
Funding JA was funded by Clinical Research Fellowships through the University of Sheffield, Western Bank, Sheffield, S10 2HT supported by Doncaster and Bassetlaw Foundation NHS Trust and Barnsley NHS Trust.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.