Objective To report the evidence for and challenges to the validity of Sheffield Peer Review Assessment Tool (SPRAT) with paediatric Specialist Registrars (SpRs) across the UK as part of Royal College of Paediatrics and Child Health workplace based assessment programme.
Design Quality assurance analysis, including generalisability, of a multisource feedback questionnaire study.
Setting All UK Deaneries between August 2005 and May 2006.
Participants 577 year 2 and 4 Paediatric SpRs.
Interventions Trainees were evaluated using SPRAT sent to clinical colleagues of their choosing. Data were analysed reporting totals, means and SD, and year groups were compared using independent t tests. A factor analysis was undertaken. Reliability was estimated using generalisability theory. Trainee and assessor demographic details were explored to try to explain variability in scores.
Main outcome measures 4770 SPRAT assessments were provided about 577 paediatric SpRs. The mean scores between years were significantly different (Year 2 mean=5.08, SD=0.34, Year 4 mean=5.18, SD=0.34). A factor analysis returned a two-factor solution, clinical care and psychosocial skills. The 95% CI showed that trainees scoring ≥4.3 with nine assessors can be seen as achieving satisfactory performance with statistical confidence. Consultants marked trainees significantly lower (t=−4.52) whereas Senior House Officers and Foundation doctors scored their SpRs significantly higher (SHO t=2.06, Foundation t=2.77).
Conclusions There is increasing evidence that multisource feedback (MSF) assesses two generic traits, clinical care and psychosocial skills. The validity of MSF is threatened by systematic bias, namely leniency bias and the seniority of assessors. Unregulated self-selection of assessors needs to end.
Statistics from Altmetric.com
Funding JA was funded by Clinical Research Fellowships through the University of Sheffield, Western Bank, Sheffield, S10 2HT supported by Doncaster and Bassetlaw Foundation NHS Trust and Barnsley NHS Trust.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.