SPRATs to catch us
Many Grumble readers will have come across the mini-PAT. For any that have not, it is a peer assessment tool . There is a big industry now in these things. They go by various names: multi-source assessment, 360 degree feedback, peer review and peer rating are all examples of the same sort of appraisal technique. And, in case you're wondering, SPRAT is the Sheffield Peer Review Assessment Tool. In Sheffield they gave up making knives some decades ago. They now have a big computer, loads of staff and do mini-ePATs. And they are thriving. This is big business. Sheffield is now the premier provider of workplace assessment across the UK. So they must be doing a good job. Or so you would think.
If you ask those that should know about the evidence base for these mini-PATs you will get a blunt reply. They will look down their noses at you and there will be the implication that you must be pretty ignorant on the topic to have needed to ask the question. The curt put-down will mention validity, reliability and the like. You will have to wait until a Bank Holiday weekend to find out what the evidence really is. And then you will get a shock. Because if you look up the rather sparse number of publications on the topic you will find statements such as this:
It is no secret that mini-PAT lacks sufficient field evaluation and has not gone through any stringent criteria that are required for the validation of an assessment tool.
And further:
No evidence of reliability has been published to date.........
Furthermore, more generally:
Non specific feedback does little to effect a change in performance.
And:
In a third of cases feedback results in decreased performance.
An improved outcome is by no means the rule.
It is then that you will understand why Dr Grumble has been given such short shrift when he has queried these dubious newfangled initiatives. There is no firm evidence to support the mini-PAT. The emperor really has no clothes. Yet every new doctor in the land is being subjected to trial by mini-ePAT. And the results will be stored on a Big Brother computer for the rest of time. A whole industry in Sheffield is dedicated to this stuff. All based on smoke and mirrors. Does anybody know where Dr Grumble can buy a decent knife?
Reference
Aza Abdulla A critical analysis of mini peer assessment tool (mini-PAT)
3 comments:
Laguiole in the Auvergne is the heart of the French knife industry.
Solingen makes the best knives in Germany.
Another "not-for-profit" organisation, this time hiding in a children's hospital?
Ha - it's a research group. They say so themselves.
"Our team is headed by clinical academics with recognised expertise in medical education research. The research group, using mixed methodologies including the latest understanding in psychometrics, support instruments and assessment programmes in delivering high quality, robust feedback for individuals and organisations within the NHS."
Since there is no tangible evidence, then it has to be a research. Have the staff involved given their permission for this experiment and what ethical permission has been sought?
This has shades of MTAS.
The 360degree thing is designed to cause hypertension among witches. We just simply would refuse to do it. And that's that!
Quite. By big business I did not mean Big Business.
Post a Comment