The University’s Appraisal policy for levels 4 and above was jointly agreed between UCU and the University of Southampton following the lengthy negotiations undertaken as part of the Reward Project.
Following the fiasco of the DAP (Development and Performance) proposals, based on the discredited General Electric or McKinsey 9 matrix, we pushed hard for a new appraisal process with core guiding principles and a developmental focus.
In recent weeks we have received copies of different local Faculty-specific guidance documents that appear to contravene the agreements and which potentially breach other agreed University policies, legislation and contractual arrangements.
We have been working with senior management representatives from HR to try to resolve the most pressing concerns about these changes to appraisal. Members will have seen we have had some welcome success clarifying that staff attendance at Open and Visit Days is voluntary and therefore should not be listed as a compulsory target in appraisal documents.
One of our concerns is with the inclusion of highly problematic metrics and performance targets in appraisal process. These include but are not limited to:
Use of Module Evaluation Questionnaire (MEQ) and NSS
One guidance document supplied to UCU sets out a target MEQ score of at least 4 in all areas, and an expectations that staff ensure a student completion rate of at least 50%. We wonder how this is to be achieved? Locking students in a lecture theatre until they complete the MEQ perhaps?
There is a wealth of evidence of gender and racial biases in these kinds of student evaluations e.g. research shows that students consistently score women lower than men. Given this the use of these scores in appraisal presents a clear risk of indirect discrimination in contravention of the University’s own equality and diversity policies, training (section 1.3 of the University of Southampton’s EDI briefing), and the law.
Moreover, it is well-recognised that MEQs are not a measure teaching quality. The link between teaching effectiveness and high evaluation scores is weak, and reliance on these can have undesirable negative consequences (e.g. grade inflation, and even rewarding bad teaching). In addition, those responsible for teaching know that an unintended consequence of such targets is to discourage staff from teaching ‘difficult’ or compulsory models (there goes Statistics 101 for all but the Statistics undergrads).
If you need more evidence, our colleagues in the UCU branch at Essex provide further detail in their report on why MEQs are unreliable. The bottom line is that any University that values and respects academic research should not uncritically use these measures.
Several guidance documents reference the use of NSS scores as performance measures, despite the fact that the causal link between the input of an individual staff member and the NSS scores cannot be established or verified. And again the spectre of ‘improving completion rates’ reappears. How is this to be ensured – bribery with meals ? Don’t laugh; we know one Faculty that gave out chocolate bunnies.
Inappropriate use of REF benchmarking scores
And just when we thought it was safe to go out again, the REF has returned to blight our lives. Some Faculties are setting targets that require the publication of REF returnable 3* and 4* publications, others the production of the same. Once again we have been forced to remind senior managers that :
i) While the production of publications is within the control of individual staff, their publication is not.
ii) The peer-review process that generates indicative and actual REF scores is known to be unreliable.
iii) No feedback loop exists to inform departments how individual publications were graded in any REF exercise, thus judgements about ‘REF-ability’ are ultimately speculative.
iv) In the last REF, the level of agreement between the internal ratings of submissions and those awarded by the panels was far from perfect.
Problematic performance targets and recognition of leadership roles
The documents in circulation create a range of problems with the assessment of so called leadership roles. What ‘counts’ as leadership is highly variable – mentoring is identified as an example of evidence of leadership in one document but not all. A more fundamental problem though is that not all roles and committee memberships are fairly and transparently allocated – we know for example that women staff are disproportionately represented in roles that relate to education and pastoral care, but Senior Tutor roles for example are seldom listed or considered as equivalent to other leadership positions.
In addition to the metrics and targets noted above, we are aware that senior managers are cascading income generation targets to individual members of staff. These too have been highlighted as a source of work stress [paywall] and may run counter to collaborative and innovative research.
We raised these serious concerns with senior management at our JNC last year and wrote a follow up letter to all the Deans and HR. Sadly we received only one response from a Dean (thank you Professor Mills) but we did get a response from the Employee Engagement team in HR. To date we have had one positive meeting with this team and we are hopeful that we will make progress this year. We did however get a ‘no’ to our request that the Faculty-specific appraisal guidance documents in circulation should be withdrawn pending these discussions with UCU. So our advice to all our members as we enter the appraisal ‘window’ for this year is to keep letting us know about problems you encounter with the appraisal process, and we will keep pushing management to ensure a positive, non-discriminatory and developmental appraisal process.