Access and Participation Plan Evaluation Framework (APPEF)

In September 2021, a trio of colleagues from the University of Derby were delighted to attend the annual Network for Evaluating and Researching University Participation Interventions (NERUPI) conference to share the progress made in designing and implementing the University’s new Access and Participation Plan Evaluation Framework (“the Framework”). The volume of enquiries received after the conference is proof of members’ commitment to evidence informed decision making in access and participation! FACE members who burned the midnight oil producing the first round of Access and Participation Plans (APPs) in 2019 will be acutely aware of Office for Students (OfS) expectations that those plans included a ‘sufficiently detailed and robust evaluation strategy’ (OfS 2019:9 a) to ensure strategic objectives were met. The regulator called for ‘the strongest evidence possible’ (OfS 2019:1 c) ‘to justify resource allocation regarding activities to deliver the strategic measures and targets within APPs’ (OfS 2019 b) and to inform sector and institutional practice.

The Framework draws on OfS and the Centre for Transforming Access and Student Outcomes (TASO) guidance, the NERUPI framework, and best practice from across the sector. It consists of two elements – the APP Evaluation Toolkit (including the APP student characteristics database) and the APP Knowledge Hub. The Framework promotes the use of Change (ToC) – a strategic planning tool which can be used across the planning, delivery, and evaluation of activity to ensure that long-term goals are defined and understood.

APP Evaluation Toolkit

The APP Evaluation Toolkit and associated guidance materials are designed to support colleagues to embed evidence-based practice and evaluation methods within their roles and across their teams. It sets out the importance of evaluation and provides access to a range of useful templates, briefings, guidance (including GDPR and ethics), links and videos. Colleagues with little or no evaluative experience can use the resources to develop their confidence in designing and delivering evaluation plans. It supports those with more experience to take their evaluative practice to the next level. Our ToC 5 Stage Model (see Figure 1) provides a ‘one stop shop’ for teams:

Outline team activity/delivery plans.
Outline how the intended outcomes of the activities included in the team model are expected to be achieved, and what evidence there is that those expectations are reasonable.
Mapping of proposed delivery plans against the NERUPI framework
Colleagues build indicator banks and identify evaluation methods. What they are going to measureand how they are going to do it.
The three-wave model which clarifies which activities are targeted at student groups (whole cohort, target groups etc).

The APP Student Characteristics Dataset

In piloting the Framework, it became apparent that most colleagues had no means (beyond simple handwritten registers) of knowing which students had accessed their support, any gaps in student engagement with their services, and how to target those services to specific groups. To address that challenge, we developed the APP student characteristics database. The database combines student enrolment data with Indices of Multiple Deprivation (IMD) and POLAR4 data so colleagues can target activity and monitor participation.

The APP Knowledge Hub

We intend to make the APP Knowledge Hub accessible to colleagues at the University and across the sector. It has four main functions:

A place where colleagues can share articles, research, and/or best practice identified through their research, attendance at conferences and their networks etc.
A platform to facilitate an APP discussion forum – based around themes such as access, awarding gaps, sense of belonging, employability/skills, financial support, student voice, learning and teaching and research and evaluation.
A repository for the findings from evaluation activity and research generated and conducted within the institution.
Facilitate the sharing of best practice, identify opportunities for collaboration and help University to meet its monitoring and reporting responsibility towards the OfS.

How has the Framework been used to date?

The Framework has supported the development of comprehensive evaluation plans for our institutional outreach programme and our Personal Academic Tutoring scheme. It informed the development of a successful institutional bid for Turing funding and will now be applied to embed evaluation within the project delivery plan. The Framework has also supported the development of our APP research programme, including work exploring the lived experience of our black students (a comparative study) and commuter students. We are currently in discussions with the Union of Students to help embed evaluation within their Peer Assisted Learning Scheme. We are planning a research project to assess the effectiveness of the Framework itself in supporting the institution to achieve its APP goals and its impact on colleagues; discovering their professional, reflective journeys as they implement the Framework. This longitudinal qualitative research piece will map their journey and explore the personal and professional challenges of embracing what for many will be an entirely new approach to their roles and their practice. A study which we believe will be valuable to the sector!

Ambitions

A key goal is for the Framework to become embedded across the University – in the first instance supporting colleagues working on the key APP work streams and activities – and then beyond. We expect colleagues’ knowledge and awareness of evaluation techniques (and consequently their confidence and evaluation expertise) to grow. We want the Framework to be an important vehicle for sharing knowledge of ‘what works,’ in access and participation and to highlight the structural failures that persist throughout higher education, which must be addressed, for higher education providers to begin the process of closing equity gaps.

Blog by Dan West, Policy Lead Student Performance Team University of Derby and Jo Astley Evaluation and Evidence Manager University of Derby, Student Performance Team University of Derby

References:

OfS Regulatory Notice 1 Access and Participation Plan Guidance – a https://www.officeforstudents.org.uk/media/0bcce522-df4b-4517-a4fd-101c2468444a/regulatory-notice-1-access-and-participation-plan-guidance.pdf OfS Evaluation Self-Assessment Tool – Practical guidance on completing the tool -b https://www.officeforstudents.org.uk/advice-and-guidance/promoting-equal-opportunities/evaluation-and-effective-practice/standards-of-evidence-and-evaluation-self-assessment-tool/evaluation-self-assessment-tool/ OfS Access and participation stands of evidence -c https://www.officeforstudents.org.uk/publications/standards-of-evidence-and-evaluating-impact-of-outreach/