Using reflective diaries to evaluate a multi-engagement outreach programme: A case study

In this post, Jo Astley and Luke Gordon-Calvert discuss how they introduced reflective diaries as an evaluation tool in the High Attaining Pupils Programme (HAP), led by the University of Derby in partnership with The Bemrose School, which is now in its second year.

Programme background

The HAP programme is an intensive, multi-interventional outreach initiative working with 15 students from Year 7 (aged 11-12 years) through to Year 11 (aged 15-16 years). The programme delivers awareness and attainment raising activities; exposing participants to the progression and careers opportunities that will be available to them in the future. Through participation in the programme, pupils will accrue the knowledge, skills and capitals that groups under-represented in higher education (HE) are often characterised as lacking in comparison to their more ‘privileged’ counterparts.

For a more in-depth overview of the programme, check out this FACE blog post from 2019.

Evaluation framework and methodologies

In response to commitments through our Access and Participation Plan (APP) the University of Derby is currently developing an overarching evaluation framework which will be embedded across the institution. The framework includes the application of Theory of Change methodology to all our activities, including the HAP programme.

A mixed-methods evaluation framework has been embedded within the programme which includes a pre-programme baseline knowledge and attitudinal data questionnaire, the monitoring of predicted and actual grades (from Year 9 – 13 to 14 years old), and individual event evaluation. Qualitative data is being collected through regular consultation with participants through focus groups and interviews with the Pupil Progress Lead (PPL). Additionally, reflective journals are completed by participants at every point of contact.

The programme is intended to be responsive to participants’ needs and interests as they progress along their learning journeys. Adopting participatory consultation methods not just to programme development but to the evaluation of the programme as well.

Reflection through activity

The Widening Access team introduced reflective diaries into their evaluative practice in 2017. They were initially developed for use during a Year 10 White working-class boys pilot programme, featured in the 2019 FACE publication. The pilot represented a significant investment of university and partner school resource and the time of participants, so it was essential that the impact of the programme was measured as effectively as possible.

Early learning for the team about using diaries as a qualitative tool was the challenge that reflective practice posed for participants; an issue that wasn’t given enough consideration at the outset.

In response to that challenge, reflective activity was introduced at the mid-way point of the original pilot programme and has been incorporated at the outset in the first programme session (the University Experience Day) of the HAP project. It is also a continuing theme.

The introductory session focuses on the advantages of reflection in helping the participants to know and understand themselves, identify their strengths and weaknesses, learn from their mistakes and track their successes. The setting of SMART targets (specific, measurable, achievable, relevant and time-bound), was also a feature of the first session.

Reflective diary prompts and responses

Reflection has also been supported through a series of prompts incorporated into revised participant diaries. You can see these, as well as a selection of participant’ reflections, in the table below.

The team’s experience of the data collected through the reflective diaries during the pilot programme was that, while in quantity it was minimal, in quality it was rich. This initial data indicates both learning and some evidence of reflection on the part of the participants. The expectation of the team is that as the participants progress through the project and develop their reflective skills, the data generated through the diaries will become more illuminating.

The programme going forward

Reflection will be a theme that continues throughout the coming year. Consultation with the PPL indicated that there is a need to increase participants’ understanding and recognition of transferable skills and how they impact and inform their education and employment pathways. In addition, informal 1:1 interviews with participants (conducted by student ambassadors) will be incorporated into the programme, with the aim of better identifying participants’ areas of interests and aspirations. Reflective diaries will continue to be central to the evaluation of the programme, and the effectiveness of diaries as a tool to collect qualitative data will be the subject of consideration for the team going forward.

FACE readership will be updated on findings from the second year of the programme at the beginning of the new academic year. If you have any questions about the project or would like to discuss the use of reflective diaries, feel free to get in touch with Jo (j.astley@derby.ac.uk) or Luke (l.gordon-calvert@derby.ac.uk).

Blog by

Jo Astley is Evaluation and Evidence Manager at The University of Derby

Luke Gordon-Calvert is a Senior Widening Access Officer The University of Derby