The current COVID19 pandemic and social distancing measures are having a huge impact on our access and participation activity. Campus closures have forced student learning, support and experience online and onto telecommunication channels. In this reactionary response to the rapid arrival of the pandemic, institutions and students alike are grappling with quality, inclusion, engagement and wellbeing as we try to continue, and succeed, in education. In the access area, our outreach activities which in the main traditionally rely on face-to-face encounters with students – either opening our campuses up to school and college students, or reaching out to engage with students in classrooms and other settings – are no longer possible .
Such challenges are likely to be more acutely felt by, and have negative impact on, students from underrepresented backgrounds. For example, IT equipment or study space at home is likely to be less available to these groups of students; financial pressure may arise as the ability to go to work is now curtailed or jobs are lost; and offer holders and prospective students are likely to be disproportionately affected by interruptions to schooling and the outreach activity that supports the journey to and transition into higher education.
As we seek to address the challenges in supporting our students and continuing to deliver access and participation activity, such are our considerations. Understanding the demographics of our student body and prospective student pipeline can help to tailor our approach and activity. However, in this crisis where rapid, reactionary innovation and change is inevitable, it is also important to remember the value of the Theory of Change (TOC) framework and evaluation practice.
A TOC approach can help us focus on the key outcomes that we want to achieve and work out the best way of delivering these. It can be applied in the development phase of mitigation activity, or as a reflective tool to help to unpack, check and validate whether what you have put in place is likely to produce the intended outcomes. A TOC framework also helps to develop effective evaluation practice that can inform and direct activity improvement. Even in this current context, evaluation is incredibly important. Three critical benefits are:
Understanding the experiences and challenges of your students (current and prospective) during this crisis and how they are engaging with their study and university experience. This will inform current ongoing mitigation activities, but also provide essential data for the mid-to-longer term, which can direct your practice into 2020-21 onwards to ensure support and resource is provided where it is needed and can be effective in minimising longer-term disadvantage and/or disruption.
Understanding what practice is working (value) and what is not (waste) is important for short-term impact, allowing you to determine how effective mitigation activity is and implement required changes to improve activity. Evaluation here can also capture the potential longer-term value of mitigation activity, if for example it is more efficient to deliver than the ‘normal’ alternative and is having the desired impact; or, if students feedback that they prefer certain activity and it generates better outcomes for them. These findings can be applied to longer-term, evidence-based practice improvements that generate efficiency measures or improve performance trajectories.
Building the evidence base required for monitoring from the OfS. In their recent letter to providers, the OfS maintains its expectation that access and participation commitments should be delivered, or that “reasonable alternatives” are put in place. Monitoring of practice during 2019-20 will take into account these alternative measures, the evidence base for changes and evaluation of impact.
At its heart, a Theory of Change is a structured way of thinking about activities and evaluation by chunking them down into a chain of thinking and focusing in detail on each part of this chain. To make things administratively light for our members, our resident TOC expert at SEER has provided the TOC model below. This is a reduced version of a more complex approach, which the SEER team will use when developing your mainstream evaluation approaches with you.
Mini TOC logic chain
Worked example – Campus visit - online campus exploration (access domain)
We know that COVID19 mitigation activities will often inevitably be less complex than normal delivery and will normally require an online approach. Using the TOC above we can focus on key outcomes for our activity and determine how these will be achieved from the mitigation activity, determining it a “reasonable alternative” (OfS).
For example, a mainstream campus visit may have a number of intended outcomes; introducing potential students to the physical space of the university, showing them what student life is like, encouraging them to imagine themselves studying there in future, introducing them to student ambassadors who can make HE study seem realistic and possible to them. Ideally each of these outcomes will be evaluated separately. COVID19 mitigation activities, however, are likely to have to be more targeted and focused.
Thinking through the mini TOC questions and focusing on one or two priority themes that you want to evaluate, will help clarify the most important parts of your activity and therefore the aspects that you need to focus on when creating your evaluation.
TOC Question
Example answers – Online Campus Visit
1. What is the problem that your mitigation activity is designed to solve?
Students can’t get onto campus to visit. This means they can’t physically look around and see what student life is like here. This means that they won’t be able to visualise themselves studying here in future and may not consider applying.
The key problem we need to solve is to raise awareness of the benefits of our campus and encourage students to develop the kind of emotional relationship with our university that might previously have been formed through face to face interactions with friendly and welcoming staff and student ambassadors.
2. How will the mitigation activity solve this problem?
An online campus tour, using videos, in a free-form format the enables users to click through and focus on the aspects that most interest them will make best use of their time and attention.
We provide videos showing key buildings and facilities, a ‘walk through’ of the Students Union, talking head videos of a diverse range of students talking about their experience here. We have laid these video links out on campus map to give an open structure to the user experience.
3. How does this solve the problem identified in question 1? What is the ‘active ingredient that makes this happen?
By giving website users a free-form experience and allowing them to focus on the areas that are most important to them, we can encourage them to feel that they are in control and not just being given a linear marketing message. They are also likely to focus on the aspects that are most important to them.
Many of our videos are presented in an informal way by actual students. This helps to make the experience feel ‘real’ and that our virtual tour visitors have engaged with real people. We hope that this will help them feel a positive emotional connection to our university.
We have clearly labelled points of contact and encourage users to follow up with questions or comments.
4. How do we know if our solution has worked? What does this look like?
We want users to come away with a sense of connection with us.
We want them to have a more developed idea of who we are, and, hopefully, to see how they would fit in here.
This might increase the likelihood of their applying in future.
5. How do we measure this?
This is a relatively light touch intervention, so we need a light-weight evaluation.
At the end of their visit, we ask users to respond to a small number of ranked answer questions:
How much did they enjoy their virtual visit?
How much could they see themselves studying here in future?
[Follow up text question – why or why not?]
How confident do they feel that they now know what student life is like here?
It may well be that your answers are more complicated or more numerous than the examples given above. It is important to record everything. Once you have all your thoughts down, we recommend going through and prioritising the two or three aspects that are most important to your overall goal. These should form the focus of your evaluation. The more focused and targeted your evaluation is, the more likely it is to produce clear usable outcomes to measure the impact of your activity and how effective it has been.