Our approach to evaluation
Different types of data
It is no coincidence that there has been very little impact evaluation in open access provision — it presents particular challenges to traditional or established approaches to impact measurement. Open access, non-formal youth provision is often based on relationships or approach, rather than activities or journeys with a defined beginning, middle and end. It may not have pre-defined outcomes. And young people are likely to engage in very different ways, or have very different experiences of provision.
Traditional evaluation tends to focus on whether change occurred or not, and how much change occurred. We are interested in capturing information about changes in young people’s personal and social development (e.g. their knowledge, skills and attitudes), which are outcomes that are closely linked with longer term life chances for young people (e.g. employment, education, health).
But sometimes it’s difficult to gather this information for capacity reasons (eg, following young people up and making time to capture this information) and for practical reasons (eg, asking young people to complete questionnaires can disrupt the flow of engagement).
Recognising these challenges, the Youth Investment Fund Learning Project is taking a broader approach to data collection.
We are focusing on five types of data which will allow us to develop a more comprehensive picture of a young person’s experience of youth provision.
We want to gain more insight into how young people engage with provision—by exploring what they do with their feet Evidence already suggests that more sustained engagement with youth provision is likely to generate greater impact for young people. We want organisations to collect this information in a consistent way and are working with Yoti to assess options for digitisation. This will help us develop a clearer picture of who organisations are reaching, how young people engage with provision, and how this is related to impact.
We believe that young people are uniquely able to explain their relationship with the provision in which they participate, and should have systematic opportunities for doing so. Feedback can provider indicators of quality, help organisations improve services and may even predict outcomes. We are working with Keystone Accountability to develop and test a core set of feedback questions for the open access youth sector, which can be used to collect, assess and act on feedback.
Quality of provision
This is more than quality assurance—we are looking for markers of quality that help us understand what young people experience in the moment, rather than just looking at what happens as a result. We are working with the David P Weikart Center for Youth Program Quality to support self-assessment, with a focus on the quality of engagement with young people, as a practical, learning-centred approach to identifying quality markers. If we can connect the quality of youth work to the change experienced by young people, the presence of quality markers in provision could predict the change that young people experience.
We are also including more ‘traditional’ evaluation approaches by developing a common outcomes framework for open access youth provision, based on a shared theory of change. This will be informed by existing frameworks such as Inspiring Impact’s Journey to Employment or the Young Foundation’s Catalyst Outcomes frameworks. By collecting this information consistently across a range of organisations, we will develop a stronger evidence base about the impact of open access youth work. Our ambition is to link these findings with footfall, feedback and quality data, and look for predictive links between the different types of data.
The hardest part of assessing the impact of any project is knowing ‘what would have happened anyway’. We will work with BPSR to estimate a counterfactual—a hypothetical world in which young people didn’t access youth provision. This will help us attribute any changes to the provision, rather than other factors.