Demonstrating Impact in Music Education

Demonstrating

Impact in

Music Education

A partnership project for the music education sector

led by London Music Masters, Pro Bono Economics,

Project Oracle and Sound Connections

Demonstrating Impact in Music Education

A briefing paper for the music education sector

Can we know with greater certainty that we are achieving the aims of our

community engagement programmes?

This is the question which drives the Demonstrating Impact in Music Education

(DIME) group and to which this paper is addressed.

1. Introduction

Music educators have an intuitive understanding of the impact that their work has on individuals

and communities. There are countless stories of lives transformed and challenges overcome. Often

we communicate these stories well and persuasively, and as a professional community we learn

from our experiences and bring that learning to bear in future programme design.

Pressure on resources means that finding the most economical, efficient, and effective way of

achieving our aims and communicating this to funders and other stakeholders has never been more

important. The impact of every pound spent in a music education context is critical. The

beneficiaries of our work require and deserve world-class provision that delivers on its claims.

Additionally, funders and stakeholders need to be convinced that a) our interventions are the most

appropriate way to meet non-musical aims, and b) the musical aims are being delivered to the

highest standard.

This paper summarises the DIME group’s exploration of these issues to date, and suggests a way

forward for increasing the skills and knowledge within the sector.

Originally an initiative conceived by London Music Masters, the DIME steering group has broadened

to include Pro Bono Economics, Sound Connections and Project Oracle. The group members

represent music organisations including NPOs, universities, Learning and Participation departments

for orchestras and venues, music education charities and festivals, as well as individual academics,

researchers and musicians. A full list of organisations represented can be found in appendix 1.

2. Why is impact measurement and evaluation important to the music education sector?

Evaluation in the arts sector

The arts sector is a notoriously challenging context for producing robust evidence, given the often

intangible nature of the outcomes they seek and cuts to funding. However, many organisations are

making progress in new and innovative ways, and generating evidence that both informs their own

practice and helps provide funders with the information they need to spend their money wisely.

A theory of change is a means of articulating what an organisation wants to achieve in terms of

outcomes for the individuals it works with, and considering whether the activities it undertakes are

likely to achieve those outcomes. The process of creating it can help to focus and refine an

organisation’s mission.

Demonstrating Impact in Music Education

Understanding and demonstrating impact paves the way to maximising impact

Impact evaluation – the process of collecting and analysing data relating to those outcomes – is a

critical tool for the sector, helping organisations to understand whether the change they are

expecting is actually occurring. Recognising and tracking a project’s key outcomes helps identify

what is going well and what could be improved so the service can be developed and refined. Music

organisations that are focused on impact measurement and evaluation better understand the

projects they run and so are in a better position to deliver those projects effectively.

Robust evidence can help secure funding

When budgets are tight, spending time and money on evaluation may seem extravagant. However,

with less money available, questions such as "what works?", and “how are we going to get the

evidence?” become all the more important. Organisations – in all sectors – need to be able to

answer these questions confidently. Impact measurement and evaluation make this possible,

empowering music organisations to communicate positively the value of their work and increase the

likelihood of further funding.

Staying interested

Good tools are essential to good evaluation. The best tools and evaluation designs are co-developed

by delivery organisations - who really know about the outcomes in question - supported by

professionals (in-house or external) with expertise in impact measurement and evaluation. So, in

order to produce the highest quality of evidence, music organisations must take an active interest in

evaluation and impact estimation. Good tools will generate more robust evidence allowing them to

better understand, improve and demonstrate their economic and social value, helping to secure

future funding and deliver the best possible outcomes for young people.

The Demonstrating Impact in Music Education group – supporting the music education sector to

develop new skills to better evidence outcomes

Recognising both the challenges and opportunities of evaluation and impact measurement within

the arts, London Music Masters brought together some of the key influencers in music education to

discuss the role of impact measurement in the sector in April 2015. The discussion asked:

 Can we tell stories about our work that appeal to the head as well as the heart?

 What role does statistical analysis of social impact have to play in the way we evaluate our

work and is it really the ‘silver bullet’ of impact reporting?

 Are we happy with the dialogue between arts organisations and funders/politicians/other

stakeholders around impact reporting?

We were keen to encourage a spirit of enquiry within the group discussion by asking more searching

questions such as “how do we know about our impact?”, and “how do we know something else

might not have worked better?”

It is not just the competition for funding which acts as the motivation behind better estimation of

impact, but also the knowledge that as educators, we have a duty of care to the children and young

people with whom we work. Time is a scarce resource – how do we know that something else might

not be a better use of pupil and teacher time?

Demonstrating Impact in Music Education

This initial discussion was a catalyst for two further gatherings (July 2015, November 2015) at which

over 30 organisations have been represented.

The first discussion aimed to discover whether or not there are commonalities in what and how

organisations measure impact.

As a result, five common themes were identified:

1. Workforce development (e.g. increasing diversity, efficacy etc. of workforce:

artists/volunteers/employees/boards)

2. Musical progression (managing trajectories from first access onwards)

3. Developing participants' soft skills

4. Developing participants' hard skills

5. Increasing levels of diversity and inclusion of participants/beneficiaries directly connected

to individual organisations

During the second meeting (July 2015), the group drilled deeper into three of the five themes

(workforce development; musical progression; and ‘soft skills’) and began to consider how best to

address each. We selected these particular themes for further discussion as there are many, widely

used tools to assess practical music skills (or hard skills), and similarly there is much existing data

around diversity specifically pertaining to age, gender, ethnicity, disability, and socio-economic

background.

Soft Skills

The majority of this session was a practical discussion of approaches to measuring soft skills – i.e.

“personal attributes that enable someone to interact effectively and harmoniously with other

people” – that music educators believe a musical education can help to develop in pupils. This was

led by Dimitrios Tourountsis (at the time Director of Learning at London Youth).

London Youth provides an interesting case study for the music education sector and a potential

model of working with regards to the evidencing of impact of social or sport programmes for young

people. In the session Dimitrios highlighted the different, successful approaches to measuring soft

skills, drawing our attention to the importance of using robust, validated techniques and tools in

order to increase our confidence in the results.

Workforce Development

Two main areas were discussed in this group:

1. Reflective practice: does it work? We felt that whilst there is anecdotal evidence to support

reflective practice, and it is a tool used widely, there is not yet a body of evidence to support

its efficacy. Therefore, it is hard to discern whether it is indeed an effective workforce

development tool.

2. There was a desire for increasing collaboration between academics and practitioners, and

growing awareness amongst practitioners of academic discussion pertaining to their work.

We discussed finding ways to measurably increase the ‘quality’ of conversations between

practitioners which some had observed as they brought insights gained from personal

research into their practice.

Musical Progression

The group explored definitions of progression, acknowledging the difficulties in comparing

progression routes in formal and non-formal music education. They also looked at tracking routes

from first access through to conservatoire, discussing the role of Music Education Hubs in reporting

Demonstrating Impact in Music Education

and monitoring progress.

There was a short presentation on diversity and inclusion, seeking to broaden the discussion beyond

the common frames of reference.

3. What next? A call to action to galvanise the music education sector

The Demonstrating Impact in Music Education group has made some positive steps since its

inception. Having identified five common outcomes themes, and learnt from the experiences of

London Youth, DIME steering group recommend that the members work towards a set of standard

evaluation tools that can be effectively used by projects and programmes across the music

education sector. DIME proposes initially that one tool per outcome theme is identified and trialled:

 Soft outcomes – The Life Effectiveness Questionnaire

 Hard outcomes – to be identified

 Workforce development – to be identified

 Progression routes – to be identified

 Diversity and inclusion – to be identified

The development of critical thinking skills and increasing knowledge are key to this process of

identifying and trialling tools, and to increasing the effectiveness of the sector in measuring impact

more broadly.

Ultimately the vision is that by working collaboratively, pooling information, expertise and resources,

projects and organisations across the music education sector will be able to provide more convincing

evidence of their impact to a wider range of stakeholders, and learn how to better deliver their

stated aims. We see this happening in three stages:

1. Conducting a needs-analysis of DIME group members.

2. Making capacity-building available and accessible to DIME group members, and modelling

this across the sector.

3. Assembling a critical mass of organisations working in similar ways, and with similar

approaches to demonstrating impact in order to affect sector-wide change.

Next steps

In 2017 the DIME steering group will be circulating a needs-analysis questionnaire to the wider DIME

group and announcing opportunities for bespoke training.

Concurrently, the steering group will be speaking with influencers at the major trusts and

foundations to better understand their perspective on demonstrating impact, and developing a

constructive dialogue with DIME about co-designing reporting structures that support the common

themes we have identified in our work.

The DIME steering group welcomes contact from anyone keen to be involved.

Demonstrating Impact in Music Education

Appendix

1. Organisations represented in DIME:

DIME Steering group | May 2017

The DIME steering group comprises senior leaders from: London Music Masters, Pro Bono Economics,

Project Oracle and Sound Connections.

Employees and leaders of the following organisations have participated in one or more DIME event:

Association of British Orchestras

Pro Bono Economics

Creative Futures (UK) Ltd

Roundhouse

Drake Music

Sistema England

English Folk Dance and Song Society

Sound Connections

Guildhall School of Music and Drama

Southbank Centre

Junior Academy RAM

Spice

Kinetika Bloco

Spitalfields Music

Lambeth Music Network

St Mary's University, Twickenham

London Music Masters

The National Youth Orchestra of Great Britain

London Philharmonic Orchestra

Tomorrow's warriors

London Symphony Orchestra

Tri-borough Music Hub

Music For Youth

Trinity Laban

Musicians (Freelance)

Wigmore Hall Learning

National Youth Orchestra of GB

2. Learning from London Youth – impact measurement in the youth sector

London Youth has invested considerably in developing impact measurement tools and building a

consistent approach across youth centres and youth work in London. Their work acts as a

benchmark for effective sector-wide practice and utilises tools that have potential application within

music education contexts.

You can find out more about London Youth’s approach in their 2014-15 Learning Report.

3. The big picture – what initiatives, resources and tools are available?

The DIME group hopes that through bringing organisations together and finding common solutions,

impact measurement within music education will become stronger and more cohesive. Ultimately

this would mean organisations can better review and improve their work, and better make their case

to funders and policy-makers.

The DIME initiative is one of a number, so some organisations may prefer to seek support elsewhere.

Here is a summary of places you can go to find out more:

Demonstrating Impact in Music Education

Project Oracle

Project Oracle empowers the youth sector to achieve the best possible outcomes for children and

young people. They support the youth sector to produce, use and share high-quality evidence so

that together they can make better decisions to improve how interventions are funded and

delivered.

In May 2015 Project Oracle brought together funders, academics and youth organisations to launch

the Arts Cohort, beginning the pilot phase of our brand new Impact Pioneers programme.

Four key areas of learning that came out of the pilot were: using outcomes frameworks; creating

high-quality tools; engaging funders in evaluation; and sustaining peer-to-peer learning.

Read the recently published Learning Report available for download at project-

oracle.com/support/resource-library/ or contact Project Oracle at inf[email protected]m if you

are a music organisation in need of impact evaluation support or if you would be interested in

joining a future Arts Cohort.

Youth Music

The National Foundation for Youth Music is a national charity investing in music-making projects for

children and young people experiencing challenging circumstances. They invest in music-making

projects where they’re really needed. Every Youth Music project measures its impact, helping us

build a unique national overview, through their Impact Reports.

Youth Music Outcomes Framework

Young Foundation

The Institute for Community Studies (ICS) was set up by social entrepreneur Michael Young in 1954.

The ICS was an urban studies think tank which combined academic research and practical social

innovation. In 2005, it merged with the Mutual Aid Centre and was renamed The Young Foundation,

in honour of its founder. In both current and previous incarnations, The Young Foundation has been

instrumental in leading research, driving public debate, and implementing social innovation in the

UK and abroad.

The Young Foundation published a Framework of Outcomes for Young People in July 2012. It is

designed to highlight the fundamental importance of social and emotional capabilities to the

achievement of all other outcomes for all young people. Click here to download the Framework

Inspiring Impact

Inspiring Impact is a UK-wide collaborative programme, working with the charity sector to help

organisations know what to measure and how to measure. It is a programme that aims to change

the way the UK voluntary sector thinks about impact.

Inspiring Impact Hub is a one-stop shop for impact resources and tools. It pulls together the widest

possible range of resources relevant to improving impact practice, and enables users to search and

filter results according to their needs.

Demonstrating Impact in Music Education

Click here to view the Inspiring Impact Hub

Sound Connections and Creative Futures

Sound Connections commissioned action researchers to conduct in-depth research about issues and

themes highlighted in their research project Taking Off, which aimed to map the musical progression

routes of young people in London faces challenging circumstances.

One of the action research themes responded to a recommendation from the project: “The success

of progression in this context should be measured based upon the individual circumstances of the

participants, their personal desires, needs and capacities.” Sound Connections commissioned Julian

Knight, Creative Director of Creative Futures, to research the effectiveness of accrediting soft

outcomes for projects targeting children and young people in challenging circumstances. His report

tackles what we mean by ‘soft outcomes’; looks at the current funding landscape, and why

measuring outcomes matters; and shares a review of the existing tools available for measuring soft

outcomes.

Click here to read Julian’s report

The Centre for Youth Impact

The Centre for Youth Impact is a community of organisations committed to working together to

progress thinking and practice around impact measurement in youth work and services for young

people. The Centre builds and supports this community, and creates space for organisations to come

together to understand and increase their impact.

The Centre for Youth Impact, working with New Philanthropy Capital (NPC) and UK Youth, will be

holding two workshops in September to progress discussions on a potential collective impact

initiative for young people in England.

Click here to learn more about the workshops

The Collective Impact Framework

Collective Impact is a framework to tackle deeply entrenched and complex social problems. It is an

innovative and structured approach to making collaboration work across government, business,

philanthropy, non-profit organisations and citizens to achieve significant and lasting social change.

The Collective Impact approach is premised on the belief that no single policy, government

department, organisation or program can tackle or solve the increasingly complex social problems

we face as a society. The approach calls for multiple organisations or entities from different sectors

to abandon their own agenda in favour of a common agenda, shared measurement and alignment of

effort.

Arts Impact Assessment

Arts Impact Measurement is a website that brings together the collective expertise and thinking of a

range of leading cultural organisations. They team behind it has debated the importance of assessing

Demonstrating Impact in Music Education

the impact of the work with children, young people and the wider community, and has explored

ways of assessing it, drawing on the wider context of the charitable sector.

The website exists to help arts professionals find resources and information on how to measure the

impact of their work in the arts, especially with young people.

The Life Effectiveness Questionnaire

The Life Effectiveness Questionnaire (LEQ) was reviewed by DIME member Julian Knight (Creative

Futures) in a recent review of soft outcomes measurement tools commissioned by Sound

Connections (see above). Julian summarises the questionnaire as follows:

The LEQ was developed at the University of Canberra, Australia. Its lead researcher James Neill

describes ‘life effectiveness’ as closely related to the notions of ‘personal skills’, ‘life fitness’,

‘practical intelligence’, ‘personal competence’ and ‘self-efficacy’. The LEQ focuses on measuring the

extent to which a person's actions, behaviour, or feelings are effective in managing and succeeding

at life, or more broadly, a person’s ability to “adapt, survive and thrive”.

The standard version of the LEQ has 24 statements, three for each of the eight ‘domains’ or ‘factors’

listed below. In total, however, there 36 different domains to choose from, and many more in

development, so assessors can design their own customized version of the tool to fit the evaluation

needs or projected outcomes of their intervention.

The eight ‘factors’ or ‘domains’ assessed in the standard LEQ are:

 Time management

 Social competence

 Achievement motivation

 Intellectual flexibility

 Task leadership

 Emotional control

 Active initiative

 Self-confidence

A composite score is created for each individual using tools available on the LEQ website.

Although developed for adults, this self-reporting tool can be used from the age of about 12 years

upwards, and there are also versions of each domain with only one or two questions (rather than

three) for younger participants. Responses to the statements are given through an 8-point Likert

scale

The LEQ is designed for measuring the areas of personal development which are typically targeted

by intervention programmes, and although designed with outdoor activities in mind is also well-

suited to arts, creative, sporting or other activities.

The website is comprehensive on how to use and understand the LEQ, and with references to

numerous studies which have used it.

Demonstrating Impact in Music Education

SUITABILITY FOR THE ARTS SECTOR: This tool is a comprehensive one, and can be tailored to the

specific outcomes of a programme, and it is free to use. As with any relatively complex tool, it will

require an investment of time to understand it, tailor it to your own needs, and analyse the data –

but specific training is not needed. It has the advantage of covering any and all of the likely soft

outcomes areas which arts interventions are likely to impact, and to be usable with groups of

participants since it is based on self-reporting.