Understanding what works
We are constantly building our evidence base to demonstrate the considerable impact of the Citizens Advice service: how we meet a need in society and have value for the individuals and communities we work with, as well as government and wider society.
But we don’t just stop there, as we want all all our direct service delivery interventions to be as effective as possible at helping clients find a way forward.
Our approach to what works
At Citizens Advice we are committed to understanding what works and ensuring organisational resources are responsibly invested into worthwhile services.
This is the right thing to do for our clients and society. It also enables our organisation to embed learning from delivering and measuring effective services:
How do we do it?
Our Impact and Evaluation team are responsible for customer insight research, assessing service effectiveness through evaluating what works, and using our evidence and analysis to understand and demonstrate the impact and value of the Citizens Advice service.
Evaluations are designed to ensure that Citizens Advice is equipped to make evidence-based decisions about the effectiveness of direct services.
Evaluations are primarily delivered in-house by our Impact team, as their understanding of Citizens Advice processes is essential to ensuring that evaluations meet organisational needs. It also enables us to build-up our internal learning of the most appropriate evaluation approaches, best practice and tools.
The impact and evaluation function is independent of those delivering our services, providing additional credibility to evaluation findings.
Where an evaluation requires external validation or requires a specialist approach (e.g. evaluating domestic violence intervention efficacy) we will engage the services of external evaluators.
Each evaluation is custom-designed to measure what matters, but will often focus on looking at understanding advice processes and resources used, as well as the needs of our clients and what outcomes we achieve for them.
Crucially, as an organisation that does so much for so many, our evaluations need to capture the nuance of what works for which clients and under what condition
We have shared outcomes measures
Citizens Advice is made up of a network of 259 individual local charities, and we have designed and implemented shared measures to systematically understand the outcomes we achieve for our clients.
We have shared outcome measures with over 800 outcomes codes built into our centralised case system management system, allowing advisers to record what happens to a client as a result of advice. Last year over 400,000 outcomes were captured this way.
These outcomes codes are reviewed annually to make sure that our measures still best-fit the likely issues and outcomes our clients experience.
We support local Citizens Advice to design and deliver outcomes research
As well as supporting local Citizens Advice to adopt shared outcomes measures, the Impact team also produce toolkits on evaluating specific interventions undertaken locally.
This includes theoretical and practical considerations around evaluation and research good practice.
Read more on our approach to understanding what works for Citizens Advice (300 KB)
What we’ve learnt: Financial capability
We support thousands of people each year who need help with money problems. That’s why we want to understand the role of people’s financial capability - or as we call it, money skills - so we can provide more holistic, appropriate services and understand what does and doesn’t work to help people find a way forward with their money skills.
Money Talks - Evaluating life event focused money guidance
Money Talks - Evaluation Report.
In 2017, we ran a service throughout Wales which focused financial capability support around key life events. People were referred to the service if they were experiencing a period of ill health, relationship breakdown or change in employment.
Clients' financial capability was measured across a range of questions in their first Money Talks session, and then again in a second follow up session, enabling us to track any changes in people's money skills.
We ran a comparative survey of 3,300 people across England and Wales so that we could compare the financial capability changes of people who received Money Talks to those who had just had face to face advice.
We found that financial capability increased significantly for Money Talks clients across all 13 measures, and that these increases in scores were significantly greater than in the comparison group.
We also conducted a process evaluation, where we explored what worked well and what didn't in terms of referrals, service design and the delivery of sessions.
Full report: Money Talks - evaluating life event focused money guidance (3.42 MB)
Understanding our client’s baseline financial capability
For our national report Understanding money skills (4.86 MB) (2017), we surveyed over 1,000 face-to-face local Citizens Advice clients from across England and Wales on their baseline money skills. This research helps us understand the complex nature of our clients’ financial capability so we can provide more holistic, appropriate services that help people find a way forward.
Our approach to measuring financial capability
As part of a year-long project, we developed a framework for evaluating our services’ impact on our clients’ financial capability. In order to do this, we carefully researched and developed a set of measures that are designed to suit our clients and the nature of the services that we deliver, while aligning with the UK Financial Capability Strategy. Learn about how we did this from our paper Measuring Financial Capability (2.09 MB) (2017) which includes a foreword from Sharon Collard. The below diagram gives an overview of the progress of this research project. The methodology we are using to develop our financial capability measures
Next steps
The process of developing financial capability measures that are fit for purpose for different service settings is an ongoing project for us. If your organisation would be interesting in the testing and development of our measures, or would like to use them to help evaluate the impact of your project, please get in touch with impact@citizensadvice.org.uk.
Previous work
We have completed a number of evaluations in the past aiming to build an understanding of our clients’ financial capability and what our impact has been in this area:
Quids In (633 KB) (2012) - We found that social housing tenants were £10 a week better off than the comparison group as a result of some targeted financial capability services we delivered to them.
Managing Migration pilot (149 KB) (2013) - We assessed how prepared our clients would be to use the new Universal Credit system. As part of this, we looked at clients’ financial capability. Focussing on skills, behaviour and mindset in relation to budgeting, banking and staying informed.
We wanted to take a broader view on how some of our biggest services, such as debt advice, could have an impact on our clients’ financial capability. This led to a number of pilots of different services, such as debt advice incorporating a financial capability element, which we evaluated:
Integrated Money Advice pilot 171 KB (2014)
Better Financial Health pilot 156 KB (2015)
We also have a financial capability team who carry out similar research and evaluation of our interventions along with the impact and evaluation team and other partners.
What we’ve learnt: Integrating advice in primary care settings
Practical problems such as debt and housing concerns have place pressures on GPs. 19% on average of GP’s consultation time is spent on patients’ non-clinical issues.
Our research, Advice in Practice (2.3 MB) – in collaboration with the Royal College of General Practitioners – highlights the crucial role that integrated support services can play in helping both patients and GPs.
Most GPs in England and Wales think that when patients receive help from advice agencies, this leads to positive effects: 75% said that there was a positive effect on patients’ health and wellbeing
GPs are an important beneficiary too: 61% said that it had a positive effect on their ability to focus and treat patients’ clinical issues, as well as on the number of repeat visits about the same non-clinical issue.
How people access advice makes a difference: GPs who used more integrated referral pathways were more positive about the effects.
Supporting clients who experience domestic and/or gender based violence and abuse
Our ASK Programme trains and supports advisers to help clients to disclose instances of gender based, or domestic, abuse by asking a proactive question when we give face to face advice in some settings.
We commissioned Lancaster University to carry out a 2 year independent evaluation of the programme. The report found that over 90% of clients were satisfied, or very satisfied, with being ASKed about their experiences of gender-based abuse. It also found that the advice and support given had a significantly positive impact on clients. The research also highlighted the importance of good partnership working including with local specialist support services.
What works in policy and practice
Understanding what works can extend beyond considering direct service delivery.
By listening closely to the people that come to us, using our real-time data and gathering insight and intelligence from clients and frontline advisers, we spot emerging issues with policies, practices and regulations that are not delivering for society. Our policy research teams have expertise in understanding what works in public policy and private practice, including proposing solutions to existing problems.