Moving to Meaningful Measurement: Three themes from our third Reimagining Government webinar of 2021
Our current system of measurement in the public sector is not working. A focus on metrics designed for control can result in gaming, perverse incentives, and ultimately makes the job of public servants harder. But what does the alternative look like?
In the third instalment of the 2021 Reimagining Government series, ANZSOG and CPI hosted an interactive webinar that explored the shift to meaningful measurement — the movement away from using measurement as a means of control — and instead using it as a way to learn about complex problems and the people experiencing them, to adapt and improve approaches.
The panel was facilitated by Jane-Frances Kelly, Deputy CEO, Education, ANZSOG, and included Adrian Brown, Executive Director, CPI; Sara Fernandez, CEO, Oxford Hub; and Professor Jenny Lewis, Professor of Public Policy, School of Social and Political Sciences, The University of Melbourne.
Our current way of measuring isn’t really working
Jane-Frances explained that under New Public Management, those at the top of the hierarchy set quantitative performance measures, and manage those at lower levels to achieve those measures. This top-down accountability can be characterised as “measurement for control” — and our panellists agreed that this way of measuring isn’t really working.
In complex environments, measurement can often be an illusion of control. “The map is not the terrain”, said Adrian. “When we look at spreadsheets and PowerPoint slides, we’re looking at an abstraction of reality. But it’s easy to forget that, because the conversation becomes about the spreadsheet, and not about the reality informed by the spreadsheet.”
As well as often being ineffective, measurement can also be damaging. Jenny pointed to the example of research performance measurement systems in higher education — while certain metrics started with good intentions, they were now having widespread negative effects: “It’s doing damage to people’s careers, because of the obsession with particular numbers. The best thing we could do for people working in universities is stop measuring everything we do.”
In this way, panelists agreed that measurement for control is ineffective, and dangerous. When done the wrong way, measurement can obscure the truth, create obsessive behaviour, and prevent us from learning.
There are psychological and structural barriers preventing a transition to meaningful measurement
Our panelists agreed measurement shouldn’t be used for top-down control, and instead be used to learn — but acknowledged the significant barriers in place that must be overcome.
Mindsets are one key barrier to measuring meaningfully. Instead of viewing measurement as something that we can learn from, a widespread belief is that measurement is something that we can use to punish and reward people, to control outcomes.
Any large organisation — and public systems — are built on a set of assumptions and mindsets that aren’t always talked about. Adrian pointed to one contradiction: “most people in their hearts don’t believe that data reveals a whole truth”, he said, “but our public organisations do have that assumption built in”.
Sara called out the more tangible, structural barriers to meaningful measurement that she saw in her work. Often there was a lack of alignment of what measurement should look like, and what should be measured, across the funding chain — measurement felt like an afterthought.
As well as what was measured, who measured was another barrier. Sara often saw people in positions of authority, far away from the detail of issues, “deciding what needs to be delivered, what needs to be measured, what is important and where the value lies”.
These barriers were often supported and exacerbated by legislative frameworks — for example, legal requirements for accountability of public expenditure set measurement agendas. But Adrian argued that just as those frameworks are created by humans, we can “unlearn them”, creating something better if we wanted, especially at a local level, where we’re able to be more flexible and creative.
We need to experiment with what meaningful measurement actually looks like
While panelists agreed that data should only be one piece of the measurement puzzle, panelists emphasised different ways of ensuring we’re meaningfully measuring — highlighting the need for experimentation.
In Sara’s work, she found success with specific tools and processes. Learning pods helped to instil continuous reflection, and the building of a more learning-oriented culture. Using the most significant change (MSC) technique, a participatory method based on stakeholder narratives, Sara was able to bring in the voices of people closest to the issues at hand — creating spaces that frontline workers and citizens found valuable, instead of generic top-down evaluation approaches which weren’t always valuable, and didn’t always result in learning: “I’m not saying we shouldn’t measure at all, but I’d like us to be more thoughtful about whose power, and whose voice comes out in measurement.”
Jenny suggested a more place-based approach to measurement — having a small number of indicators that don’t take too much trouble to collect, and then for different groups and areas, asking what was truly important to them, and then measuring those things. She stressed the need to not just measure for the sake of it, but measuring to learn and improve — “getting towards some kind of balance and not going through the motions simply because we have to”.
Adrian suggested that a lot of the problems we currently face come from the “eternal desire to do things at scale”. Our public systems are large and hierarchical — built for scale — and so require “simple, clear, comparable, consistent sources of information” to measure and manage. Adrian suggested challenging the notion of scale — because that’s what necessitates the current paradigm of measurement. He suggested thinking about accountabilities and relationships nested lower in the system — so we could start to have “more meaningful, more human, more rich conversations that weave data and stories together”.
***
The current paradigm of measurement just isn’t working. At best, top-down measurement for control is ineffective, and at worst it’s damaging and demoralising. We should instead seek to use measurement to learn about complex problems and those that experience them, so that we can continuously adapt and improve our approach.
There are barriers that are preventing an orientation towards meaningful measurement, such as implicit assumptions, hierarchical structures and legislative frameworks. But there are also tools, processes and questions we can use to enable our journey towards a more meaningful, learning-centred approach to measurement.
Sara summed up some of the guiding questions we should use when measuring — an important reminder when engaging in complexity:
“Are you involving different voices? Are you shifting power? And are you being creative in the way that you’re doing it?”
The Australian and New Zealand School of Government and the Centre for Public Impact are hosting an interactive webinar series, content hubs with curated resources (including a hub for meaningful measurement!) and a learning community. Find out more here.
