Indicators 101: How Data Scientists Measure Change

At Nicholson Consulting, we partner with many organisations that are dedicated to making Aotearoa a better place. From te reo Māori revitalisation, to providing opportunities to rangatahi, to improving the health and wellbeing of our communities – many of the organisations we work with aspire to change things for the better.

But how do we know that the mahi we support is leading to the changes we all want to see? One tool that we can use to measure whether things are on the right track are called indicators. Read on to learn more.

What are indicators?

Indicators are data points that can be tracked over time to show – or indicate - whether a change has happened. They track things that can be difficult to measure, from the delivery of services to social change. The understanding we get from monitoring how indicators change over time provides important feedback that can shape the design of policies, services, and initiatives that are implemented to address the biggest challenges facing our communities.

To illustrate how indicators work, let’s say you have a goal of improving your fitness. To measure whether your fitness is improving or not, you may rely on indicators, such as whether the number of kilometres you can run increases over time. If this distance increases over time, you might conclude this as a good indication that your fitness is improving.

Indicators are great tools that can help us track our progress towards a goal or change we want to see. In a monitoring and evaluation context, we call these goals or changes outcomes.

Indicators can help us understand:

• If we have achieved, or if we are heading in the right direction towards a desired outcome

• If our actions are leading to the outcomes we want, or if we need to modify our approach

Here, it's important that we distinguish between two main types of indicators that are often referred to within a monitoring and evaluation context.

1) Output indicators

These indicators measure outputs, which are the results of the activities or interventions we carry out to achieve an outcome. Output indicators measure things like whether a service has been delivered, or whether a certain action has been taken. Output indicators demonstrate whether we’re on track to our bigger picture goals, or the outcomes we’re aiming for.

2) Outcome indicators

The indicators used to track progress towards outcomes are called ‘outcome indicators’. Outcomes are the bigger changes we want to see.

Logic flow:

While both types of indicators are important, for this article we will be focusing on outcome indicators and how they’re used to demonstrate change over time.

Selecting the right indicators to measure an outcome can be tricky as often the outcomes we want to focus on don’t have a straightforward way of being measured.

For instance, using the fitness example above, there are many outcome indicators that could be used to measure fitness. Is it about running speed, step count, muscle mass, or the number of push-ups someone can do? The criteria for success, desired results, and activities / interventions (outputs) will differ drastically depending on whether muscle mass or running speed is selected as the indicator to track against.

The challenge of selecting the right outcome indicators becomes more apparent when we start to think about outcomes such as health, wellbeing, or quality of life. There are many ways to interpret and think about each of these outcomes, with each interpretation leading to different solutions and ideas of what success looks like. As a result, people will not always agree on what outcome indicators are best to use – but choosing the right outcome indicators is crucial for shaping our understanding of how we address challenges that face our society and for tracking the progress we make towards achieving desired outcomes.

How do we develop the right outcome indicators?

When we work with organisations to develop outcome indicators, the first step is to identify the outcomes we want to track (often these outcomes come from an existing framework). After that it’s about prioritisation. This is a consultative and collaborative process where we work closely with stakeholders to determine which indicators will be most useful and relevant in relation to measuring and monitoring the desired outcomes.

Some important things to consider when developing outcome indicators are:

• Does the outcome indicator have a robust link with the outcome of interest?

As illustrated in the fitness example above, it is important that the outcome indicators we select are appropriate for the outcomes we want to achieve. Something that helps us select the right outcome indicators is defining a clear criterion of success for our outcomes from the outset.

Theory of change frameworks are another useful tool for creating robust outcome indicators. Theories of change articulate the reasoning behind how an initiative’s activities and outputs lead to its outcomes. Understanding the theory of change behind an outcome can help better align our indicators with the outcomes we want to measure.

• Is there data available for this outcome indicator?

For an outcome indicator to be useful we need a way to source the data required for the outcome indicator, and for that data to be updated frequently so we can track changes over time.

• Measuring outcomes and not just outputs

Ideally, indicators won’t just track the delivery of a service or activity (i.e. outputs) and use that as evidence of progress towards our desired outcomes. For instance, just counting the number of people who attend a mental health and wellbeing workshop won’t measure the impact that workshop had on improving participants’ overall mental health and wellbeing.

However, at times output data is all we have to work with. When this happens, it is good to highlight why the output can work as a proxy indicator for the outcome we are measuring. I.e., why it is reasonable to believe that this output would impact the outcome we are measuring.

• Should certain indicators be prioritised / given more weight when interpreting the data?

There may be some indicators that should be weighted differently when interpreting data. This could be for several reasons such as the quality of data behind an indicator, how strong the link is between the indicator and the outcome of interest, or if a particular indicator is of special interest to stakeholders.

• Ensuring that indicator data is interpreted accurately and consistently by those using your framework

This may involve providing additional information for certain data points or explaining gaps in the framework due to a lack of available data so that people have the context they need to interpret the data correctly.

• Ensuring that the selected indicators promote equitable outcomes and are not exacerbating existing inequalities

This involves ensuring that indicators reflect the aspirations and understandings of success held by the communities which are impacted by the targeted outcomes.

• Ensuring that existing legislation, organisational strategies, and frameworks are considered (particularly relevant in a government context)

This could include any outcome indicators that are required to be monitored by legislation or could include any organisational or strategic priorities that should be considered when selecting indicators.

Indicators are a fantastic tool for measuring progress.

However, there are some limitations to be aware of. The story indicators tell will always be shaped by factors such as the availability of data, the way data is collected and updated, and the biases, priorities and worldviews of the people and organisations creating indicators. It’s vital we stay mindful of the gaps that may exist in these frameworks. This is especially true for indicators that focus on big, broad issues such as health, wellbeing, climate change, economic prosperity etc. where different values, worldviews and cultural frameworks can lead to different understandings of what success looks like.

When all things are considered, indicators can be a fantastic method for tracking change over time and telling a positive story of change for the better.

To learn more about how indicators could strengthen a policy, service, or strategic initiative at your organisation, get in touch by emailing hello@nicholsonconsulting.co.nz!