Section 4: How To Measure Your Impact

Whether you are working online or offline, measurement is key to understanding the impact of your program—to gauge how and how much your work is making a difference to the people, groups, organizations, or communities involved. 

Why measurement is important

Your instincts might provide you with a good sense of positive or negative outcomes, but only a proper impact evaluation will comprehensively identify the strengths and weaknesses of your work. Impact evaluations are important regardless of whether your activity is online or offline, simple or complex, cheap or expensive. 

Regardless of whether the impact is large or small, long- or short-term, positive or negative, intended or unintended, you need to know this information for several reasons.

1) Lessons for the future.

Well-designed impact evaluations allow you to understand what aspects of your activity worked better than others, and to some extent why. This will increase your ability to focus on the things that really matter and will tell you what you need to change in the future. 

2) Save time and resources. 

Impact evaluations can help you to make sure that time and money are well spent. They can help you maximize your return on investment.

3) Make it easier to attract funding.

Impact evaluations can help you protect ongoing funding and even attract new donors. Being able to provide evidence of your impact can help you better show the effectiveness and value of your program and demonstrate your capacity for project management and accountability. 

4) Become an agent for social change.

Impact evaluations are very powerful because they provide evidence that you can contribute to solving a significant social problem. This evidence can be used to show others that what you do matters. 

Impact evaluations can:

Impact evaluations might also help you to become part of a larger network of people, in your country and beyond, who advocate for similar policy changes. 

Ways to measure your impact

Finding the best tools to measure the impact of your program can be difficult. Here are some tips for finding the best evaluation strategy and choosing the best measurement tools for your project.

1. Use existing measurement tools.

Avoid, so far as possible, creating your own measurement tools. Research and select existing and proven measurement tools that were created by experts in the field of evaluation. You can find a library of tools to measure levels of violence or prejudice and related concepts (like fundamentalism, extremism, resilience) here—just scroll to the bottom of the page.

2. Partner with research professionals. 

A good option can be to invite a professional researcher to join the evaluation team. Collaboration between researchers and CSOs can be mutually beneficial: you will get important help to design, measure, analyze, and interpret the data, and the researcher will have the opportunity to apply their knowledge to real-world problems. Depending on ethical and security constraints, they may also be able to use some of the data for teaching and research.

3. Use a mixed-methods approach to measure your impact. 

This means gathering different kinds of data from different evaluation tools. You can look at how many likes, shares, and views your online content gets. You can also conduct an online survey to ask your audience what they thought of your content. You can interview your audience, via Zoom or in person, and ask them to give their opinions in their own words. Many good evaluations of social media campaigns adopt a mix of different approaches.

4. Discuss your evaluation plan with program stakeholders.

We suggest discussing your evaluation plan with selected stakeholders in a focus group or round-table discussion. Describe your evaluation plan and ask them what should be measured and how. Their different points of view can provide important guidance for your evaluation. This sort of consultation can also generate goodwill for your project, because your stakeholders will better understand what you’re doing and will feel that they’re part of the initiative.

Your stakeholders can include members of the relevant communities, government representatives, influential citizens, and activists. 

Using real-world indicators

We need to be able to clearly define the boundaries of concepts like extremism, hate, tolerance” and resilience so that we can measure them effectively. Let’s imagine that we want to measure tolerance for religious minorities in a religious Facebook group. Do we only count as markers of intolerance calls for violence against religious minorities, or do we also include openly offensive remarks? At the same time, what are the markers of tolerance? Are we only looking for open expressions of respect and appreciation for religious minorities, or do we also count expressions of ambivalence? Choosing to count only one or all types of expression as markers of tolerance or intolerance could significantly change the results of our evaluation.

To transform the idea of tolerance into real-world indicators, you need to be able to measure tolerance. You need a measurement scale. To give a simple example, you could employ a scale in which the following values are assigned:

On the negative side: 

–10 tolerance points = a call for violence.

–4 hate points = an expression of social distancing.

On the positive side: 

+1 tolerance points = an expression of ambivalence (“I am not opposed to Hindus”). 

+10 tolerance points = an open expression of appreciation for a religious minority (“I really like and respect Christians”).

 Points can also be assigned to different actions people may take, not just what they say. For example, writing a post that shows signs of tolerance or intolerance toward religious minorities could be assigned a certain point-value,  while liking or sharing such a post would count for half the points of writing it. Whether you aim to measure tolerance, discrimination, gender-based violence, resilience, trust, or any other concept, you need to assign different values to different terms.

Using comparison groups

There are two main elements that should be included in your evaluations  whenever possible: pre-post intervention design and comparison groups.

The idea behind a pre-post intervention design is simply checking a situation before a project begins and checking it again after the project ends. For example, a runner could record their time before starting a new, three-month training program. They could then record their time at the end of the three months of training. This would be a pre-post intervention design. Similarly, if you want to know whether your intervention to reduce antimigrant discrimination on Facebook worked, you need to measure levels of discrimination before and after your intervention and check for differences.

Comparison groups are made of people who do not participate in your program at the time of the evaluation. You need to collect data from them too. This allows you to check what would have happened if you had not run your project. 

Comparison groups allow you to control for other things that might affect your program. For example, what if, during your intervention to improve tolerance toward religious minorities on Facebook, a member of a religious minority were to commit a heinous crime against a local girl? It is very likely that this act, and the media discussion about it, would increase the level of intolerance toward religious minorities on Facebook. In this case, having a comparison group would be the only way to measure the impact of your program.