How To Define Real-World Indicators

Ideas such as hate, extremism, tolerance, and resilience can be difficult to define. They are subjective concepts that may be interpreted differently by different people in different places. In order to assess the impact of your project on values like these, you need to transform them  into concrete, real-world indicators.

Here we explain the main issues on the journey from abstract ideas to real-world indicators and provide some concrete ways to measure your online campaign.

From abstract ideas …. 

We need to be able to clearly define the boundaries of concepts like extremism, hate, tolerance, and resilience so that we can measure them effectively. Let’s imagine that we want to measure tolerance for religious minorities in a religious Facebook group. Do we only count as markers of intolerance calls for violence against religious minorities, or do we also include openly offensive remarks? At the same time, what are the markers of tolerance? Are we only looking for open expressions of respect and appreciation for religious minorities, or do we also count expressions of ambivalence? Choosing to count only one or all types of expression as markers of tolerance or intolerance could significantly change the results of our evaluation.

… to real-world indicators

To transform the idea of tolerance into real-world indicators, you need to be able to measure tolerance. You need a measurement scale. To give a simple example, you could employ a scale in which the following values are assigned:

On the negative side: 

–10 tolerance points = a call for violence

–4 hate points = an expression of social distancing

On the positive side: 

+1 tolerance points = an expression of ambivalence (“I am not opposed to Hindus”). 

+10 tolerance points = an open expression of appreciation for a religious minority (“I really like and respect Christians”).

 Points can also be assigned to different actions people may take, not just what they say. For example, writing a post that shows signs of tolerance or intolerance toward religious minorities could be assigned a certain point-value, while liking or sharing such a post would count for half the points of writing it. Whether you aim to measure tolerance, discrimination, gender-based violence, resilience, trust, or any other concept, you need to assign different values to different terms.

Concrete solutions to find out how to measure your impact

Finding the best tools to measure the impact of your program can be difficult. Here are some tips for finding the best evaluation strategy and choosing the best measurement tools for your project.

1.  Use existing measurement tools.

Avoid, so far as possible, creating your own measurement tools. Research and select existing and proven measurement tools that were created by experts in the field of evaluation. You can find a library of tools to measure hatred and related concepts (like fundamentalism, extremism, resilience) here—just scroll to the bottom of the page. 

2.  Partner with research professionals.

A good option can be to invite a professional researcher to join the evaluation team. Collaboration between researchers and CSOs can be mutually beneficial: you will get important help to design, measure, analyze, and interpret the data, and the researcher will have the opportunity to apply their knowledge to real-world problems. Depending on ethical and security constraints, they may also be able to use some of the data for teaching and research.

3.  Use a mixed-methods approach to measure your impact. 

This means gathering different kinds of data from different evaluation tools. You can look at how many likes, shares, and views your online content gets. You can also conduct an online survey to ask your audience what they thought of your content. You can interview your audience, via Zoom or in person, and ask them to give their opinions in their own words. Many good evaluations of social media campaigns adopt a mix of different approaches.

Use this handy “Checklist for conducting effective interviews” as a guide.

4.  Discuss your evaluation plan with program stakeholders.

We suggest discussing your evaluation plan with selected stakeholders in a focus group or round-table discussion. Describe your evaluation plan and ask them what should be measured and how. Their different points of view can provide important guidance for your evaluation. This sort of consultation can also generate goodwill for your project, because your stakeholders will better understand what you’re doing and will feel that they’re part of the initiative.

Your stakeholders can include members of the relevant communities, government representatives, influential citizens, and activists. 

See here for a handy infographic of things to consider when conducting evaluations during times of crisis or conflict. 

Contributed by Professor Greg Barton and Dr Matteo Vergani, Alfred Deakin Institute for Citizenship and Globalisation, Deakin University.