If you work for a non-profit you will inevitably be asked, probably by a funder, how you measure your effectiveness or impact. You might even be told you need to hire an evaluator.
But, what even is evaluation?
The formal answer to that question is that evaluation is collecting, analyzing, and using information about a program, initiative, strategy, or other effort. Put more simply though, it’s a way to find out if what you are doing is working.
- Are you reaching the right people?
- Are you those people getting full access to all the information or services you were aiming to provide?
- Did that information you gave or the service you delivered help?
Evaluation is a process you can use to assess the relevance, effectiveness, or efficiency of your work. You might also use what you learn to make improvements or make judgments about its quality or impact.
Some of what you might learn by evaluating a program depends on the program’s stage when it’s being assessed.
At the start of the program, an evaluation typically focuses on understanding how to optimally design the project to best meet its goals. For example, this might be as simple as finding out that one day of the week is better than another to hold a small group program and that the optimal number of people is more or less than you had planned. It could also be as complex as using two different program curricula and comparing the progress of the groups to decide which program you want to use in future groups.
As the program progresses, the evaluation looks to see how the initiative is taking shape, where there is early progress, and how to maximize the ongoing success of the project. So, continuing our small group program example, you might start to track attendance and active participation to see who is getting full exposure to the program and who is not. You can then make adjustments to the program to ensure that everyone gets the full benefit.
At the end of the program, the evaluation assesses the extent to which project aims were met and identifies circumstances that led to both high and low levels of success. This is probably the stage most people think of when they think of evaluation. It’s where you measure how well the program was functioning and how big of a difference it made. Our small group program participants might fill out a questionnaire giving feedback and measuring whether the intended program outcomes were achieved.
Together each of these pieces helps to tell the program’s full story. When done well, and planned in advance, program evaluation can provide essential information to inform every decision in an organization’s programs and operations, including revisions to strategic goals, resource allocation, program modification, redesign, implementation, and policymaking.
Evaluation Uses
- Improve design and implementation
- Monitor progress toward goals
- Demonstrate impact
- Ensure that effective efforts are maintained
- Ensure resources are not wasted on ineffective efforts
Types of Evaluations
As you may have guessed by now, each of the different stages of evaluation requires a different type of evaluation. There are formal terms for the different types of evaluations that might be needed. The different purposes of, and motivations for, conducting evaluations determine the different types of evaluations.
We touched on these types of evaluations already, though, we didn’t call them by their formal names:
- Formative: Assesses early stages of development or tests the work on a small scale before broad dissemination.
- Process: Examines the implementation and operation of a program, initiative, strategy, or other effort
- Outcome: Assesses the short and long-term results.
Most non-profit organizations will stick with these three. But there are more methodologically complex evaluations that are especially important when you are considering expanding the scale of what you are doing. These are:
- Cost-Benefit: Program outputs and/or benefits are compared to input costs to provide a ratio of cost to benefit.
- Impact: Investigates the immediate positive and negative changes produced by a program and usually corresponds with the measurement of intervention objectives.
We’ll go more in-depth about each of these evaluation types in future blog posts.
Upstream's Approach to Evaluation
Upstream’s mission is "to support systems level changes to promote public health, specifically related to the prevention and reduction of youth substance use, improved mental health of the community, and creating an environment conducive to healthy behaviors for all citizens".
We use and support the use of the Strategic Prevention Framework (SPF) as an overarching planning process for organizations, however, Upstream also believes in tailoring technical assistance, training, and evaluation partnerships to meet the local circumstances each organization faces in their community.
Our approach to evaluation is rooted in local usefulness, first and foremost, so that the results can be assessed and learning applied in context. Upstream believes in using a clear theory of change with a strong framework, and has a standard approach to get entities started.
However, with a focus on local usefulness, we have a starting place for a conversation to then encourage individual tailoring of the evaluation planning, execution, and reporting to best meet entities needs.