Evaluation

Continuous improvement should be a primary concern of every learning program. Continuous improvement is the process by which you regularly, systematically, and honestly evaluate what your program is doing well and what it needs to improve. Today, more and more funders are requiring this kind of comprehensive evaluation as a condition of continued financial support. 

Early in the process of establishing your program, you should begin to set up a system of ongoing evaluation. A strong evaluation design will provide information that you can use daily to fine-tune your program.

To understand the continuous improvement process, it is important to understand the terms of evaluation. Evaluation is the process of analyzing data to assess what works and what does not work in achieving goals. In this context, the word "goals" is synonymous with "objectives," "outcomes," and other terms your program may use to convey changes you expect or desire.

Data collection is only a part of full evaluation—one step among many. The evaluation forms that staff and participants are asked to complete are tools for data collection; they are not the evaluation itself. To be useful, data collection has to be followed immediately by analysis. This analysis drives the continuous improvement process.

Outcome Measurement

Currently, most funders prefer an evaluation process called "outcome measurement," because they see it as the best gauge of how a program is meeting its objectives. Traditional models of evaluation focused on outputs, such as products delivered (e.g., classes held) and people served. Outcome measurement goes further: It takes outputs into account, but it focuses on how well these outputs produce end results (e.g., measurable improvements in reading ability).

Outcome measurement is not easy. It can be expensive and time-consuming, especially if it is not introduced into the initial design of the program. It requires you to establish appropriate and reasonable desired outcomes, which can be a complicated process. And if a program is not achieving its outcomes, outcome measurement doesn¹t always indicate exactly why.

On the positive side, however, outcome measurement can help with the following tasks:

    • Help an organization clarify its mission and ask of itself, "What exactly are we trying to accomplish, and how are we going to accomplish it?"
    • Increase staff and volunteer satisfaction by providing evidence, when appropriate, that the program is meeting its goals
    • Provide a steady stream of information and feedback to help with the continuous improvement process
    • Identify services that need more attention
    • Assist in developing and justifying program budgets
    • Display a program¹s achievements for board members
    • Help define a program¹s long-term goals.

In short, outcome measurement is big step beyond simple data collection. As a result, it is a considerably more involved process and yields considerably more valuable results. Given that today's funders are demanding more accountability from the programs they fund and that government agencies and national nonprofit organizations are moving toward outcome measurement as their primary method of evaluation, there is great incentive to explore it early in the process of planning your center.

Who Should Do the Evaluation?

As you begin to examine what system of evaluation is right for your center, you should answer the following questions:

  • Do we have the expertise on staff to design and conduct the evaluation?
  • Do we have the time to manage all aspects of the evaluation?
  • Are we able to look objectively at our work?
  • Will our funders permit us to do our own evaluation?

If you answer no to any of those questions, you might want to consider hiring an external evaluator or team of evaluators. An evaluator can be either someone who holds an advanced degree in evaluation or an experienced practitioner in your content area. Colleges and universities are a good source for external evaluators. There are private firms that specialize in program evaluation. It is also possible that your school district has people who are experienced in conducting evaluations. Whenever possible, work with an evaluator who is familiar with your type of program.

Emphasize the Ongoing Nature of Evaluation

An annual evaluation may allow problems to remain for an entire year before they become apparent and steps are taken to address them. To avoid this unnecessary delay, programs should make time and resource decisions that allow for ongoing assessment. Consider the following ways to continually integrate evaluation:

    • Provide structured time for program staff to collect important data and to meet and discuss the long-term goals of your program.
    • Make the review of evaluation results a regular part of meetings between program partners.
    • Establish a procedure by which changes will take place. Determine the "who, what, when, where, and how" for distributing evaluation results and other information. Gather suggestions for changes. Decide which ones to follow through on, and then make the changes.
    • Develop a strategy for keeping all potential supporters and participants informed about the process of change. All people who have a stake in the program will need to be notified of the evaluation results and informed about the process by which changes based on those results will be made.
    • Promote a culture in which the purpose of evaluation is to create continuous improvement. Although this may be easier said than done, programs can try to foster a positive attitude toward evaluation by celebrating the opportunity for improvement rather than bemoaning it as another bureaucratic requirement. Strong, positive leadership makes a difference in staff perceptions of evaluation activities.
    • Bring partners together regularly to discuss data in light of "big picture" issues. The daily demands of running a program lead to a "putting out fires" management approach. The process of data collection and analysis offers the opportunity to refocus on both collaboration and the underlying purposes of operating a program.

Adapted from Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), pages 4-5 and from Who Should Do the Evaluation? and Emphasize the Ongoing Nature of Evaluation from K.E. Walter, et al, Beyond the Bell: A Toolkit for Creating Effective After-School Programs, Second Edition (Naperville, IL: North Central Regional Educational Laboratory, 2001), page 13-14.