General Evaluation Resources
In our scan of evaluation tools and plans, we found a variety of materials including evaluation guides, reports and research papers, toolkits, surveys, questionnaires, rubrics, and certification checklists. Ranging from simple to complex, open-ended to highly circumscribed, these tools are not necessarily designed for youth media and technology programs, or even youth programs, but they provide information, methods, and tools that are applicable in some cases. In addition, the practitioners we heard from reported using portfolios and journals; tracking participation, competencies, and web site traffic; and carrying out other forms of monitoring and evaluation, such as participant interviews, in the course of their work.
Evaluation GuidesA host of service providers, grantmakers, and other nonprofit organizations have produced general and specialized evaluation guides for practitioners. These guides provide background information and steps to implementing an evaluation plan. Depending on the user's preference and needs, s/he can read through a guide from front to back, or just reference particular sections. Evaluation guides are especially useful for walking practitioners through the process of designing tools and implementing an evaluation process. They tend to be more hands-on and applied than reports/research papers, but they offer more support and explanation, particularly with regard to assembling an integrated suite of tools, than individual instruments.
Among the organizations that have produced evaluations guides are:
Administration for Children & Families published The Program Manager's Guide to Evaluation.
http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/
Authenticity Consulting provides evaluation workshops and guidebooks for non-profit organizations, which includes the Basic Guide to Program Evaluation by Carter McNamara.
www.managementhelp.org/evaluatn/fnl_eval.htm
The Distance Learning Resource Network published a "sourcebook for evaluators" entitled Assessing the Impact of Technology in Teaching and Learning.
http://www.rcgd.isr.umich.edu/tlt/handbook.htm
Girls Incorporated has developed Assess for Success, a needs assessment and evaluation guide for its programs.
www.girlsinc.org/ic
The W.K. Kellogg Foundation has produced two books for non-profits to assist in their evaluation efforts; one is the very comprehensive Evaluation Handbook, and the second is the Logic Model Development Guide.
http://www.wkkf.org/knowledge-center/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx
The National Science Foundation/EHR published a User-Friendly Handbook for Mixed Method Evaluations, which features an overview of evaluation methods and information on designing and reporting mixed method evaluations.
www.nsf.gov/pubsys/ods/getpub.cfm?nsf97153
United Way of America produced Measuring Program Outcomes: A Practical Approach, a "step-by-step manual for health, human services, and youth-and family-serving agencies" on evaluation techniques.
http://www.unitedwaystore.com/product/measuring_program_outcomes_a_practical_approach/program_film
In our research, we discovered a wide variety of reports, papers, articles, and other publications related to evaluation. Investigations into different issues, analyses of trends and developments, commentaries, and profiles all shed light on what is working in program evaluation. While such publications can inform practice, they are often less directly applicable to the day-to-day evaluation activities carried out by practitioners in the field.
Several organizations produce reports and other publications related to evaluation. Again, most of these pertain to evaluation more broadly, not just within the arena of youth media and technology programs, though they may be relevant to practitioners working in these programs.
Examples:
The Brookings Institution recently published a paper that they commissioned on The Growth in After-School Programs and Their Impact.
www.brookings.org/views/papers/sawhill/20030225.htm
Evaluation Forum seeks to build outcome-based planning and evaluation capacity in community agencies; information on their training, publications, and technical assistance is available online.
evaluationforum.com
Harvard Family Research Project, through their Out-of-School Time Learning and Development Project, has an extensive body of research on the effectiveness of out-of-school time programming.
http://www.hfrp.org/out-of-school-time
Independent Sector has collected information on measurement tools and publications and with The Urban Institute produced the 2001 report Outcome Measurement in Nonprofit Organizations.
The National Institute of Out-of-School Time (NIOST), part of the Wellesley Centers for Women at Wellesley College, has been analyzing the effectiveness of after school programs and the youth development field for twenty years.
http://www.niost.org/
Evaluation toolkits comprise a collection of information, tools, and resources that organizations can work with to develop customized evaluation plans and instruments. While we did not find a toolkit specifically designed for youth media and technology programs, typically the contents of such toolkits are easily adapted. Some of these include steps for carrying out early phases of an evaluation plan, such as needs assessment, as well as sample tools and additional resources.
Examples:
Benton Foundation provides links to other reports and websites that may prove useful in designing an evaluation; and in conjunction with EDC/Center for Children and Technology has produced the E-Rate Student Evaluation Toolkit: A Work-in-Progress.
http://cct.edc.org/report.asp?id=106
Community Technology Centers' Network (CTCNet) has assembled an Evaluation Toolkit containing a variety of resources for CTCs.
http://ctcnet.org/
Innovation Network is a nonprofit organization that seeks to build the evaluation capacity of nonprofits, featuring a workstation for designing program evaluation plans and tools.
www.innonet.org/workstation/about.cfm
A number of other relevant tools and materials have been developed to help strengthen youth programs or advance evaluation and assessment. The following are worth noting: documentation and reports on evaluation projects, training modules, checklists for designing and implementing evaluation, and resource libraries.
Examples:
The Evaluation Support Services Website at Western Michigan University's Evaluation Center is a clearinghouse of valuable evaluation resources.
http://www.wmich.edu/evalctr/home/
Ryakuga Community Communication Needs, Skills, and Resources Assessments, originally designed for the Youth for Social Justice Network, is a collection of links regularly updated to help community organizations develop effective community resources.
www.ryakuga.org/tutorials/map.html
Washington Regional Association of Grantmakers Resources offers links to evaluation resources, publications, and organizations.
washingtongrantmakers.org