Program Staff Reflections on Strategies & Challenges

While youth media making can have a powerful impact on a young person, the nature and complexity of the process can be difficult to effectively measure. In exploring evaluation capacity building with the youth media organizations Time Warner has funded, we asked them to share from their experience challenges with regard to program assessment and strategies they have found to address these challenges.

Strategies

• Use multiple forms of data collection. Consider video-taped classroom observations and focus groups with both youth and staff members. Combine objective data from observations with self-reported data and journaling, to create a more complete picture. Remember though that the interpretation process can be time-consuming.

"Evaluation helped us better define our programming goals so both instructors and students are clear about how their success will be measured."
- Stacy Ringo, Spy Hop Productions

• Know your focus and and be clear about what you want to learn from self-reported data before launching the program. Combine ongoing program self-assessment with the expertise of outside evaluators.
• Develop rubrics for portfolio assessment that articulate how works are demonstrations of learning. Use industry certification for measuring high-level production skills. Look to field-specific tools that can support media assessment.
• Collect stories from many different perspectives (e.g., assessment from teacher and youth, journals as a reflection tool, conversations from staff meetings, ongoing curriculum adjustment, end of year progress assessment, etc.)
• Be aware of issues around standardized measurement, both pros & cons. There may be some common goals that you have for all participants, but each young person brings a unique set of experiences and needs that will impact how much they may benefit from a particular programmatic strategy.
• Be efficient in data gathering. Aim for quality over quantity.

Challenges

• Evaluation efforts are often limited by a lack of resources. Funding, the availability of planning time relative to program time, and staff capacity to conduct relevant evaluation activities can all have an impact on what work an organization is able to do.
• Assessing multimedia projects adds complexity to youth media evaluations. The range of program goals, which may include the creative process itself as well as the completed project, and young people's personal development, as well as the acquisition of technical and production skills, can make comprehensive evaluation difficult to attain.
• The application of evaluation data is not clearly defined. While many program staff are articulate about their evaluation activities and what they are designed to measure, they are less clear about the process of applying the information they collect.
• Audience is a complicating factor. The unique relationship between media maker and their audience, intended and otherwise, creates a complex evaluation challenge that is often beyond the capacity of grassroots organizations. It also creates an opportunity for more authentic assessment.
• Making evaluations useful for funders and improving programs is a difficult balance to strike. There may be several goals for evaluation itself. Information that is useful in reporting to funders may not be as useful in improving program.