Animating Democracy held its second Evaluators Circle (E-Circle) at the Japanese American Cultural & Community Center, inviting Los Angeles-based cultural leaders, researchers, and evaluators to share evaluation projects, findings and learning. It was a rich and hearty exchange.
Theory
The Annotated Guide to Tools & Resources provides tools, frameworks, and resources to help you develop and implement your evaluation. It’s a repository of useful, practical materials that can help you create an evaluation plan; design your evaluation approach; develop or adapt tools and instruments; and otherwise move your evaluation forward.
This Guide was originally assembled from many sources and fields and annotated by evaluator Suzanne Callahan of Callahan Consulting for the Arts. We continue to add resources. Your suggestions are welcome!
Do you have a useful tool or resource to add? Contact animatingdemocracy@artsusa.org.
Publication Date: January 22, 2016
Resource Format: book / article
Continuum: Attitudes, motivation (values, aspirations) | Awareness, knowledge, understanding | Behavior, participation (action, activism, advocacy) | Systems, policies, places (legislation, issue/problem resolution)
Publication Date: February 1, 2011
Resource Format: practical tool
FSG and Knight Foundation’s IMPACT: A Practical Guide to Evaluating Community Information Projects aims to help organizations collect useful information about the impact of their community information projects. Community information projects incorporate media, news, and online and digital communication to build knowledge and engagement capacities in community.
Resource Format: book / article, practical tool
Written for evaluators, this 17-page report is a very nice guide to communicating about culture in evaluation practices and may be used as a reference in data collection. The report emphasizes that cross-cultural competency is a necessary skill for evaluators to have. The author encourages evaluators to think of culture as a factor to be considered as much as sampling and measurement. In addition to the introduction and conclusion, the paper has three main sections: culture, social identity and group membership, and privilege and power.
This 33-page document calls itself a study but can serve as a guide to indicator development. Though done in 1996, it an extremely thorough, thoughtful overview of indicator consideration and development. It defines indicators, gives examples of types, and talks of the challenges of measuring difficult areas such as political development. It discusses issues related to indicators in governmental agencies versus NGOs.
From the Centers for Disease Control and Prevention, this 71-page report defines process evaluation and describes the rationale, benefits, key data collection components, and program evaluation management procedures. Within the framework of discussing tobacco use prevention, this paper is a very good primer for process evaluation for readers in a variety of fields. It provides clear and well-presented charts, graphics, principles, and summaries to help guide readers.
Resource Format: book / article
Continuum: Awareness, knowledge, understanding
Understanding how knowledge moves through communities can help evaluators select the components of a program that are best suited for scaling up. “Scaling” is the practice of replicating programs for implementation at new locations, by other organizations, or to serve larger groups of people. Knowledge-for-action theories address how learned information becomes the measureable impact of arts for change work. This article highlights five theories of knowledge transfer: utilization, diffusion, implementation, transfer, and translation.
Resource Format: book / article, case study
This 35-page report provides a summary of findings and lessons learned by the Neighborhood and Family Initiative (NFI), a comprehensive community initiative launched by the Ford Foundation that began in 1990. NFI was a 10-year effort that sought to strengthen a single neighborhood in each of four cities and to improve the quality of life for the families who live in them. The report is divided into two sections; the first provides a brief overview of NFI while the second section distills the lessons learned by NFI over the course of implementation.
Publication Date: January 1, 2013
Resource Format: website
In Making Data Visualizations: A Survival Guide and Other Resources, Beth Kanter, author of Measuring the Networked Nonprofit, blogs about the importance of data visualization strategies in the nonprofit sector. In this list, Kanter offers her own take on six resources that provide an entry point into the field of visualization. These resources range in length from full books on graphics to brief tips on creating effective yet simple presentations. Below, Animating Democracy notes resources that we believe to be the most useful:
Publication Date: July 1, 2014
Resource Format: book / article
Patterns of participation for arts and cultural activities have resisted significant change. In Making Meaningful Connections, The James Irvine Foundation researches attendance patterns of small, community-based cultural organizations and events. Attendance diversification is not only important to organizations that produce “benchmark arts activities,” but for smaller organizations that may work within specific demographic populations and would simultaneously like to expand their community participant base.
This piece is an excellent primer on how to think about outcomes and the hurdles that may arise in measuring them. It addresses and deals with challenges such as measuring intangibles. Published by GrantCraft (a division of the Ford Foundation), the eleven-page guide is written for grantmakers to describe outcomes-based evaluation. It defines key terms and makes a case for why outcome measurement is important.
"What are we doing, and why do we think it’s going to make a difference? Are we being effective?" Grantmakers ask evaluation questions like these of their grantees and themselves. This brief guide explains why grantmakers use theories of change to guide their questioning, unearth assumptions that underlie their work, establish common language, and develop strong action plans. Contributors to the guide also describe how a theory of change sets the stage for evaluation by clarifying goals, strategies, and milestones.
Publication Date: February 1, 2014
Resource Format: book / article, practical tool
This report provides a comprehensive overview and assessment of the approaches that are currently employed in impact assessment of media as a socially valuable tool. “Social value,” in this context, is described in terms of outcomes that extend beyond financial measures of success such as improving the well-being of individuals and communities. This report seeks to identify relevant analytical approaches, methodologies, and metrics for assessing media’s social impact that can inform further work in this area and that appear particularly promising.
Publication Date: October 9, 2014
Resource Format: book / article
Measuring the Networked Nonprofit offers tools and strategies to nonprofits searching for reliable and measurable data from their social media efforts. This book is a hands-on resource for nonprofit professionals who must be able to accurately measure the results of their social media ventures.
Publication Date: December 31, 2009
Resource Format: book / article
In answer to the challenges that face corporate philanthropy in identifying a shared definition of impact measurement, the author sets out to assess current measurement practices, clarify what is needed in terms of impact evidence, and identify next steps. The article is organized into three conversations between key stakeholders engaged in corporate philanthropy.
Resource Format: book / article
This paper is one of a number of working papers produced for the project “Understanding the drivers of, and value and benefits afforded by, engagement in culture and sport”. The objectives of the project are to define and model the following broad relationships: 1. The impact of policy options and other factors on the level of engagement in sport and culture. 2. The outcome of engagement in sport and culture. 3. The value of these outcomes. This paper focuses on the last of these points.
Publication Date: May 1, 2012
Resource Format: book / article
Outcome Harvesting is a method that enables evaluators, grant makers, and managers to identify, formulate, verify, and make sense of outcomes while acknowledging the complexities of social change projects. Unlike some evaluation methods, Outcome Harvesting does not measure progress towards predetermined outcomes, but rather collects evidence of what has been achieved, and works backward to determine whether and how the project or intervention contributed to the change.
"In so many evaluations,” said a program officer, “no one thinks to ask the users.” Participatory action research offers grantmakers a way to do so. It engages all parties in all aspects of an evaluation, from defining the problem to gathering and analyzing data to preparing recommendations. In this guide, learn about a unique evaluation method and how grantmakers used it to evaluate programs in agriculture, early childhood development, and immigration.
Publication Date: December 31, 2009
Resource Format: book / article
This piece suggests that the accountability movement is “setting a floor for minimum standards” (p. 84) and has consequences for effective social change work. Foundations, in particular, measure impact in terms of attentiveness to accountability standards, but this is a false measure of success. Instead, the organization’s focus should be on its transformative value to society.
Resource Format: practical tool, website
Point K Learning Center ("Point K"), a website and online workstation, was created by Innovation Network (Innonet), a leader in the participatory evaluation field and one of the first to make online tools available that could be used by smaller as well as larger organizations. Point K features practical tools and resources for nonprofit planning, evaluation and action. It is a major part of Innonet’s mission of sharing know-how to create lasting social change. Point K aims to help users assess strengths, articulate goals, use data and better tell the organization’s story.
Publication Date: January 1, 2014
Resource Format: book / article
Presenting Data Effectively is a step-by-step guide to making research results more useful and interesting. This book provides guiding principles for designing effective data presentations and reports.
These complex concepts are outlined in straightforward manner and can easily be applied by students, researchers, evaluators, and non-profit workers. The book guides the reader through design choices related to four primary areas: graphics, type, color, and arrangement.
Resource Format: book / article, practical tool
A bible on the topic of qualitative inquiry, this book introduces the principles and methods of qualitative research and evaluation. Chapters cover conceptual issues in qualitative inquiry (including strategic themes and variety), the design of qualitative studies, fieldwork strategies and observation methods, interviewing, analysis, interpretation, and reporting. Important controversies are outlined, and key points are illustrated with examples. This text is a resource and training tool for applied researchers, evaluators, and graduate students.
Resource Format: practical tool
This paper reflects Animating Democracy co-directors’ thinking following Working Group discussion of Chris Dwyer’s “Arts and Civic Engagement: Briefing Paper for the Working Group of the Arts & Civic Engagement Impact Initiative.” The discussion got to the heart of the challenges of measuring the social impact of arts-based engagement and somewhat shifted the Initiative’s theory of change.
Publication Date: June 12, 2014
In a response to the article Strategic Philanthropy for a Complex World, Darren Walker calls on philanthropists to change the current paradigm of evaluative practice employed in social change work to develop sharper tools and more inclusive frameworks that account for the considerable complexity of real-world problem solving.
Publication Date: June 25, 2014
Resource Format: practical tool, website
Ripple Effect Mapping (REM) engages program and community stakeholders to retrospectively and visually map the "performance story" of their work. The REM method, presented here by University of Minnesota, is a participatory group method for evaluating the impact of complex programs or interventions.
Since 2000, the Saguaro Seminar's mission has been both to improve social capital measurement and the availability of social capital data and to undertake analysis of building social capital in increasingly diverse communities. The web site includes extensive information on measurement of social capital including Putnam’s 2000 Social Capital Community Benchmark Survey, as well as evaluation tool kit, guide, and links to useful organizations and resources. The Saguaro Seminar is an ongoing initiative of Professor Robert D. Putnam at the John F.