Document and Evaluate Your Program

 

 

One of the most effective things you can do to strengthen your study circle program is to create an accurate process for documenting and evaluating the entire effort. This doesn’t have to be a complicated undertaking. It’s up to you to decide how detailed you want the process to be.

Most everyone who takes part in your program will want to know who participated in the study circles, how effective the organizing strategies were, and what the outcomes were. Moreover, grant-making foundations, public officials, news media, and other people who can help you expand, strengthen, and institutionalize study circles in your community will all want to know about your efforts and their impact.

Revisit the goals set by your coalition, and identify benchmarks that will tell you whether you are meeting your goals.

Refer back to the goals you identified when you developed your initial work plan. (See “Develop a Plan” on pages 31-36.) Think about your impact goals and your process goals. Recall that impact goals describe the intended effects of the study circles upon individuals, communities, and institutions. Process goals describe the strategies that organizers use to try and meet the impact goals.

Next, think about what you want to measure, and how you would know whether you have reached your goals. Here are a few examples of goals and some indicators of whether they are being met:

Impact goals

•  If one of your impact goals is for participants to develop a better understanding of each other and build trust in one another, then you can ask questions about this on a participant survey.
•  If one of your impact goals is to promote new community projects or policy changes, then you can devise a system for tracking the work of action task forces that emerge from the study circles.

Process goals

•  If one of your process goals is to have a diverse mix of participants in each study circle, as well as in the entire community-wide program, then you can measure the diversity in the group with a participant survey.
•  If one of your process goals is to ensure quality facilitation of the study circles, you can do several things:
  • Get feedback from the facilitators – Convene an informal, focused conversation with small groups of facilitators to let them share their experiences, and listen and learn from each other. This will also give you a chance to monitor their progress, hear more about their needs, and identify any problem areas.
  • Use performance evaluations – Ask facilitators and participants to fill out performance appraisals at the end of a round of study circles. (Samples of evaluation forms are available in the evaluation section of SCRC’s A Guide for Training Study Circle Facilitators.)

Note that the purpose of these measures is not to say, “Gotcha - you didn’t do a very good job,” but instead to help you build on your areas of strength and to improve the quality and impact of the study circles.

Note also that the best strategies for documentation and evaluation are often those that are simple, clear, and straightforward. There is a place for more complex approaches to evaluation, but you can accomplish a great deal by making sure you are focusing on the questions that are most relevant to the short – and long-term success of your program.

Identify the types of documentation and evaluation that will be most useful to your program.

Here are four types of documentation and evaluation that can be of great use to your program:

1. Document the basics

Keep track of things, such as the number of participants and their demographic profile (age, gender, income, race/ethnicity, education level) and ideological preference (selfidentification as a conservative, moderate, or liberal). You might also want to note which neighborhoods participants come from.

2. Track the outcomes

The outcomes of study circle programs are many and varied. Here are a few examples of the types of impacts documented by other programs:

• New understanding of an issue – In Wilmington, Delaware, 79 percent of participants reported that study circles on racism and race relations had increased their understanding of others’ beliefs and attitudes.
• New relationships between individuals, groups, and organizations – High school students who participated in study circles in Kuna, Idaho, arranged to hold a “senior prom” at the town’s senior center. Teen-agers and seniors dressed up in formal attire and danced the night away.
• New collaborations – A hospital in Marshall, Minnesota, created a program to teach recent Somali immigrants to sew, and provided them with new employment opportunities.
• Policy changes – The city government in Springfield, Illinois, changed its hiring practices for the police department, resulting in the first African- American, the first Hispanic, and the first female to be hired in the last ten years, all in one police class.
• Structural changes (new organizations and institutional arrangements leading to other changes) – In Orford, New Hampshire, study circle participants concerned with declining school enrollment proposed a regional solution. The resulting Rivendell School District is the first two-state, K-12 district in the country, spanning four small towns in New Hampshire and Vermont.

3. Communicate the “public voice”

Community-wide study circle programs generate a great deal of information about participants’ hopes, concerns, and ideas for change. Brief, accessible reports with this type of substantive information will be of great interest to public officials, the news media, foundations, researchers, and others who can help you expand, strengthen, and institutionalize study circles in your community. Such reports are usually based on notes recorded in all of the study circles.

4. Evaluate program implementation

Take a close look at how program implementation is shaping up. How do both the facilitators and participants feel about the quality of facilitation, the discussion guide, and logistics (that is, accessibility of meeting places, transportation, child care, etc.)? How about your coordinator, working group, and coalition – how do they feel about the way they have been working together?

Organizers use evaluations to measure progress, and adjust strategies and practices over the life of the program.

Identify the right tools for the job.

Once you know what you want to measure, the next step is identifying the most appropriate tools for the task. Here are the types of tools you are most likely to use:

Taking notes

Keep careful records of the organizing process: Take notes on all the work you do – from meetings with your working groups, to discussions with potential sponsors, to briefings with members of the news media.

Keep records from all the study circles: It is especially important to keep a brief session-bysession record of the discussion within each study circle. Notes from all the study circles can be combined into a report for the program as a whole; they will be useful in planning action forums; and they can be used to give updates to public officials and journalists.

Two facilitators, working as a team, can easily take turns leading the discussion and taking notes. (Participants can also be asked to volunteer, but make sure it’s not the same person for every session, because taking notes takes the person out of the discussion.) Keep track of the most important points,