By Silvia Garcia, Director of Research and Assessment, IU Indianapolis Office of Community Engagement
Help us gather feedback: invite your partners to participate
Indiana University Indianapolis has long recognized its role as an anchor institution, deeply woven into the fabric of the community and committed to community engagement through its mission and leveraging of financial, physical, intellectual, and human capital resources to support local needs and drive development. However, for too long, the evaluation of the campus’ impact has primarily focused on institutional outputs – number of hours, number of people, amount of funding – overlooking the true impact these engagements have on the community.
That is why, in 2022, three IU Indianapolis Office of Community Engagement staff members—Silvia Garcia, Director of Research and Assessment; Latosha Rowley, program manager; and Jim Grim, former director of community and school partnerships—embarked on a project to develop the Partner Survey, a tool that captures community voices and perceptions of engagement outcomes, allowing the community to define what matters in community-university partnerships. This project exemplifies the transformative power of involving community partners in the evaluation process. By actively engaging community voices, a survey tool was developed that more accurately captures engagement outcomes and is more relevant and understandable to community partners.
The challenge of measuring impact
Measuring community-university engagement outcomes is notoriously complex. The diverse nature of institutions, communities, and activities, coupled with the unique contexts of each engagement, makes a one-size-fits-all evaluation framework elusive. By involving local community partners in the process of developing a survey, invaluable knowledge and experience were gained, ensuring that the questions and methods developed are relevant to both the campus and the community.
To identify community partners to participate in the survey development, community engagement activities were mapped using the IU Indianapolis Collaboratory—a database where faculty and staff enter detailed information about their community-engaged activities—and Digital Measures Activity Insight (DMAI), now Elements at IU, the annual faculty activity reporting system. These sources enabled mapping that took into account stakeholders, geography, purpose and duration of the partnership, and interdisciplinarity.
The collaborative effort to develop the Partner Survey involved four phases of development and validation.
Phase 1: Identifying meaningful outcomes
The survey development process began with interviews that provided a deep dive into community partner perspectives. Seven project coordinators (IUI faculty) were asked about their projects, intended and observed community impact, and community organizations they recommended provide input. This led to interviews with twelve community partners holding leadership positions across diverse organizations representing a historical museum, environmental organizations, neighborhood groups, youth-serving organizations, a technology for-profit, and agencies with re-entry initiatives. During these interviews, the results and impact of their collaborations with the campus were explored, as well as the positive and negative aspects of the campus involvement.
Phase 2: Analyzing and validating indicators
The interview data were analyzed and categorized using the Democracy Collaborative Framework for Anchor Institutions’ outcome areas. This process identified 102 codes and eight themes, with 70 codes classified as process indicators and 32 as outcome/impact indicators. Process indicators offer information on the quality of the engagement and other short-term results that contribute to outcomes. In contrast, outcome indicators measure long-term changes linked to engagement goals, including expected changes (impact) in the community.
To ensure external validity, outcome indicators were compared with the Long-Term Impact Code Book (Kuttner et al., in progress) based on the Collaboratory national dataset and the IU Indianapolis’ 2030 Strategic Plan goals. This comparison led to the inclusion of new outcome indicators, particularly those related to science, technology, and innovation.
Phase 3: Crafting the survey tool
With a robust list of validated outcome/impact indicators, an initial version of the Partner Survey, consisting of 97 items, was developed. It included sections on collaboration types, specific outcome items across various engagement areas (arts and culture, educated youth, health and wellbeing, environment and sustainability, community safety, science, technology, innovation, and community building), contribution to IU Indianapolis’ 2030 Strategic Plan goals, and questions to measure the quality of partnerships based on the Carnegie Foundation Elective Classification for Community Engagement’s Community Partner Survey.
Phase 4: Survey validation using focus group-based cognitive interviews
The next critical step was survey validation. Grant funding was received from the Community Engaged Alliance to support six focus group-based cognitive interviews (focus group CI) which gathered in-depth feedback from community partners, and IU Indianapolis staff and faculty, to understand how respondents comprehend survey questions. Eleven community partners, seven faculty and one staff member participated in the focus groups.
This method allowed assessment of whether the survey accurately captured community partner perceptions of engagement outcomes and whether the outcomes were contextualized for their experiences. For example, there were discussions about whether the language was adequate and the categories used in the survey, such as areas of collaboration, aligned with how community partners experience these collaborations.
Participants completed the survey, noting questions they found easy or difficult to understand and providing written comments. Moderators asked questions to uncover participants’ interpretations and identify misunderstandings or ambiguities. They also encouraged participants to suggest modifications to questions based on their experiences. With this invaluable feedback, the survey was significantly modified. Key changes included:
- Adding questions about the populations served by the organization.
- Revising collaboration types to include more specific options like internships, professional assistance, professional development and training, project/program evaluation, technical assistance, event organization, and international exchange. Focus group participants also expressed that the survey’s use of the word ‘partnership’ in key questions failed to capture more casual forms of community-university collaborations, which could influence how they perceive engagement outcomes. Therefore, the new version consistently uses “collaboration” instead of “partnership” in survey questions.
- Reducing the number of outcome items, simplifying language, and adding open-ended questions to capture perceived outcomes.
- Reducing the number of items in the quality of partnerships section.
- Expanding the question on community contributions to IU Indianapolis.
- Adding two final open-ended questions about ways IU Indianapolis can develop more effective collaborations, and any other information that can help us understand their experiences in collaborating with IU Indianapolis.
Testing the tool
The next step was to further examine the validity and feasibility of the survey by piloting it with a small sample of community partners. Results were favorable, but more feedback from faculty and staff using the survey with their partners is being sought.
Faculty and staff are encouraged to invite community partners to complete the revised Partner Survey, building on the successful pilot test.
The Partner Survey is available here for faculty and staff to use with their community partners. For questions or to share feedback, please contact Silvia Garcia at sigarcia@iu.edu.
About Community-Engaged Alliance
As a 501(c)(3) nonprofit, Community-Engaged Alliance is dedicated to advancing the field of community engagement through education, support, and collaboration with partners in higher education. Through funding, professional development, and collaboration opportunities, Community-Engaged Alliance supports capacity and enables students to graduate with the knowledge and skills necessary to create positive social change. Visit www.communityengagedalliance.org for more information.