After the evaluation line-item runs out
I recently had the opportunity to delve into the insightful report published by the National Science Foundation regarding Perspectives of Broader Impacts. The report raises numerous thought-provoking questions, many of which I am going to think through and respond to in the coming weeks. One particular question that stood out to me was: “what are the impact stories beyond the life of a project?”
In my experience, nearly all the projects I've been involved with strive to achieve some form of distal impact, whether that be some form of social or behavioral change. However, out of the many evaluations I've participated in, only two received funding to facilitate longitudinal evaluation. That means, in most cases, my role involves assessing proximal (short-term) impacts, rather than the broader, longer-term impacts. While it would be a dream to conduct longitudinal evaluations for all projects I contribute to, I propose two interconnected strategies (that could, and I think - should) be funded to enable long-term evaluations in a streamlined and cost-effective manner:
Evaluation Capacity Building: This encompasses the allocation of resources, including personnel, tools, and time, necessary to conduct an evaluation. Instead of simply hiring an external evaluator, funding could be directed towards training project team members to carry out evaluations themselves. This support would involve building the tools or approaches required for the teams to collect, analyze, and report their own data. Integration of these tools or approaches into existing systems can occur if the team lacks the resources or capacity to acquire an entirely new skill set. In past projects, I have:
Developed excel spreadsheets, potentially extracting data from existing sources, to facilitate data collection and analysis.
Assisted teams in incorporating evaluation practices into their routine meetings (e.g., conducting regular evaluations, learning sessions, Before Action Reviews/After Action Reviews).
Educated team members on how to collect qualitative data (e.g., conducting focus groups, creating interview protocols, assessing data trustworthiness).
Fostering a Culture of Evaluation: While somewhat abstract, this is undeniably one of the most crucial elements in ensuring the implementation of long-term evaluations. A culture of evaluation requires a genuine commitment from individuals within an organization, not only to engage in evaluation , but also to utilize the findings to guide decision-making. Without this commitment and dedication, evaluation capacity building would only fulfill half of the equation. There are several ways to gauge whether a project or organization has a culture of evaluation. A few concrete methods I've employed include:
Assessing organizational readiness for evaluation.
Conducting asset mapping to identify individuals capable of leading and supporting long-term evaluations.
Engaging in participatory evaluation to demystify the evaluation process, thereby instilling confidence and skills in teams to undertake evaluations.
I am enthusiastic about the prospect of engaging in more evaluation projects that employ these approaches, paving the way for long-term assessments of broader impacts to become a standard practice.
