The Critical Need for Program Accountability & Evaluation
Lately, I have been doing a lot of thinking and reading about nonprofit evaluation. As part of an evaluation team for a project, I have also been working directly with a range of nonprofits, providing coaching and guidance on evaluation design. I have come to believe that program accountability and evaluation is an area of conceptual and practical disconnect both within nonprofit agencies and between nonprofit agencies and the government and philanthropic organizations that fund them. In this blog I want to begin a conversation about the role of evaluation & program accountability in nonprofit organizations and in the next blog discuss how nonprofits can (and should) use evaluation to achieve greater social impact.
As a starting point, I want to create a visual for the concept of nonprofit program accountability. Among the technology assets in the empire of Google is a tool called NGram Viewer. Using this tool, one can enter a word or concept and graph the use of the word over time (using as a reference the database of Google Books). I recently used NGram Viewer to graph the concept of nonprofit program accountability. As you can see in the figure, nonprofit accountability first emerged as a concept in the late 1970’s and bumped along until the early 2000’s. It was at that point that the term began a steep incline with nonprofit program accountability becoming a living, breathing, much talked about idea. Note: please don’t overestimate the value of this word picture as I am not suggesting there is any rigor to the underlying data –its just a illustrating concept. With such a steep growth, one would expect nonprofit program accountability and evaluation to be a priority within nonprofit organizations. Enter conceptual and practical disconnect.
Quick story. I was once talking with a nonprofit board about program accountability and evaluation. At one point in the conversation, a board member sitting back with his arms crossed asked, “is there any reason to think our funders have a problem without performance?” The response was “no, our funders are happy.” The second question of the board member was, “Does the agency staff think we’re doing what we need to be doing?” The response to this question was definitive, “Yeah, we’re doing great work.” The third question began to sound like the case was being built question by question, “So, is anyone asking us for greater program accountability and evaluation?” The third answer was more tentative, “um, no, we aren’t being asked for anything new.” With an air of pronouncement came the board member’s final rhetorical thought. “So why are we having this conversation about program accountability and evaluation?” Needless to say, program evaluation got little traction that evening.
When we turn to the literature, we fare no better. For example, a series of articles published by smart academic Joanne G. Carman and her colleagues, support the notion that the practice of program accountability and evaluation varies tremendously among nonprofits (See Dr. Carmen’s CV for a nifty bibliography). Abstracting one statement from the studies illustrates the theme of her and her colleagues work.
“The picture that emerges is one that is decidedly mixed, illustrating a range of behaviors that challenges the current perception that most, if not all, funders are asking nonprofit organizations for more evaluation and performance measurement data.”(1) – emphasis mine
Even more recently, The Center for Effective Philanthropy published a report titled, “Grantees Report Back: Helpful Reporting and Evaluation Processes” that concluded, “On average, grantees do not find current reporting and evaluation processes to be very helpful in strengthening their organizations and programs.” In differentiating the use of evaluation and reporting, the study concluded that “Grantees who report discussing their report or evaluation with their funder perceive the reporting or evaluation process to be more helpful — yet nearly half of grantees say no discussion occurred.” -emphasis mine.
The conceptual and practical disconnect between program accountability and evaluation and the perceived usefulness of such efforts is clear. While we talk about nonprofit accountability and evaluation, the practice of such efforts is uneven and inconsistent. If this is the state of field practice, the next logical question we need to ask is “what are the barriers that get in the way of program evaluation?”
1. First is the wall. Money –or more appropriately the lack of it. Many will say that the disconnect between evaluation talk and practice is directly connected to the lack of resources. “We can’t afford to do evaluation.” is the typical response to the lack program accountability and evaluation data. However, I suggest that you can’t afford not to. I personally believe that the day is rapidly approaching when funding agencies and donors begin to say, “The emperor has no clothes.” I have written before that, at the organizational level, transparency and accountability are becoming increasingly important. Without supporting program level accountability and evaluation data, I also believe that nonprofit organizations will come under greater and greater skepticism. So while at face value the barrier of “no money” may be true, the wall must come down and every organization must make it a priority to dismantle this barrier. A final thought is that the wall of “no money” actually hides the real barriers to program accountability and evaluation, of which, I believe there are three:
2. Skills, Support, Confidence & Value: I believe that a significant barrier that prevents many organizations from seriously addressing program accountability and evaluation is found in the alchemy of “lacking skills” in evaluation and “lacking the support” to conduct evaluation. Too often the leadership of organizations fails to invest in developing the organizational skills and support to conduct evaluation. Skill development and support does not have to be costly. There are a variety of online resources that are a Google search away. Relationships can be built with local colleges and universities or seek out a qualified consultant. Finally, if you need money, then raise it. Enough of the excuses.
The second part of this barrier is the lack of confidence and values. Some have labeled these two variables as core motivation. If one understands the value of a task and has confidence that they can do the task then there a greater likelihood that the task will become a priority. Putting these pieces together, to overcome this barrier, an organization should develop an approach to understanding and resourcing evaluation that builds confidence and create a culture and an organizational value that supports evaluation and the use of the resulting data.
3. Missing or Misaligned Incentives Another barrier to program evaluation and accountability is found in the area of incentives. To start, ask yourself some inquiry questions such as
• When was the last time that your organization celebrated a data report that demonstrated program effectiveness?
• When was the last time you presented program evaluation results in a public venue?
• When was the last time that your major funder provided you with adequate resources to conduct an evaluation?
• Have you ever been penalized for having or not having evaluation and accountability results?
• Has anyone ever asked for your program accountability and evaluation results?
These are a just a sample of questions that can be used to probe where the incentives for evaluation are (and are not) for your organization. Exploring the incentives, lack of incentives, or misalignment of incentives is another way to, not only identify an evaluation barrier, but to dismantle it. As a result of this inquiry, an agency should clearly be able to identify positive incentives for program evaluation and, if they can’t be identified, they should be created.
4. Fear that We are the Emperor: Perhaps the largest barrier to program accountability and transparency is the fear that evaluation outcomes would reveal that we are the proverbial emperor with no clothes. “What if we measure and the results are negative?” can be a paralyzing specter if our mindset is one of fear. However if we want to develop ourselves into a socially innovative nonprofit organization we must dismantle fear and embrace inquiry. We need to understand that program accountability and evaluation is the source of power and empowerment. Without evaluation how can we improve or measure progress? Program evaluation is the stuff that makes program giants, changes things, and disrupts unmet needs. Conversely the lack of data simply perpetuates the myth of nonprofits as nice organizations doing good in the community.
What I am arguing for this is post is the need for nonprofit organizations to consider their relationship to program accountability and evaluation. The literature and practice might suggest that the field of the social-citizen sector talks about accountability and evaluation but at the same time, also suggests that we have been less than successful at operationalizing that talk with any consistency. However, I would argue that this is case where field practice does not matter. I believe that socially innovative nonprofit organizations are those that invest in the development and implementation of a solid plan for program accountability and evaluation. As illustrated by the Venn diagram, organizational strength and impact is more durable in the presence of a solid evaluation and accountability approach that is connected to strategic and resource planning.
Program accountability and evaluation demonstrates a nonprofit’s commitment to excellence. As nonprofit agencies continue to look for ways to innovate in a resource-constrained environment, building and implementing strong strategic plans, resource plans, and evaluation plans will position agencies well to effectively meet compelling community needs.
A always, your thoughts are welcome.
Photo Credit: qimono
(1) Carman, JG. (2009). Nonprofits, Funders, and Evaluation Program Accountability in Action. The American Review of Public Administration 39: 374-390.