Approaches in Impact Measurement

Running head: Approaches in Impact Measurement for the Nonprofit Sector

AAApppppprrroooaaaccchhheeesss iiinnn IIImmmpppaaacccttt MMMeeeaaasssuuurrreeemmmeeennnttt

fffooorrr ttthhheee NNNooonnnppprrrooofffiiittt SSSeeeccctttooorrr:::

PPPrrraaaccctttiiiccceeesss aaannnddd MMMeeettthhhooodddsss

Rhonda S. Magee

In Fulfillment of Requirements for: MNM 697 Professional Project

A Paper Submitted to:

Master of Nonprofit Management Program Regis University,

Denver, CO

April 27, 2013

Approaches in Impact Measurement for the Nonprofit Sector

2

Table of Contents

ABSTRACT ..............................................................................................................................................................3 INTRODUCTION ....................................................................................................................................................4

PROBLEM STATEMENT ......................................................................................................................................8 LITERATURE REVIEW...................................................................................................................................... 10 THE NATURE AND DEFINITION OF IMPACT.......................................................................................................................11 PERCEPTIONS, AND CONCEPTS IN IMPACT MEASUREMENT: ITS VALUE, NECESSITY AND UTILITY........................14 APPROACHES IN IMPACT MEASUREMENT..........................................................................................................................16 CHALLENGES AND DIFFICULTIES IN PERFORMANCE MEASUREMENT..........................................................................17

METHODS............................................................................................................................................................. 20 PROCESS ...................................................................................................................................................................................23

RESULTS, DISCUSSION AND RECOMMENDATIONS................................................................................. 23 FINDINGS..................................................................................................................................................................................23 DISCUSSION, OBSERVATIONS................................................................................................................................................29

CONCLUSIONS..................................................................................................................................................... 30 REFERENCES ....................................................................................................................................................... 33 APPENDIX A ........................................................................................................................................................ 35 APPENDIX B ....................................................................................................................................................... 36


3

Abstract In nonprofit organizations, variables such as mission; organizational performance; the

activities and efforts of the organization; and the capacity of the organization all have

influence its impact. Impact, as used in this paper, is defined as: the changes that result

from the activities, efforts and programs of nonprofit organizations. This concept can be

referred to as outcomes or as performance.

The practice of measuring impact is examined from the perspective of a small group of

nonprofits in the Denver, Colorado area and a consulting firm the National Research Center

(NRC). The interviews were conducted to determine the extent of impact measurement

being practiced, and the methods used.

This exploratory study shows that organizations are actively measuring impact, carefully

selecting indicators and that overall, they are satisfied with their methods and tools.

Measurement processes and data to being used proactively, especially to refine programs

and scale efforts through collaborations.


4

Approaches in Impact Measurement

for the Nonprofit Sector: Practices and Methods

In the nonprofit sector, “multiple bottom lines” exist, making management of 501(c)3

organizations complex, challenging and very different from the for-profit sector. Evaluating the

performance of an organization’s efforts is just one of the management challenges.

It is generally accepted that in the private sector, one common reporting tool, the

Statement of Profit and Loss (with its single bottom line) provides an excellent measure of both

organizational performance and its current state of effectiveness. Other analytical tools exist to

further measure and analyze performance of for-profit entities. Periodic measures of

organizational performance, stated in the financial perspective of profit achievement, when

compared against the investments (or inputs into the firm) help define the degree of efficiency

inherent in the performance. This is known as Return on Investment (ROI) analysis, and provides

a valuable metric in private sector management. These and other common reporting tools provide

a standardized representation of the performance of the private sector entity. In this context,

financial and monetary measures form the common language, and when presented in consistent

and commonly seen formats, they provide a baseline of understanding the financial results and

performance of the profit motivated firm, to and for its constituents.

Some of those same calculations and reports can and should be generated for the

nonprofit organization, but they don’t provide a complete analysis of organizational performance

or effectiveness. The information contained in them doesn’t carry the same weight as in the

private sector. The nonprofit organization operates from the perspective of multiple bottom lines,

the rationales of which are very different and dependent upon multiple constituencies. This

creates complexity in the attempt to recognize and identify performance indicators and measure


5

related performance. (Anheier, 2005) While nonprofit leaders and stakeholders know these

reported numbers don’t tell an adequate story, they face many challenges when attempting to

identify other assessment approaches. Despite what they know about the limitations of financial

reports in nonprofit sector organizations, it is easy for directors—who are often successful

members of the private sector business community—to habitually over-focus on traditional

reporting tools, such as the profit and loss statement, balance sheet, cash flow forecast, and

salary budgets. Perhaps if there were better standards of practice established for non-profit

entities that provided easily managed processes for reporting and analyzing impact, performance

and effectiveness, leaders—both board and staff—of nonprofit organizations could focus on

more relevant measures. The traditional financial reporting tools could then be used

appropriately and proactively to serve as an important means to an end, but not as the end itself.

In this approach, the means include the transparent illustration of the fulfillment of accountability

and fiduciary oversight, the efficient management of financial resources for program and

budgetary management. An ends-based perspective of the organization management—with ends

defined as mission and desired results—becomes the catalyst for an entirely different set of

reporting instruments, designed around impact measurement. John Carver, (1997, p. 31) in

Boards that Make a Difference sees ends-based leadership as essential to governance and

organizational success. Ends can be seen as “results, impacts, goals, or outcomes.” (Carver,

1997) This model of reporting on both financial means and organizational ends/objectives forms

the underlying assumption for this research.

Impact for the purpose of this study, is generally understood to be very closely related to

the definition of outcomes, ie, the changes, or the perceived changes created by the work or

programs delivered. The changes can be at the individual, community, or societal level. Impact


6

often goes a step further, however, encompassing the value or worth of those outcomes, or a

measure of the ripple effect created in community or society by certain outcomes. Impact is thus

the result of the outcomes, and impact cannot always be measured. Impact can be focused

directly on the subjects or clientele served, or it can refer to a broader host of beneficiaries. For

example, an organization that is heavily engaged in advocacy efforts will be more interested in

how those efforts effect change in society, without an emphasis on individuals per se.

Organizations in various subsectors will define their desired impact differently, so how the

subjects in this study see or define impact differently will be highlighted and discussed.

Impact as a term is used in the literature at times interchangeably with the term

organizational performance and with the term outcome. The use of terms is largely dependent on

the perspective of the speaker or writer, the audience, or the sub sector context. For example,

Greenway, (2001) explains that she uses the term “outcomes” to discuss organizational

performance and results, “because of the considerable disagreement in the human services sector

about what impact means.” (p. 218) In her work, she makes a distinction between the

beneficiaries of the resulting changes, by using Program Outcomes to point to individual benefits

or changes, and Community Outcomes for the cumulative changes that may occur.

There is evidence, presented elsewhere in the Literature Review, that the nonprofit sector

is becoming more aware of the need for a greater emphasis on end outcome or impact

measurement and reporting. While some nonprofit and governmental leaders and experts are

developing standards of practice for impact measurement, the degree to which organizations

have adopted impact measurement as their primary measure of their “bottom line” is not clear. It

is also not clear how impact is defined or measured within and across organizations.


7

The challenge has been that the social impact created by nonprofits is difficult to

measure, whether qualitatively or quantitatively. There can be resulting ripple effects or

influences that are not directly related to the program or outcomes, and impacts may have future

or long-term characteristics, making calculation of values difficult. In some cases, determining

the value of social change created requires a reliance on external data of various sources,

suggesting a heavy research component of the process. For example, if a training program

provides job readiness skills, the selected impact indicator may relate to the future income

earning potential, over a period of years, derived from a resource such as the National Bureau of

Economic Research. Issues such as these add to the complexity of any measurement task.

With an understanding of the challenges and complexities, and knowing that it is in the

best interest of mission achievement to attempt qualitative or quantitative measurement in spite

of difficulties, how do organizations approach the issue? The extent to which and the manner in

which impact measurement is actually being performed in local organizations is addressed in this

study. Would these organizations be more willing to invest in impact measurement if the benefits

of doing so were more tangible, clear and apparent? The Literature Review seems to reflect this.

Likewise, if there was a shared understanding of the measurement processes and practices

similar organizations have been able to develop; would these measurement theories and practices

become more accessible, common, disciplined and useful? This paper explores this latter

possibility, addressing the issue from the perspectives of both leadership and practice. The study

shows some of the specific indicators of impact that are being selected by organizations and

how—in general terms—they attempt to measure their bottom line impact. The focus will be on

the selection of indicators and how satisfied they are with their methods—standards and tools—

being used for measurement. Beyond the scope of the study is what the targeted organizations


8

have learned as a result of putting performance and impact measurement practices into place.

Motivations and criteria of measurement efforts are connected to the main objective of indicator

identification. Mechanics of methodology and feasibility may drive the process of indicator

selection for many organizations, because measurability is a key factor, but these technical

details are not the focus of this project.

The author of this paper is a student in the Regis University Masters of Nonprofit

Management program, for which this Professional Project will partially fulfill the requirements

for graduation. Throughout the paper this student will refer to herself as the “project researcher”.

Exploration of the subject matter was prompted by the researcher’s own experience in nonprofit

management, primarily in financial management, where she observed gaps in organizational

capabilities in the arena of performance measurement and management, and over-emphasis on

financial reporting. Thus, there is likely an inherent bias while the overriding motivation is to

objectively observe what organizations are doing to improve capabilities.

Problem Statement

This study will explore the ways that selected organizations are working to improve

impact measurement practices in the nonprofit sector. The question focuses on to what extent

and in what ways are the leaders of selected organizations seeking to improve the way that they

measure organizational impact? This question begins with how do these organizations currently

approach impact measurement and analysis? Organizations need to demonstrate their value to

those who provide support for their efforts, activities and programs. If we assume that measuring

impact—even when such efforts are imperfect or less-than precise—is essential for

organizational accountability, then there may be value in observing how organizations actually


9

approach the effort, and the extent to which they are seeking ways to better communicate their

achievements.

This study explores the specific metrics or indicators of impact that are currently

important to a small selection of subject organizations, and identifies the extent to which they are

satisfied that their measurement methodology is succinctly capturing and communicating their

impact. The selection of the specific indicators will be of interest and will hopefully shed light

on how impact is understood by leaders who are sampled in this study. The detailed mechanical

processes that measure impact and performance will not be directly studied, but it is anticipated

that these concerns will have a bearing on the extent to which the leaders are satisfied with their

current measurement direction. Another aspect of understanding what is being measured is to

also compare it to what is not being measured. Exploration of such omissions may help us to

understand why the variables selected for measurement were chosen.

The topic will be explored from three unique perspectives, described below.

The historical, case specific perspective of The Chanda Plan Foundation, a

nonprofit in Colorado with a mission to provide alternative and integrative

therapies to disabled persons to improve their quality of life, and to demonstrate

the benefits of such therapies so that Medicaid will cover the cost of these

treatments to the disabled. This project researcher’s involvement over time

contributes to this collection of data, as does the interview with one of its board

members.

The perspective of two Denver organizations, working in education and Youth

Development:

o Denver Kids, Inc., a mentoring program which supports DPS

students, grades K-12, who face the personal challenges of higher risk

environments, to successfully complete high school, pursue a post-

secondary option, and become contributing members of the community.

(Who We Are: Denver Kids, Inc, n.d.)


10

o Project PAVE (PAVE), an organization, working with children, youth and

families to prevent relationship violence and abuse. The organization's

goal is to stop the generational cycle of relationship violence through

prevention, education, and early intervention. (Home: Project PAVE ,

n.d.) This project researcher’s involvement over time contributes to this

collection of data, as does the interview with one of its board members.

The perspective of a professional evaluation firm, the National Resource Center,

Inc. (NRC), a research firm specializing in performance measurement and

evaluation. (Home: National Resource Center, Inc, 2012)

The selection of subjects is designed to reveal differences and similarities across a variety

of organizations and from different perspectives in the sector. The Chanda Plan is a health and

therapeutic focused organization, with a measurement and evaluation history well known to the

researcher. The Chanda Plan Foundation is included in the study due to their intense impact

measurement process relative to their advocacy efforts. The two organizations that focus

primarily on adolescents provide the opportunity for comparison within a sub-sector. The

inclusion of the evaluation consultancy group will lend a broad—and hopefully balanced—

perspective on the practices and attitudes currently observed by them in their professional

engagements.

Literature Review

This chapter will discuss the nature and definition of impact and organizational

performance; current perceptions of and concepts around the value, necessity and utility of

impact measurement; observed approaches to measurement in practice; along with the challenges

and difficulties concerning measurement of impact.


11

The Nature and Definition of Impact

Discussion of impact and organizational performance in the third sector has become a

much written-about topic. The book Forces for Good: The Six Practices of High-Impact

Nonprofits, (2007) by Grant and Crutchfield, presents the qualities of high-impact organizations,

and begins with an acknowledgement of the difficulty in defining the term. “Our first major

challenge was to define success in a sector that has no universal means of measuring it.” (p. 225)

Without the use of financial metrics, as used in the for-profit sector, the lack of accepted and

universal nonprofit measures of success present a major challenge. Grant and Crutchfield

acknowledge that defined outcomes will be varied, and are dependent upon “…mission, model

and issue area.” After exploring the theory that budgetary information might be used for their

study purposes to gauge organizational impact, the authors concluded, “…budgets are not useful

for measuring impact or effectiveness.” (Grant, 2007, p. 226) The size of a budget can certainly

point to the scope and extent of an organization’s activities, but it really does not address the

results achieved by those activities, or the worth of those results. After first determining the

parameters for the design of their study, they decided on a “two-pronged” definition of impact.

(p 227) The first part is comprised of measurable outputs, and addresses the question: “did the

organization achieve substantial and sustained results at the national or international level?”

(p.227) The second part of the definition was “..more abstract and qualitative. We wanted to

study organizations that had achieved the most significant impact on a system or provided a

model that other groups had adopted.” (Grant, 2008, p. 227) The first part of this definition could

readily be used in discussing any organization, with the “national or international” level modified

to serve the specific realities of the organization. For example, if we assume that a small


12

organization can have dramatic impact in a small community, then a localized geographical

scope can be part of the stated goal against which to measure real achievement. While the second

part of their definition certainly recognizes that impact includes abstract and qualitative

components—and thus can be difficult to measure—it seems to this project researcher to be very

customized to the specifics of their study. It may not be a desired goal for most organizations to

change existing and established systems or to be held up as a model for others to replicate.

In The 21st Century Nonprofit, Firstenberg (2009) echoes the observation on the absence

of a profit motive: this “…does not mean nonprofits cannot be efficiently run, provided they

devise a system of measuring how effective and efficient their operations are.” Rigorous review

of the measures is essential to nonprofit efficiency. (Firstenberg, 2009)

A good way to understand impact is to consider the changes produced by the activities of

the organization. What changed and what differences were produced as a result of its efforts?

What difference did it make? (Stubbs, 1998) Stubbs relates this question to nonprofit measures

of achievement, which involves an “acceptance of standards of evaluation.”

Ralser (2007) introduces the concept of organizational value, and the Organizational

Value Proposition, which is concerned with the value of outcomes. He proposes this as a concept

that organizations must embrace to achieve long-term sustainability. The interchangeability of

related terms comes into focus too, when he says, “Outcomes are broadly defined as the impacts

the organization wants to have on people or society.” (2007, p. 81) Other authors define impact

in almost the reverse, as being comprised of outcomes.

In Nonprofit Lifecycles, Stevens, (2008) explains capacity for nonprofits from the

perspective of three concepts: “capacity, performance, and effectiveness create a continuum of

organizational success with capacity as the means, performance the measurement, and


13

effectiveness (shown as impact in her illustration) as the ultimate goal.” (2008, p. 13) This

understanding of impact is most commonly seen in the literature, and is based on what ends are

hoped for in the operating and delivering of nonprofit outputs. It seems clear that in order to

measure performance, an organization must know what that ultimate goal will look like. The

ability to articulate what success will look like is the first step in selection of indicators of

desired impact.

Similarly, Saul in Benchmarking for Nonrofits: How to Measure, Manage, and Improve

Performance, ( 2004) understands impact as the nonprofit’s “equivalent to profit…While

businesses make a profit, nonprofits make a difference.” (p. 2) His premise is that if a process

can be “quantified and measured, it can also be benchmarked and improved.” (Saul, 2004)

Finally Flynn, (2001) makes further distinctions between outcomes as the “what

occurred” factor of impact measurement, and the inclusion (in impact) of “why” the change

happened. Impact includes typical views of outcomes, but assigns credit or blame on the

outcome produced. (p. 8) Her approach relies on cause and effect.

To talk about impact measurement, one also needs to understand the definition of

indicators, as this is essential to the process. Flynn (2001, p. 23) defines these as “statistics that

are collected or used with a purpuse and imbued with meaning...” Indicators point to what

“should be” for comparison to what “is”. The development of appropritate indicators should

never be sloppy, as a good indicator can clarify relationships and values, even when the

mechanical instruments make the final data reports appear less than precise. Even with less than

perfect technical processes, an organization can gain valuable information if the right indicators

are selected. (Flynn, 2001)


14

While the literature suggests a variety of definitions, there is much support for the

definition (as previously described in the introduction) of impact used in this paper: the changes,

or the perceived changes created by the work or programs delivered. The changes can be at the

individual, community, or societal level. Impact often goes a step further, however,

encompassing the value or worth of those outcomes, or a measure of the ripple effect created in

community or society by certain outcomes. Impact is the result of the outcomes, and impact

cannot always be measured. Most importantly, though, impact will be subjective, dependent

upon the realities of each organization.

Perceptions, and Concepts in Impact measurement: Its Value, Necessity and Utility

In The End of Fund Raising, Saul, (2011), observes that changes in the world at large

have changed the landscape for nonprofits. As nations, cultures and communities face economic,

social and environmental crises, new solutions outside established conventions are being sought.

This has led to the rising interest in the social capital market. “Today social change is no longer

exogenous to our economy. As a result, neither are nonprofits.” (p.16) He proposes that this

means that the “product” manufactured by nonprofits is social impact, and this, in his view has

become mainstream economic currency. If what he states is true, it goes a long way to explain

much of the value, current necessity and increased interest in impact measurement. In the social

capital markets, impact can be quantified in economic terms, beyond a simple reporting of

outputs. (Saul, 2011) Further, Saul reminds us, it is perfectly acceptable to have economic

benefit resulting from charitable activities. He also contends that because of the importance of

solving social and environmental problems—due to the perceived urgency of the issues—people

really do need to know what is being achieved by nonprofit activities. Performance is something


15

that is of real concern. “Nonprofits will no longer be able to duck the measurement question by

citing the complexity of their work.” (Saul, 2011)

The value of impact measurement as a promotional and relationship-building tool is

gaining more attention. Ralser’s (2007) work on impact measurement focuses largely on what he

sees as the motivation and needs of today’s sophisticated donor. He characterizes these donors

as investors in social change, and finds that “prospective investors in a nonprofit want to know

‘what was accomplished with my money?’ not ‘how much was spent getting my money?” (2007,

p. 6) Indeed nonprofit practitioners and the philanthropic community have long focused on the

fund raising ratio as an important metric, highlighting what is spent by organizations in the fund

raising effort. This ratio is still used extensively to “rate” the performance of nonprofits. Ralser

argues that this model of measurement will not serve organizations well as the competition for

funding becomes more pronounced. Donors want to know to what extent negative social

conditions are being eliminated, and to what extent positive social change is being created.

(Ralser, 2007, p. 6) Saul, (2011, p. 23) echoes Ralser’s view, and when he talks about

innovation, he speaks of donors as those who are really “buying impact, not just funding

programs.” Saul promotes the Value Proposition concept as a powerful way to articulate impact,

sell impact and ultimately to attract “impact buyers”—those who will support and invest in the

efforts.

The ratio that really matters, to contemporary philanthropists and social investors,

according to both Ralser (ROI, 2007) and Saul (2011) is the ROI, or return on investment, not as

defined in the for-profit sector, but as a measure of efficiency in the nonprofit sector. As

described by Saul, ROI allows stakeholders to see what impact is produced, and at what rate of


16

efficiency, or “bang for the buck.” When data such as this is used to communicate with

stakeholders, the relationship changes.

Approaches in Impact Measurement

The extent and degree of actual impact measurement practice according to writings

reviewed shows a range of approaches. According to “Nonprofit Governance in the United

States, Findings on Performance and Accountability” (Ostrower, 2007) was able to show that

approximately 26% of boards surveyed did not evaluate at least once every two years whether

their organization was accomplishing its mission. She does not discuss the implication of this

statistic: that this may mean that a higher number do evaluate at least this frequently.

One approach to impact data utilization was highlighted in the literature repeatedly is

impact as a tool for strategic planning. “Value metrics” bring objectivity to the process of

strategic planning, because “decisions can be based on the value of the outcomes produced.”

(Ralser, 2007, p. 75) When the strategic plan is developed with impact and performance

measurement in mind, it becomes a plan that has great value as a management tool. It becomes a

road map focused on definitive destinations. Talley echoed this thee, (Using Imperfect Metrics

Well: Tracking Progress and Driving Change, 2010) stating, “Without a means to measure

progress, strategic plan implementation is ineffective at best and impossible at worst.”

Organizations need hard data on outcomes in order to manage and assess the strategic plan. It is

often the case that strategic plans are developed after intensive and time consuming efforts, only

to sit on a shelf without further reference once compiled. The lack of measurability in the

composition of the strategic plan may be a major reason for this lost opportunity. Considering

metrics as plans are developed and then adopting a regular discipline of measurement ensures

that the strategic plan has value and utility to the organization.


17

Greenway (2001) published her study of working approaches to measurement in The

Emerging Stastus of Outcome Measurement. The various types of measurement tools were

observed in her work. The organizations she observed as engaging in rigorous measurement

practices included Girls Incorporated, Big Brothers, Big Sisters of America, Girl Scouts of the

USA, The American Red Cross, and several Certification and Accreditation organizations, all of

which have sophisticated national/affiliate organizational structures. Her study concludes that

most smaller human service organizations work in isolation to address measurement processes,

and much needs to be developed to provide essential and reliable tools and methods. A platform

for sharing of successful methods would benefit the sector, and could be especially valuable to

those smaller organizations not supported by a national office or umbrella organization.

Proper implementation and method selection is key to effective measurement, but only a

small amount of the available material on this topic was reviewed. One useful piece was

“Measuring Social Impact: Seven Deadly Sins of Impact Evaluation” (Forti, 2012). Knowing

what to avoid is an important step to understanding best practice and developing a quality

program. Forti promotes, among other things, the formation of an evaluation advisory

committee, the acceptance that impact is an ongoing process, and that methods do not always

have to be scientifically correct to be of great value. Another approach mentioned repeatedly is

benchmarking, deemed especially useful to nonprofits for problem solving, (Letts, Ryan, & and

Grossman, 1999) but difficult to implement in the nonprofit realm. Benchmarking utilizes,

external, comparative data to gauge the performance of the organization.

Challenges and Difficulties in Performance Measurement

While the literature reveals much about the obstacles to reliable and credible

measurement practices, Saul, in The End of Fund Raising: Raise More Money by Selling Your


18

Impact, (2011) says that the selection of indicators—and thus the entire process— becomes

much easier when nonprofits begin at the right place. He says that the right place is the knowing

of “exactly who (your) stakeholders are and what they want!” Clarity around this foundational

point is essential to impact measurement. He prescribes less reliance on fancy methodology,

expensive software and consulting teams, and recommends to organizations “let your

stakeholders define your success”. (Saul, 2011, p. 60) He does not discuss in direct terms the

issue of competing goals, different perceptions and cultural differences between the stakeholder

groups and how that might complicate the task. He does discuss the complexity of stakeholder

engagement to arrive at understanding of what stakeholders want. He sees great relationship-

building value in a proactive stakeholder engagement process, beyond merely assisting in the

identification of impact indicators. The benefits to the organization are far reaching.

Some of the authors reviewed spoke to some underlying philosophical concerns with

measurement in the nonprofit sector. DiMaggio (2001), discussed “sentimental attachments to

rationality” inherent in measurement processes, leading —according to DiMaggio—to unrealistic

expectations of rational techniques. There are certain instances, he points out, where rationality is

ill suited for the subject at hand. He concludes however, that even when rationality fails, the

application of rational approaches “may move people to self-improvement and cooperative

actions and so enhance the capacity of organizations.” (DiMaggio, 2001, p. 250) It becomes, for

many organizations, a “legitimatizing ritual” demonstrating a willingness to take responsibility

and commitment to accountability. Essentially, he proposes introduction of logic and mechanical

processes to balance the “softer” side of nonprofit organizations: the compassion motive. It

seems reasonable that his suggestion incorporates an emphasis on competency while not

replacing or undermining the passion-driven model.


19

Perhaps even more profound is considering the words of Weisbrod (2001), warning that

what is not measured—because it may be impossible to do so—may in fact be the very important

and critical deliverable of the nonprofit organization. The danger is that those unmeasured

outcomes may, by default, be valued at zero. The resulting misallocation of resources or strategic

decision-making can be significant. (2001, p. 6)

Measurement, when well-designed and well-implemented, is seen as an exhausting,

resource-intensive and expensive endeavor. It can be equally difficult when practiced in the most

simplified and crude levels. According to Talley and Fram (2010) , nonprofits must use whatever

value they can find in even the most imperfect of metrics. The science behind sophisticated

methodologies of analysis may exist, but nonprofits simply don’t have the resources to approach

measurement in sophisticated ways. (Talley, 2010) They describe the “imperfection” of typical

measurement as being related to anecdotal, subjective, interpretive or qualitative metrics, or

those based on small samples or situations that are not easily replicated. They conclude that

rather than strive for a perfect system or abandon the process altogether, organizations should

strive for using imperfect metrics well. (Talley, 2010) A professional, well-designed approach

will overcome many of the problems associated with imperfect data. They suggest

professionalizing the management of impact measurement by adopting discipline in the process.

Examples include internal agreement on relevant outcomes, agreement on specific indicators,

and establishing rules for comparison of outcomes to strategic objectives. Their approach

demands much emphasis on relationship factors, which they see as critical to a positive result.

For example, developing a respectful, transparent and credible measurement process builds

essential trust.


20

To summarize, the literature reviewed reveals that there is a strong movement towards

not only outcome measurement, but towards a more expansive measurement of social and

community impact, while pointing out the challenges and difficulties of those approaches. In

spite of challenges, many nonprofits are looking for effective methodology and approaches,

because current social and economic realities demand it. As stated well by Saul (2011, p. 21),

“ ‘I don’t know’ becomes a very expensive proposition when people are attaching economic

value to actual results.” Stakeholders and the public demand to know how well—in credible

terms—organizations are performing.

Methods

This chapter will describe the approach used to identify and compare current attitudes

and practices related to impact measurement from three unique perspectives, with the specific

data collection strategies described below.

Subjects for the study were selected to show practices in a small group of similar

organizations working in the same sub-sector (youth services/education support), while also

illustrating practices in a same sized organization in the health care subsector. Because one of the

selected organizations is currently working with an outside consulting firm on an impact

measurement project, I saw this as an opportunity to draw upon their broader perspective.

Consistent with the project’s exploratory nature, this illustrative and observational approach

seemed appropriate at the outset.

The Study subjects are as follows -

1. The Chanda Plan Foundation. The Chanda Plan Foundation is a nonprofit in

Colorado with a mission to provide alternative and integrative therapies to

disabled persons to improve their quality of life, and to demonstrate the benefits

of such therapies so that Medicaid will cover the cost of these treatments to the


21

disabled. This project researcher has been engaged in a voluntary role at the

Chanda Plan Foundation for approximately 18 months. In this role I have worked

in the development of evaluation and outcome measurement. The historical notes,

records and reports will be utilized to build the timeline and story of this practical

application, according to mutual agreement established at the outset of the

evaluation project. The perspective of the CEO is woven into the study via this

historical involvement, i.e., through a series of conversations and discussions

during the course of the evaluation project. The perspective of one Board

member, Joe Brucker was collected via an interview on April 26.

2. Two Denver organizations, each working in educational support services and in

Youth Development: Denver Kids, Inc., and Project PAVE. A third organization,

Florence Crittenton Services of Denver was initially selected, but declined to

participate.

a. This project researcher worked two years ago with Project PAVE in a

consultancy capacity. The consulting work revolved around selection of

key performance indicators to support a strategic planning effort. This

points to some bias on the part of the researcher; in so far as interest in

seeing what the performance measurement processes are in place now,

relative to that earlier period. Certainly, there was a desire to see that

early effort may have created more focus in that organization in the arena

of impact measurement. Due to the previous involvement, gaining access

to the Executive Director was quite easy; Mike Johnson was happy to

participate, and the interview was conducted by phone on April 19.

b. The inclusion of Denver Kids, Inc was suggested by the course facilitator,

and upon review of the organization’s web site, it appeared that they were

well suited for this project. The website reveals a sophisticated level of

data sharing than is not always seen, which seemed to indicate a more

highly developed approach to impact measurement. It appeared that they

are actively engaged in measuring their clearly defined indicators of

success. The first contact with them resulted in a connection with the

CFO, who was eager to connect this project researcher to a board member.


22

Access to the board member arose, necessitating a change in the original

plan: ultimately the CFO provided responses via email due to the

timelines.

c. Study data was gathered via phone interviews with either the CEO or a

board member from each organization. Interview questions were

standardized, but interview formats were informal, and evolved per the

specifics of each situation. Due to the conversational tone of the

interviews, some non-standardized information is incorporated into the

final collection of responses.

3. A representative from The National Resource Center, Inc. an evaluation-

consulting firm was interviewed by phone on April 15. Interview questions were

customized to the nature of this participant’s work, while maintaining as much

consistency as possible. It was hoped that the inclusion of this unique perspective

would add valuable inputs to the study. This consulting firm is currently working

on an impact study for The Chanda Plan Foundation so that critical data can be

used in both a state and in a federal advocacy effort to change the benefits

provided to disabled people who receive Medicaid. The National Resource

Center, Inc.—as its name suggests—“is a research firm specializing n

performance measurement and evaluation.” They work primarily with local

governments, foundations and nonprofits. (Home: National Resource Center, Inc,

2012)

All those interviewed indicated interest in impact measurement, and they were eager to

discuss their experiences.

The project is exploratory in nature, and serves to inquire as to current practice. The

responses will be subjective, and the responses will be from the perspective of the person

interviewed.

A disadvantage of this collection of information is working with a small sample of

organizations. The time frame and scope of the project dictate this limitation.

The full set of interview questions is shown in Appendix A.


23

Process The responses collected from this survey were compiled in a consolidated format for

comparison purposes. An edited and condensed version of this is shown in Appendix B. This

then formed the basis of a narrative that describes trends and observations. The narrative below

describes the significant aspects of the responses.

The analysis, discussion and comparisons are not intended to show relative rating or any

hierarchical determination of the practices represented, but rather, to illustrate any noticeable

trends, and to understand the practices and experiences of those participating. This understanding

was helpful in the formation of recommendations for additional efforts, improvements and

further research.

Results, Discussion and Recommendations

This chapter reviews and discusses the content gathered during the interviews and

provides recommendations based on the information. It was anticipated, due to the variety of

study participants, that the measurement practices would be diverse and unique, and the findings

reflect that to be the case. To prevent unintended and out-of-context judgments from being made,

the identities of the nonprofit organizations have been somewhat disguised in the discussion,

while the participating consulting firm has been named throughout.

Findings Those interviewed were eager to share their views on the subject and to share details of

how they approach measurement, as well as some of their frustrations. It was clear not only from

the content of the responses, but from the tone as well, that there is genuine interest and

engagement around impact measurement. Those in leadership positions in organizations want to

know that they are making a difference, and want to utilize the best methods to communicate

their results.


24

Mission focused approaches. The organizations studied are firmly focused on mission

as the primary indicator of impact. As stated by one CEO, “ Mission is used to form every

decision in the agency.” That agency’s evaluations are made against the mission, and recent

evaluations drove the agency to make programming changes to ensure that they achieve mission

more effectively through their programming efforts.

The professional firm NRC, Inc reveals that they provide evaluation services mostly in

regards to individual outcomes, and that their clients are clear about linking results to mission.

The tools and materials they provide support the process from the earliest stage, explaining the

importance of mission and related indicator selection and the best standards for the selection

process so that indicators are closely aligned with mission.

Indicators of impact: selection. For some organizations, and at certain stages in

organizational life cycles, impact relates most directly to those served. The emphasis tends to be

toward: how are the lives or experiences of our clients changed as a result of the work of our

organization? as opposed to: how does our work change the community or society? NRC works

with many organizations to provide client-focused measurement services. Likewise,

Organization A uses many different instruments, most of which—at this time—emphasize the

individual impacts. Based on participant responses, these individual-focused approaches to

impact are effective and provide a definite value to these organizations.

The selection of indicators of impact, when looking at individual-focused measurement,

varies widely across the organizations observed. At the Chanda Plan, the indicators are very

specifically defined within the Quality of Life evaluation program—which is highly structured

both from a theoretical and a practical perspective—and are developed by the CEO and program

management team, with no involvement from the Board. For the development of the participant


25

survey instruments, the board was not consulted regarding the specific outcomes to be measured,

although the board does see the results reported in a dashboard format, quarterly.

Organization A has developed a set of clearly defined client-centered indicators which

are measured rigorously, but when initially asked if they have identified specific indicators, the

answer was “no”. Upon further exploration, it was clear that while the indicators were not

considered “formal” because the board had not undertaken any action to ratify or state them in

some official way, indeed the selected indicators were being used consistently, clearly and with a

high degree of transparency—in other words, all team members are clear about the specific

indicators and bench marks.

Organization B, is clear that their performance indicators drive the HR performance

management process as well, which while often the seen in the for profit sector, is less common

in nonprofit organizations.

There was evidence, too, that there is interest and activity in the arena of community and

social impact measurement. For example, NRC is engaged with Organization C to measure the

broader impact of the pilot program. This will be a powerful project, producing data that

potentially could change the way Medicaid funding is provided to those with spinal cord injuries

in the state of Colorado. Understanding how the program can impact the long-range health and

medical costs for these persons, and thus impact the funding sources is critical to the advocacy

efforts of the organization.

Indicator selection in regards to these far-reaching (looking at the community and

society) impact measurements reflects a high degree of specificity. For example, at Organization

C, in considering the potential benefits of the special waiver program (Colorado Pilot Program),

they view medical savings as a key indicator of success, and a measureable one as well. So, cost


26

comparisons between those in the program and those who are not are being developed for

rigorous analysis and demonstration of the impact—and thus the social value—of the program.

For Organization A, being able to demonstrate long-term behavior changes and life experience

improvements is key to demonstrating significant impact. While Organization A has not yet

developed those longitudinal study instruments, they have begun to identify and discuss the

indicators that would be used in such a study.

Measurement practices. Practices seem to be dependent upon the capability of the

organization and the relative value of impact measurement to its specific cause. For example,

Organization A has gradually developed evaluation methodology and related philosophy over its

26-year history, so it utilizes various tools well, and has the understanding in place of how to do

so even better. Its stability as an organization at this life-cycle stage puts it in good position to

develop those improvements effectively. Because of the collaborative nature of its partnerships

with the city and the Denver Public Schools in which it works—which carries with it an

significant accountability burden— Organization A has been motivated to continually develop

and improve upon its strong evaluation program.

Organization C has been able, at a relatively earlier life-cycle stage, to develop some

well-structured, reliable and useful metrics for its quality of life program. Their evaluation

program has formed a strong foundation for its further-reaching impact measurement relative to

its advocacy efforts and pilot program. The Foundation has been able to implement these

processes by leveraging its partnerships, including the local grant-making foundation

community, qualified volunteers, and its governmental project sponsors. This case illustrates the

relationship between the importance of demonstrable impact and the investment that is made in

the evaluation processes. Because expansion of advocacy efforts to other states and at the federal


27

level requires the demonstration of individual, community and social value, the need to develop

rigorous measurement practices was clear.

Organization B, uses their impact data to populate their internal dashboard reporting

tools, which are presented regularly to the board of directors. This is an excellent example of a

movement away from financially focused presentations, which was recommended throughout the

literature. As stated in the interview, “The most effective measurement tool is the dashboard

report that is presented to the board of directors each quarter (which connects performance to)

our Employee Performance Management System…it includes organization-wide goals,

individual performance planning (goal-setting), ongoing communication/dialogue and year-end

evaluations.” Putting impact measurement into management practice in this way enables

organizations to leverage their investments made in a complex process.

The clients served by the NRC are typically looking for a low-cost tool so that

they can meet funder’s evaluation requirements. This implies that they don’t currently have the

resources for more sophisticated processes. The template approach does a good job of measuring

the primary and most easily measured indicators of impact, while simplifying the process for

those administering the process. NRC does not provide facilitation services relative to the

selection of indicators; they provide the instructions so that clients can properly approach this

aspect of the project. Presumably, other needs related to impact measurements have not yet been

identified, simply don’t exist within those organizations, or are not yet being addressed due to

resource or capacity limitations.

What is not being measured? Organization A has been through a cycle of studying their

various evaluation tools and making programmatic changes based on measured results. Now they

are addressing gaps and looking to expand and improve. This means they are studying the


28

feasibility and practicality of implementing more broad based impact measurements: longitudinal

participant studies, peer group evaluations and community based approaches, and looking for

assistance and partners to help with these developments. Resources and capacity are a major

issue, as these efforts will require staff time, and financial resources. The interviewee was clear

about this gap in their overall approaches, and is developing strategies to implement far-reaching

measurement processes and reporting, but also acknowledged that perfection is not achievable,

and that it will always be an ongoing process of knowing there is more to be done.

The NRC interview revealed that most of their nonprofit work focuses on individual

outcomes, to the extent that it fits with the template they offer their clients. They don’t see a lot

of broader-impact studies being done, but that is because those studies are costly, and most

organizations don’t have the resources. The social-impact study being conducted by them for

Organization C is an exception and is being funded by the State of Colorado. This is consistent

with their business model; most of their large scope projects are conducted for their

governmental and municipality clients, where funds tend to be available to support the effort.

Board involvement in impact measurement. The extent of involvement varies from

one organization to another. At Organization A, the board is significantly engaged in the process.

The PAVE program subcommittee (a committee of the larger board) reviews impact reports

regularly, the last time being Fall 2012. The CEO states that the current measurement and

reporting processes are working well for the board, they understand, use and appreciate the

information that is being provided.

The discussion with the NRC representative reveals a different dynamic regarding board

involvement. In most cases, the nonprofit staff members are initiating the measurement

programs. In many cases, the first driver is external, that of funders who are requiring impact


29

data to support either a grant request or a grant report. Members of staff are the recipients of the

funder request, and those staff members then put the evaluation process into place. NRC sees

cases where the board is actually pushing back, hesitant to allocate the necessary resources.

At Organization C, in regards to its original program, the board is involved as a recipient

of the information (dashboard reports), reviewing the information thoughtfully, and then using

the data to make or ratify decisions. The board was not involved in the initiation, selection of

impact indicators, or direction of the evaluation program. When provided with the arguments and

recommendations of the CEO, they allocated the resources to fund the project, but the primary

driver was the CEO, and the volunteers, including members of the Healing Committee, who

serve the function of the program staff. The board did not define or establish the indicators at the

outset, but as the evaluation process has evolved it appears that they do have (based on the board

member interview) strong interest in the indicators that are being measured.

In all the study cases, there was the equivalent of an evaluation advisory group, whether

staffed by board members or staff member, tasked with overseeing and guiding the measurement

practices. This is consistent with the best practice recommendations made by Forti, (2012)

Future Direction As noted, organizations such as Organization A are looking forward to

ongoing improvements and expansion of both their scope of impact measurement and the

indicator selection. Both Denver Kids, Inc and Organization C are recognizing that limitations in

present capacity influence their evaluation efforts. There is a noticeable trend towards social

impact, but it seems to align with larger scale efforts and later life-cycle stages.

Discussion, Observations The discussions with the participants did verify that measuring program and

organizational impact is not easy, for a host of complex reasons. One of these is that


30

organizations and leaders within them understand the definition differently, and another is that

impact is not always quantifiable. This was brought up, for example, in the discussion of how to

measure advocacy efforts and impact. The role of the board in establishing of practice standards

and impact selection was not consistent from one organization to another, which is to be

expected.

The participants spoke directly and with clear awareness about the indicators that are not

being measured at their respective organizations. This was not always a point of frustration,

implying a broad acceptance that impact measurement is not an exact science and that perfect

measurement practices are not possible in the nonprofit sector. Some comments directly echoed

language seen in the literature review. In the words of one respondent, “it is not an end-game and

we will never be finished fine-tuning.” In spite of the recognized deficiencies, all participants

were clear that they find great value in the data they have available and general satisfaction with

the practices they’ve been able to put into place.

Conclusions

The participating organizations are actively engaged in impact measurement to varying

degrees, and with intentional focus on intensifying their measurement practices, so that

organizational performance can be improved. The professional consulting firm NRC stated that

their clients, while only at the beginning stage of evaluation programming, are engaged in

measurement efforts and have a “sincere desire to learn what needs to be better.” This was

echoed throughout the study. All of them are satisfied with their efforts and the tools they are

currently using.

A significant result of the study is the confirmation of the assumption in the problem

statement, that “how organizations define impact is purely subjective, dependent upon the


31

specific realities of their circumstances.” Some of them look only at client-specific metrics,

while some are looking at social impact. This works for these organizations; they address impact

from whatever perspective makes sense for their situation. All of the nonprofits participating

directly are proactively engaged in impact measurement, devoting significant time, attention and

resources to the efforts.

It is especially noteworthy that each of them is also engaged in partnering or

collaborative projects with other nonprofits and/or governmental agencies (Colorado Department

of Health Care Policy and Finance; Denver Public Schools). This project researcher concludes

that these collaborations impart—as noted in the earlier discussion—a high measure of

motivation, and sometimes a direct requirement of accountability, for the demonstration of

impact and performance. The organizations referenced by NRC—those just beginning to engage

in evaluation efforts—tended to be smaller, more self-contained organizations, but for them too,

the external stakeholders, i.e., their funders, were the primary motivators for their efforts.

From this observation, it would appear that as collaborations, innovative funding schemes

and partnering ventures increase in the nonprofit sector; the practice of impact measurement will

become more widespread. It is further predicted that this trend will drive more sophisticated

approaches and measurement projects will target larger scale indicators, so that a social

perspective will more often be the norm. Organizations that have already begun to build

evaluation practices will be able to build upon those, as the study participants have done.

The perceived connection between measurement practices and community/public

partnering demands further research. Is there a definite trend in the direction of partnering and

new funding strategies for the nonprofit sector? If so, how does that directly define the

requirements for impact measurements? Such a study would complement this one greatly.


32

This project would have benefited from a larger pool of participants; the timeline of the

project imposed limits on the size of the pool. The results are useful nonetheless. Based on the

literature and the information collected from the respondents, it is not expected that additional

observations would reveal significant differences in general, but specific variations would be

interesting to explore.

The interview questions did not include a reference to the anticipated audiences who

would potentially see the data. This would be an interesting topic, because the target audience

may drive the selection of indicators. Similarly, the study did not explore the various

stakeholders who drive the objectives of the organization, and how that relates to impact

measurement, as suggested in the literature.

As suggested by various literature references, benchmarking for nonprofits might hold

value and exploring how that is being implemented might be an excellent topic for further

research. An interesting aspect of benchmarking is how it might serve to increase cooperation

and collaboration among organizations in the sector—which this study has shown to be an

important factor in impact measurement.

From a theoretical point of view, impact measurement is an inexact and evolving science.

In practice, this is confirmed—it is an developing, imperfect process for most organizations, but

one that holds value and utility, for decision making, accountability and for relationship building.


33

References

Anheier, H. K. (2005). Nonprofit Organizations: Theory, Management and Policy. New York,

N.Y.: Routledge. (pp 225-241)

Carver, J. (1997). Boards That Make a Diference (2nd ed.). San Francisco, CA: Jossey-Bass.

DiMaggio, P. (2001). Measuring the Impact of the Nonprofit Sector on Society is Probably Impossible but Possibly Useful: A Sociological Perspective. In Flynn, P. & Hodgkinson, V. A., (Ed.), Measuring the Impact of the Nonprofits Sector. (pp 249-272) New York, NY: Kluwer Adademic/Planum Publishers.

Firstenberg, P. B. (2009). The 21st Century Nonprofit: Managing in the Age of Governance (2nd ed.) New York, NY. Foundation Center.

Flynn, P. & Hodgkinson, V. A., (2001). Measuring the Contributions of the Nonprofit Sector. New York, NY, USA: Kluwer Academic/Plenum Publishers.

Forti, M. (2012 Feb22). Measuring Social Impact, Seven Deadly Sins of Impact Evaluation. Stanford Social Innovation Review

Grant, H. M. & Crutchfeld, L. R. (2007 vol/ Fall). Creating High-Impact Nonprofits. Stanford Social Innovation Review .

Grant, H. M. & Crutchfeld, L. R. (2008). Forces for Good: The Six Practices of High-Impact Nonprofits. San Francisco, CA: John Wiley and Sons, Inc.

Greenway, M. T. (2001). The Emerging Status of Outcome Measurement in the Nonprofit Human Serice Sector. In Flynn, P. and Hodgkinson, V. A., (Ed.), Measuring the Impact of the Nonprofit Sector. New York, NY: Kluwer Academic/Plenum Publishers.

Home: National Resource Center, Inc. (2012). Retrieved 2013, 7-April from National Resource Center, inc: www. n-r-c.com

Home: Project PAVE . (n.d.). Retrieved 2013, 20-March from Project PAVE Web Site: http://www.projectpave.org/

Letts, C. W., Ryan, W. P., & Grossman, A. (1999). High Performance Nonprofit Organizations: Managing Upstream for Greater Impact. New York, NY: John Wiley & Sons, Inc.

Ostrower, F. (2007). Nonprofit Governance in the United States: Findings on Performance and Accountability from the First National Representative Study. the Urban Institute Center on Nonprofits and Philanthropy. The Urban Institute.


34

Ralser, T. (2007). ROI for Nonprofits: The Key to Sustsainability. Hoboken, NJ: John Wiley & Sons, .

Saul, J. (2004). Benchmarking for Nonrofits: How to Measure, Manage, and Improve Performance. Saint Paul, MN, USA: Wilder Publishing.

Saul, J. (2011). The End of Fund Raising: Raise More Money by Selling Your Impact. San Francisco, CA, USA: Jossey-Bass.

Stevens, S. K. (2008). Nonprofit Lifecycles: Stasge-Based Wisdom for Nonprofit Capacity (2nd ed.). Wayzata, MN: Stagewise Enterprises, Inc.

Stubbs, R. (1998, January). A Recipe for Non-Profit Success: Managing The Linkages and Key Elements of Successful Organizations. Fund Raising Management .

Talley, J. L. (2010). Using Imperfect Metrics Well: Tracking Progress and Driving Change. Leader to Leader (vol 55), (pp. 52-58).

Weisbrod, B. A. (2001). An Agenda for Quantitative Evaluation of the Nonprofit Sector: Need, Obstacles, and Approaches. In Flynn, P. & Hodgkinson, V. A., Measuring the Impact of the Nonprofit Sector (pp. 273-290). New York, NY, USA: Kluwer Academic/Plenum Publishers.

Who We Are: Denver Kids, Inc. (n.d.). Retrieved 2013, March 20 from Denver Kids, Inc. Web Site: http://denverkidsinc.org/who-we-are


35

Appendix A Interviewee:

Title:

Organization:

Date of interview:

An outline of interview questions is as follows:

1. What is the mission of your organization? Does your organization use it as the

primary statement of purpose against which you try to assess your impact on

individuals and the community?

2. How are formal indicators of performance selected on the behalf of the

organization?

3. What are the 2 or 3 best indicators of impact for your organization?

4. How does your organization view the importance impact measurement?

5. Of the indicators mentioned, which, if any are you currently attempting to

measure?

6. What is your most effective tool for impact measurement? (please describe this)

7. When was the last time the board addressed the measurement standards being

used—for best fit, for reliability etc.

8. What important known impacts are not being measured, and why?

9. Has any member of the board every expressed any frustrating on measurement

reporting standards at the organization?

10. Have you ever engaged professional help in developing, assessing or improving

measurement practices?

Approaches in Impact Measurement

Documents

Transcript of Approaches in Impact Measurement