Analysis of Dissemination and Impact of the Integrated Delivery System Research Network. Washington,...

Contract No.: 233-02-0086 MPR Reference No.: 6029-310

Analysis of Dissemination and Impact of the Integrated Delivery System Research Network Final Report December 2004 Marsha Gold Erin Fries Taylor Tara Krissik

Submitted to:

DHHS/AHRQ Redland Technology Center 1st Floor 540 Gaither Road Rockville, MD 20850

Project Officer:

David Introcaso

Submitted by:

Mathematica Policy Research, Inc. 600 Maryland Ave., SW, Suite 550 Washington, DC 20024-2512 Telephone: (202) 484-9220 Facsimile: (609) 863-1763

Project Director:

Marsha Gold

iii

ACKNOWLEDGEMENTS

David Introcaso at AHRQ was our project officer for this study, continually encouraging us to focus not just on the narrow questions but the broader ones as well. Cynthia Palmer at AHRQ is responsible for the IDSRN program and provided substantial support in gaining access to information and people. We are indebted to many staff at AHRQ and with the IDSRN partners and their collaborators for their willingness to talk openly with us about the program and their experience. At Mathematica, Debra Draper provided critical staff leadership in interviewing partners and collaborators. Judith Wooldridge served as an outside reviewer for the report and project as a whole, providing valuable insights at multiple stages. Felita Buckner provided secretarial support to the entire product and produced this document. Daryl Hall provided editorial support.

v

CONTENTS

Chapter Page

EXECUTIVE SUMMARY.........................................................................................xiii

I OVERVIEW OF THE IDSRN AND EVALUATION OBJECTIVES.......................... 1

A. PROGRAM DESCRIPTION .................................................................................. 2

1. IDSRN Goals................................................................................................... 2 2. IDSRN Program Timeline and Partners.......................................................... 3

B. EVALUATION OBJECTIVES AND CONSTRAINTS ........................................ 6

II DESIGN AND METHODS ........................................................................................... 9

A. THEORETICAL FRAMEWORK: LOGIC MODEL FOR THE IDSRN ............. 9

B. ANALYTIC APPROACH .................................................................................... 13

1. Assessment Area 1: Moving Research to Action......................................... 13 2. Assessment Area 2: IDSRN Administration................................................ 14

C. DATA SOURCES, COLLECTION, AND ANALYSIS METHODS.................. 15

III FINDINGS ON IDSRN FINANCING AND TASK ORDERS................................... 19

A. IDSRN FINANCING AND SOURCES OF SUPPORT ...................................... 19

B. AWARDS PROFILE ............................................................................................ 21

1. Size ................................................................................................................ 21 2. Topics ............................................................................................................ 21 3. Portfolio Areas .............................................................................................. 22 4. Deliverable Requirements ............................................................................. 23 5. Distribution of Awards by Partner/Collaborator Team................................. 23

C. IDSRN ADMINISTRATION WITHIN AHRQ ................................................... 24

CONTENTS (continued) Chapter Page

vi

IV TASK ORDER TYPES AND OUTCOMES ............................................................... 29

A. IDSRN PROJECT STRATEGIES........................................................................ 29

B. PROJECT OUTCOMES....................................................................................... 33

1. Measurement of Outcomes............................................................................ 33 2. Findings ......................................................................................................... 34 3. Operational Effects by Task Order Type ...................................................... 35 4. Conclusions ................................................................................................... 39

V CHARACTERISTICS AND PERSPECTIVES OF IDSRN PARTNERS/COLLABORATORS.............................................................................. 41

A. CHARACTERISTICS OF IDSRN TEAMS......................................................... 41

B. TEAM STABILITY.............................................................................................. 47

C. HOW TEAMS WORK.......................................................................................... 49

1. Task Order Decisions and Response ............................................................. 49 2. Focus of Task Order Work............................................................................ 51 3. Conducting Task Order Work ....................................................................... 52 4. Disseminating Results and Translating Them into Action............................ 53

D. IDSRN: PERCEPTIONS VERSUS EXPECTATIONS ...................................... 59

1. IDSRN Partners/Collaborators ...................................................................... 59 2. Congruence Between IDSRN and AHRQ Views ......................................... 60

E. PARTICIPANTS’ ASSESSMENT OF THE IDSRN AND SUGGESTIONS

FOR THE FUTURE.............................................................................................. 61

1. Overall View of the Program ........................................................................ 61 2. Issues for Future Attention: “The Big Picture” ............................................ 62 3. Issues Needing Future Attention—Narrower Concerns................................ 67

VI ASSESSMENT: IDSRN INFRASTRUCTURE TO SUPPORT EFFECTIVE TRANSLATION OF RESEARCH TO ACTION........................................................ 71

A. FOUR COMPONENTS ESSENTIAL TO THE IDSRN INFRASTRUCTURE ........................................................................................... 71


vii

VI (continued)

B. CLARITY IN GOALS AND PRIORITIES.......................................................... 72

1. Areas of Strength........................................................................................... 73 2. Areas of Weakness ........................................................................................ 74 3. Recommendations ......................................................................................... 75

C. PARTNER/COLLABORATOR TEAMS APPROPRIATE TO THOSE GOALS.................................................................................................................. 76

D. TASK ORDER FOCUS AND STRATEGY ........................................................ 78

E. UNDERLYING INFRASTRUCTURE TO SUPPORT ADOPTION AND DISSEMINATION................................................................................................ 78

VII ASSESSMENT: IDSRN ADMINISTRATION.......................................................... 81

A. OVERALL PLACEMENT, FUNDING AND DECISION MAKING ................ 81

1. Present Structure and Processes .................................................................... 81 2. Strengths and Weaknesses ............................................................................ 82 3. Recommendations ......................................................................................... 82

B. PROCUREMENT DECISIONS-MASTER CONTRACT ................................... 83


C. PROCUREMENT DECISIONS—TASK ORDERS............................................ 85


D. TASK ORDER OVERSIGHT .............................................................................. 88

1. Current Structure and Processes.................................................................... 88 2. Strengths and Weaknesses ............................................................................ 89 3. Recommendations ......................................................................................... 90

E. GENERAL INFRASTRUCTURE FOR IDSRN WITHIN AHRQ ...................... 91


viii

VII (continued)

1. Current Structure and Processes.................................................................... 91 2. Areas of Strength and Weakness................................................................... 91 3. Recommendations ......................................................................................... 92

F. THE QUESTION OF CONTRACTS VERSUS GRANTS.................................. 93

REFERENCES............................................................................................................. 95

APPENDIX A: CASE STUDY ANALYSIS............................................................A-1

APPENDIX B: TASK ORDER CHARACTERISTICS ........................................... B-1

ix

TABLES

Table Page

I.2 IDSRN PARTNERS AND COLLABORATORS AT TIME OF AWARD................. 5

II.1 TOPICS FOR INTERVIEWS WITH AHRQ STAFF ................................................ 17 II.2 TOPICS FOR INTERVIEWS WITH PARTNERS/COLLABORATORS ................ 18 III.1 IDSRN TASK ORDER AWARDS AND FUNDING SOURCES, FY 2000-FY 2003 ....................................................................................................... 19 III.2 IDSRN TASK ORDER AWARDS BY PRIORITY AREA, FY 2000-FY 2003 ....... 22 III.3 TASK ORDER AWARDS FY 2000-FY 2003 BY AHRQ PORTFOLIO AREA ..... 24 III.4 IDSRN TASK ORDER AWARDS FY 2000-FY 2003, BY TYPE OF DELIVERABLES ............................................................................... 25 III.5 DISTRIBUTION OF IDSRN TASK ORDER AWARDS AND FUNDS FY2000-FY2003 BY TEAM ...................................................................................... 25 III.6 DISTRIBUTION OF TASK ORDERS BY AHRQ CENTER ................................... 27 IV.1 IDSRN AWARDS FY 2000-FY 2003 BY TYPE ...................................................... 30 IV.2 IDSRN TASK ORDER OUTCOMES BY PROJECT TYPE, FY 2000-2003........... 35 V.1 CHARACTERISTICS OF IDSRN PARTNER/COLLABORATOR TEAMS AND WORK ................................................................................................ 42

xi

FIGURES

Figure Page

II.1 A STYLIZED LINEAR MODEL OF ACTION................................................... 10

xiii

EXECUTIVE SUMMARY

The Integrated Delivery System Research Network (IDSRN) is a major initiative of the Agency for Healthcare Research and Quality (AHRQ) that seeks to further the agency’s goals of knowledge transfer and movement of research to practice. Through the IDSRN, AHRQ has, since 2000, contracted with nine teams that include researchers affiliated with a diverse array of large delivery systems. During fiscal years 2000-2003, the IDSRN awarded 58 task orders, mostly on a competitive basis. The IDSRN spent $14.2 million over this period, including $6.5 million in core AHRQ funds, $4.0 million from AHRQ’s dedicated patient safety funds, $3.0 million in department bioterrorism funds, and $1.1 million from other sources. The IDSRN is scheduled for renewal in 2005, and this project was solicited by AHRQ to help them understand whether there are ways the IDSRN could be strengthened in the future.

The IDSRN evaluation that Mathematica Policy Research, Inc. is conducting addresses

questions in three main areas:

• Assessment Area 1: To what extent is the IDSRN’s infrastructure designed to encourage “real-world” adoption of research findings? As AHRQ looks at program renewal, what types of changes in the infrastructure might be considered to clarify IDSRN goals and to promote “moving research to action” as an explicit goal?

• Assessment Area 2: What are the strengths and weaknesses in the way the IDSRN is administered? Are there ways in which administration could be improved? To what extent has the IDSRN’s use of contract awards supported or impeded IDSRN goals?

• Assessment Area 3: What major accomplishments achieved under task orders seem most successful? What can we learn from these selected task orders about successful examples of translating research to practice and the characteristics that advance or constrain the adoption and dissemination of changes in practice-based research?

This final report includes findings in the first two areas and provides an overall assessment of outcomes across the task orders. Case studies of successful task orders are included in Appendix A, completing the assessment in area 3. The report draws on analysis of multiple information sources gathered in the year since this project was first funded. Sources include: initial interviews with program staff and a review of core documents; an analysis of the task orders funded between FY 2000 and FY 2003 based primarily on document review; interviews with AHRQ staff in the leadership of the agency, in the IDSRN including, its task orders monitors, and in the contracts office; and interviews with partners and their collaborators based on a combination of site visits and telephone interviews.

The report describes and analyzes the program with the aim of helping AHRQ staff

understand how the IDSRN works on the ground, what it has accomplished, how it is viewed by its partners and collaborators, and the implications for questions relevant to the first two assessment areas. Chapter I provides background on the IDSRN and this evaluation. Chapter II reviews the theory and methods behind the analysis, and Chapter III profiles IDSRN’s funding

xiv

and task order awards. Chapter IV describes the IDSRN partners and their collaborators, how they work within the IDSRN, and how they view the program. Chapter V analyzes how task order topics and experiences relate to AHRQ’s interest in moving research to practice. The last two chapters (VI and VII) build on these findings to address the specific questions raised in Assessment Areas 1 and 2 and to make recommendations for future change.

FINDINGS

Funding Levels and Awards

• IDSRN funding levels have fluctuated substantially over time, totaling $2.4 million in FY 2000, $3.5 million in FY 2001, $5.3 million in FY 2002, and $2.9 million in FY 2003, with an additional $6.3 million or more expected in FY 2004.

• IDSRN task order awards have varied substantially. Our analysis indicates that task orders have been awarded in every portfolio area except prevention and training. Most awards have been in five areas: quality improvement and patient safety; bioterrorism; cost, organization and socioeconomics; health information technology; and data development. In an effort to encourage dissemination, IDSRN task orders have diversified the types of deliverables they require. Of the 58 task orders, only 10 require just a final report. Requirements for presentation and manuscripts are common, and tools to aid implementation are required for some types of projects. Beyond this, the IDSRN has very limited infrastructure or resources to support communication and follow-up activities that might support work to leverage task order experiences. Almost all resources are devoted to task orders and their oversight, with only a single person at AHRQ responsible for the program overall.

Team Characteristics and Views

• The nine IDSRN teams vary substantially, with the lead partner for five based in delivery systems and four in independent research firms or universities. A researcher attracted by the applied focus leads all except one team; the system CEO leads Denver Health. All teams have actively participated in the IDSRN. Only three have had fewer than six awards: Marshfield/Project Hope, whose renewal was on hold pending resolution of various issues; Emory, whose ability to compete was affected negatively by their movement from Aetna to a university base; and the University of Minnesota.

• IDSRN partners and collaborators did not have a strong sense of what to expect from the IDSRN when the program started, but most feel relatively satisfied with their participation and what they view as interesting projects. However, both within AHRQ and within the teams different views exist on what it means to turn “research into practice,” and most views seem formed in general rather than specific terms. Partners and their collaborators focus more concretely than AHRQ staff does on what the IDSRN means for them in terms of award of specific task orders.

• The main “big picture” issues for the future raised by partners and collaborators are: (1) that the task orders are not very well funded for the goals or at times for the

xv

specific scope of work; (2) that the IDSRN lacks a clear identity, and the diffuse and varied nature of task orders contributes to this lack of clarity; (3) that the focus on individual task orders limits the ability to pursue a topic to its logical end in practice application that is “to scale,” and (4) that insufficient funds are in place to support important program infrastructure. Many, however, appeared hesitant to “rock the boat” since they had learned to work within the system, and major changes in the IDSRN would dilute the value of their investments.

Outcomes of Task Orders

• We classified IDSRN task orders by the way in which they aim to link research to practice. Most task orders seem to fall into two types.

- The first type builds on the IDSRN tie to operational delivery to undertake research that can only be done with that linkage, including operational data assessment and validation, clinical intervention and assessment, and research using IDS data. Projects of this type account for about one-half of the awards and spending.

- The second type aims in various ways to stretch the boundaries of traditional research so that the work is more directly and immediately focused on addressing an operational issue or need. These include systems analysis to identify targets for improvement, interaction, and tool development to help respond to bioterrorism threats and achieve other changes in health services. Slightly more than two-fifths of awards and spending is of this type.

• We reviewed available information on task order experience to assess their experience on various measures of moving research to practice in ways that build on those used by AHRQ. Of the 50 task orders that have been completed to date, 30 were judged to have had some effect on practice, and 20 did not. Operational changes were generally at the level of the individual delivery system, except in the case of tools, which typically had a broader audience. Of the 20 awards with no known direct outcome on delivery, five did result in a paper that had already been published or accepted for delivery in a peer-reviewed journal, and another was judged important enough for a follow-on task order.

ASSESSMENT AREA 1: INFRASTRUCTURE TO TRANSLATE RESEARCH TO PRACTICE

• From existing theory, we identified four dimensions of the IDSRN infrastructure that contribute to how well the IDSRN moves research to action: (1) clearly stated goals that support an effective strategy for operationally moving research to action; (2) partners and collaborators appropriate to those goals and strategies; (3) work appropriately scaled and focused to move research to practice in ways consistent with the overall goals and strategies; and (4) vehicles for building on IDSRN projects that make them known to appropriate audiences or take them to the next step. We find

xvi

that the IDSRN has substantial weaknesses in three of these four dimensions (appropriate partners and collaborators exist). Yet the IDSRN also has creative and unique elements that have promise and merit strengthening the IDSRN to build on this promise.

• Arguably the fundamental weakness of the IDSRN is also linked to its strength—flexibility to pursue cutting-edge applications in ways that are not present in other vehicles available to the agency. But flexibility also means that ambiguity exists about what it means to turn research into practice, and what appropriate operational goals for the IDSRN should be. The current structure provides a reasonable way of diversifying the field of health services research and linking it more closely to operational concerns of health care delivery, public health, and to researchers who are more active in those areas. However, the current structure of the IDSRN lacks the formalized processes and strategic focus that would be needed to actually generate important changes in health care delivery that take advantage of the body of existing research.

• Box 1 highlights the recommendations we make about steps AHRQ can take to strengthen the IDSRN infrastructure so that its goals are clearer and its infrastructure is better developed to support these goals. We also provide guidance on our recommendations for criteria AHRQ might consider in recompeting the IDSRN. If allowable by procurement rules, we also suggest that consideration be given to continuing to work with those existing partners and their collaborators who have performed well and also have adequate scale while adding new ones as needed to provide the requisite number and types of teams. The investment in developing working relationships is not trivial under the IDSRN for either teams or AHRQ and the strategy we suggest can help balance current investments with future needs.

ASSESSMENT AREA 2: ADMINISTRATIVE INFRASTRUCTURE AT AHRQ

• The way funding is committed to the IDSRN within AHRQ makes strategic planning difficult, because the funds available vary by year, and the amount available each year is not known until late in the fiscal year. While there are external constraints, it would appear that AHRQ has sufficient authority to generate improvements in this area and to establish clearer priorities for IDSRN funding from the top on an agency-wide basis.

• The master contract application process is a demanding one, though current partners believe, with hindsight, that the source of much of the burden (documenting data capabilities) is important. A key need, particularly for new applicants, will be to determine how best to demonstrate links with and commitment by the delivery entity in ways that take into account (1) the diversity of organizations involved in teams, and (2) informal as well as formal processes that bear on the strength of the underlying relationship.

• Teams view the speed of award with a high probability of success as a major strength of the IDSRN and as an offset to the administrative requirements imposed in the program. Improvements in providing better information on upcoming task orders as

xvii

well as requests that do not all come late in the fiscal year should be maintained and further developed.

• Better communication within AHRQ could exist about what the IDSRN is, what types of projects are appropriate for it, and how to effectively write and monitor task order contracts.

• IDSRN partners and collaborators commented more about the contracts office than other administrative components of the IDSRN. Opinions among teams vary widely, and most had developed ways of addressing unique system problems associated with working with the contracts office. However, concerns remain about whether sufficient flexibility is employed in responding to contract requirements in ways that support the needs of those in the IDSRN and the program goals as a whole.

• IDSRN participants feel they could have more input on the agenda for annual meetings, and that decisions about other support (e.g., newsletters and web-based tools) also could benefit from user input. Currently, insufficient distinctions are made about the diverse audiences for IDSRN findings and how best to reach each of them.

• While a preference for grants still permeates parts of AHRQ and some types of researchers in the field, the IDSRN, to our mind, well embodies the reasons that task order contracts were created. The stature of these task orders could be enhanced with user experience and more flexible interpretation of contract authority. The development of processes that provide outside input to the review of proposals and products could enhance the credibility of the awards and products with external audiences used to peer review processes. The main limitation of contracts is that some kinds of funds cannot be awarded through them. Creative “work arounds” may be needed to complement the general reliance on contracts in these circumstances.

• Specific recommendations are listed in the second box provided with this summary and their rationale is more fully discussed in the text.

ASSESSMENT AREA 3: CASE STUDY ANALYSIS

To highlight several “success stories” form the IDSRN, we developed case studies of four separate projects: Weill Medical College’s work on bioterrorism models, Lovelace Clinic Foundation’s cultural competency guides, work by RTI and Providence Health System on transferring medication data across patient settings, and work by RAND and the Center for Health Care Policy and Evaluation (CHCPE) on racial/ethnic differences in health care quality and outcomes.

These cases highlighted the importance of environmental factors in generating interest in the

work. The generalizability of the tools developed under these projects was another key to their success. Furthermore, each of the cases included follow-on work funded by the IDSRN, which was very important to making a tool more operational or taking an inquiry to the next level. Finally, because most of the cases involved “demand-driven” research, they had an operational focus and an audience interested in the project findings.

xviii

SUMMARY OF RECOMMENDATIONS

AREA #1: INFRASTRUCTURE TO TRANSLATE RESEARCH TO PRACTICE

Goals and Priorities and Related IDSRN Infrastructure:

1. Clarify in specific terms: (1) AHRQ’s vision of what translating research to action means for IDSRN goals; (2) what projects should have priority for the IDSRN, by topic and type; and (3) what kinds of work are outside the IDSRN scope.

2. To the extent the IDSRN is expected to generate actual change in diverse systems, establish a formalized vehicle to obtain strategic input from the user community on the kinds of questions that are most relevant to their types of organizations in the medium term (1-3 years) that they think could benefit from input of research findings. AHRQ can begin by convening a meeting of leading operationally based “thinkers” from systems affiliated with the IDSRN, adding perhaps a few similar outsiders already involved in advising AHRQ on related topics. The intent should be to establish a two-way dialogue that aims to identify the most promising targets of opportunity for the IDSRN.

3. Review overall planning in AHRQ to determine how best to provide continuing and formalized input on the status of research and on targets ripe for IDSRN work moving findings the next step forward to practice application.

4. Support more strategic and proactive efforts to draw on outside funding sources for the IDSRN; and identify which opportunities are most synergistic for AHRQ and the IDSRN’s needs.

5. Restructure the position of the IDSRN project officer to focus her responsibilities on IDSRN-wide needs, including input on emerging priorities, external linkages and funding opportunities, IDSRN communications with partners and collaborators, tracking IDSRN findings and dissemination needs and activities, and formalizing internal AHRQ processes and procedures. Provide a staff assistant for support and develop a stronger cadre of appropriate AHRQ-based task order monitors.

6. Consider using a small share of the IDSRN’s current funding allocation to provide IDSRN partners with flexibility to fund dissemination efforts both inside and outside their system.

Criteria for Partners and Collaborators in the Recompete

1. Continue the current focus on partnering with teams, including specified sets of delivery organizations that span the substantive priority areas for the IDSRN.

2. Because of the existing investment by both teams and AHRQ, consider how to give priority to existing teams that have performed well and are of “sufficient” scale to provide a critical mass while adding additional teams to strengthen the mix.

xix

3. Beware of applicants proposing newly formed teams with no history that include more than about three to five delivery systems. Because teams newly formed to compete under the IDSRN take more work to develop and maintain than those based on preexisting relationships, applicants will need to demonstrate that the commitments they make are realistic.

4. Give priority to applicants with a lead partner in delivery-based research entities when they have sufficient delivery scale and a strong research affiliate because their location builds most closely on established relationships linking research to practice. Lead partners based outside of delivery systems can provide a valuable alternative to the IDSRN, but such applicants should be required to demonstrate that their system contacts are sufficiently strong to provide high-level access to decision makers and sufficient internal skills to effectively partner. Outside partners that “add value” by their willingness to broker teams of multiple outside systems should have priority over single-system affiliations.

5. The RFP should take into account AHRQ’s likely willingness to tradeoff more projects for greater efforts at multi-site work to help bring action “to scale.” Clear guidance on this topic, as well as any other likely characteristics of work under the IDSRN, would help potential partners decide where to put their efforts and would help AHRQ assess the merits of competing applications.

6. AHRQ should resist the temptation to enhance their flexibility by contracting with many teams or teams encouraged to have many collaborators. Involvement of too many organizations relative to the likely funding level will make it hard for partners to provide sufficient work to offset the costs of participation.

7. Consider having applicants qualify to participate in the IDSRN in specific substantive areas tied to single or multiple portfolios of work with similar qualification requirements. If IDSRN requirements differ by portfolio area, this approach will provide each portfolio with access to a sufficient number of qualifying teams without adding substantially to administrative costs.

Task Order Focus and Strategy

1. Consider ways of revising the task order process to move beyond its focus on single task orders in order to better take into account linkages among task orders over time and the ability of the task order to produce findings that are relevant beyond a single site.

2. At a minimum, require requests for task orders to include a justification for how the work relates to the goal of translating research to practice as defined for the IDSRN. The justification should include the basis of: (1) the question and why it is or will be important operationally at some future point to a relevant segment of users; (2) what research shows now on that issue; (3) what the task order will contribute; and (4) specific outcomes expected from the task order and next steps that would be reasonable to pursue afterwards.

3. Pay more attention to the potential of the IDSRN to support multiple site work either within a single team or across teams.

xx

SUMMARY OF RECOMMENDATIONS

AREA #2: ADMINISTRATIVE INFRASTRUCTURE AT AHRQ

Overall IDSRN Placement, Funding, and Decision Making

1. Within the scope of external constraints, improve the predictability of funding levels for the IDSRN from year to year, and make information available to teams about minimum funding levels earlier in the fiscal year.

2. Clarify via AHRQ’s top leadership the role of the IDSRN and priorities for funding to make it easier for the IDSRN project officer to follow through on goals and navigate the challenges associated with brokering arrangements across the agency.

3. Tie IDSRN decisions closer to planning around portfolios to encourage more meaningful buy-in agency wide.

Recompetition of the Master Contract

1. Be as clear as possible about what the IDSRN is and how it will be used in the future, so that potential applicants can best judge their suitability and judge where to focus their efforts (and resources) in developing teaming arrangements and the proposal.

2. Maintain requirements to demonstrate data capabilities in the recompete, but consider whether any distinction is possible between entities whose capabilities have already been demonstrated and those who are new applicants or collaborators who have not yet demonstrated them.

3. Reflect the nature of the complex organizations represented within the IDSRN in how requirements are structured to assess whether the applicant has the support of the leadership of their associated delivery systems. Set requirements that allow responses to reflect unique features of each system as well as formal and informal communication processes. Requirements for partners based outside of delivery systems will need to be structured differently from those within systems.

Task Order Awards

1. Develop clearer guidance for AHRQ staff on the criteria that determine whether a topic is a “good” one for the IDSRN and the process to be followed in communicating ideas for potential topics.

2. Institute steps to space out decision making about IDSRN funds so that requests for task orders can be better planned and distributed over the year.

3. Convene an informal workshop to mentor AHRQ staff about how best to write a statement of work that leaves room for creativity while providing a clear statement of AHRQ expectations and of a scope of work that matches available funds.

xxi

4. Consider ways of establishing clearer criteria for evaluating task order applications and ways of incorporating outside input into the reviews—if this can be done without jeopardizing the timeliness of the awards.

5. Develop vehicles that give the IDSRN project officer or others in AHRQ more flexibility and resources to build on the experience with completed task orders where there are promising dissemination routes or next steps that would further the goals of the program.

Task Order Oversight

1. Do not award task orders in specific areas unless the units within the agency most closely affiliated with that work have demonstrated that: (1) they have a staff member appropriately skilled and invested for monitoring that task order and (2) they have the time to do so.

2. Develop a series of “brown bags” to enhance communication across task order monitors, both experienced and inexperienced, to share insights on substantive accomplishments of work in which they are involved and strategies they have developed for effective performance of the monitoring function.

3. Review how the contracts office makes decisions in specified areas with a view toward making the decisions more transparently predictable and consistent for IDSRN teams and toward making them responsive to team concerns. Priorities for initial attention include: (1) granting of no-cost extensions; (2) the ability to use unspent funds remaining after work has been completed to support dissemination; (3) the timeliness of feedback on the acceptability of deliverables; and (4) early communication about any concerns with performance under the task order.

4. Meet with IDSRN directors based in teams to identify ways of using flexibility available through the contracting process to better address their needs and make the process of interaction around contracts more collegial.

Communications Activities

1. Obtain more feedback from IDSRN partners and collaborators about shaping the agenda for the annual meeting so that it is most relevant to them and their needs. Consider restructuring the format of the annual meeting after recompetition to provide more opportunity for clustered discussions among participants with more common interests.

2. Obtain feedback from IDSRN partners and collaborators on the value of the content currently in the newsletter. With the growth of web-based technology, periodic emails, a web site with information and resource material may better serve team needs. Shifting the IDSRN communications focus to these modes could allow the newsletter to focus on ways of reaching out to other audiences, especially those with similar characteristics to those participating in the IDSRN who would be expected users of the findings.

xxii

Funding Mechanisms

1. Continue to have task order contracts (versus grants) form the basis of the IDSRN, since they seem highly suited to what the IDSRN seeks to achieve. However, there may be specific instances where a creative “work around” is needed to capture available support that cannot be committed using contracts.

1

I. OVERVIEW OF THE IDSRN AND EVALUATION OBJECTIVES

The Agency for Healthcare Research and Quality (AHRQ) has modified its mission over the years to give more emphasis to improving health care delivery and outcomes (Gray et al. 2003; AHRQ Profile 2003). In practical terms, this means involving the intended users of research in the development of the research from the start. AHRQ’s ability to advance this depends critically on forming partnerships both with the private sector and with more operational and delivery-based providers and delivery systems.

Unfortunately, AHRQ’s traditional relationships and funding mechanisms, developed in

response to an earlier, much different environment, do not necessarily provide the support the agency needs to achieve its current goals. Most funds are allocated through a time-consuming RO1 grant process that is not well suited to work with a quick-turnaround and may be difficult for applicants to navigate if they are not completely familiar either with the perspectives of those on the grant-review panels or with the awards process. In the RO1 process, it may be difficult to garner support for “cutting edge” work in health services research. Like the RO1 process, most of AHRQ’s funding vehicles have been established to advance the development of research that may be relevant to policy or practice but not necessarily to take the next step toward implementation. Finally, because existing funding processes also tend to involve mainly developers of research, users have little opportunity to provide feedback on the topics to be explored or on the relevance of different types of research.

To achieve its current mission, therefore, AHRQ has begun to test out methods of modifying

the way it provides funding and how it focuses work. For instance, it has developed more targeted grant solicitations based around cooperative agreements. It has also restructured its “user liaison” work by focusing on disseminating knowledge beyond government to an audience that includes delivery systems and other private sector entities with a stake in health services research. AHRQ also has developed new initiatives, like the Integrated Delivery System Research Network (IDSRN) and the Primary Care Practice-Based Research Networks (PBRNs) for the purpose of reaching out to users and helping to translate research to practice.

To discern whether the IDSRN and efforts like it are bringing AHRQ closer to its goals, the

agency must learn from these initiatives and strengthen them accordingly. Toward this end, AHRQ contracted with Mathematica Policy Research, Inc. (MPR) to evaluate the IDSRN. This draft report documents the results of the first two evaluation components (the third, still underway, will be incorporated as an appendix later). This chapter reviews the purpose and background of the IDSRN along with the objectives of the evaluation. Throughout the report, we provide more detail on the theory and methods behind the analysis (Chapter II), the IDSRN’s financing and funded task orders (Chapter III), the results achieved through task orders issued through FY 2003 (Chapter IV), how IDSRN partners and collaborating teams view the IDSRN (Chapter V), whether there are ways to strengthen the IDSRN so it can more effectively move research to practice, including implications relevant to the recompete for partners in 2005 (Chapter VI); and what specific steps, if any, AHRQ might take to better support the internal administration of the program (Chapter VII). Appendix A provides case studies of selected IDSRN task orders or bodies of work that provide insight on the process of translating research to practice.

2

A. PROGRAM DESCRIPTION

1. IDSRN Goals

Developed in 1999 (fiscal year 2000), the IDSRN was one of the earliest efforts to foster public-private collaboration in health services research and application (AHRQ 2001). The IDSRN’s explicit goal was:

To determine what works in terms of data and measurement systems, and identify organizational “best practices” related to care delivery and research diffusion. The IDSRN comprises a cadre of delivery-based researchers and sites to test ways to adapt and apply existing knowledge in real world settings (AHRQ 2001, p.18).

AHRQ staff view the IDSRN as a “signature” initiative that is highly visible within the agency because of its relevance to AHRQ’s interest in knowledge transfer and rapid dissemination (Program review 2003). The primary impetus for the program was the shift to managed care that changed the face of service delivery in the last decade. AHRQ (formerly AHCPR), then under the leadership of Cliff Gaus, was concerned about the need for data on care in the outpatient setting, particularly care provided through managed care organizations, which were perceived as an emerging dominant force in service delivery. The agency commissioned studies exploring the feasibility of creating and centralizing a database at AHRQ by purchasing or acquiring health plan data but found that the barriers were likely to be insurmountable. As an alternative, the agency developed the IDSRN program as a vehicle through which it could partner with integrated delivery systems to encourage health plans to use their data to improve practice (or to make their data available to others for research toward the same end). The other impetus for the program was the agency’s interest in moving away from more “academic” models of research toward a more or less applied approach that involved plans and providers in the process. The IDSRN represented a user-driven approach to research in which plans would do much more than supply the data to researchers. The IDSRN process was also intended to encourage more timely results and a more efficient system for awarding task order studies—largely through “prequalifying” partners who would compete on task order work under the IDSRN, linking providers based in delivery or managed care systems that had data, using the task order contracting mechanism. By the time the RFP for the program was being developed, Dr. John Eisenberg was director of AHRQ. While staff say he was a strong supporter of the program, he also left his mark on it by encouraging broader competition so that delivery systems as well as managed care organizations could compete and so that academics as well as research firms would be encouraged to affiliate with plans that did not have their own research capability. At the time, the PBRNs were also being developed by AHRQ, which was concentrating on making them complementary programs. Both are viewed as “living labs” and “real world settings”. But the PBRNs are grants and not contracts. They are focused on how small, clinical primary care systems can take action on such specific issues as disease management. Ideas come directly from the physicians, not from AHRQ. The practices did not have extensive data at the start of the

3

program and needed to build infrastructure. In the minds of AHRQ staff, the IDSRN’s key strength was its linkage with large managed care and delivery systems that had data that could be used in research and that could provide a setting in which the results of that research might be used. This contrasted with the PBRNs, which were focused on smaller practices of primary care physicians based in academic settings.

Those knowledgeable about the history of the IDSRN agree that the emphasis on “moving research to action” was not nearly as dominant earlier in the process as it is now, though they say it was there nevertheless. It is difficult to discern what was actually the case several years ago, as many significant individuals are no longer with the agency and even those who are still there may filter history through the lens of current concerns without realizing that their thinking has, in fact, evolved. However, the current agency focus appears consistent with AHRQ’s new leadership and the director’s belief that showing such relevance of health services research to practice is key to the agency’s survival.

2. IDSRN Program Timeline and Partners

The IDSRN program, located in AHRQ’s Center for Delivery, Organization and Markets (CDOM), is headed by Irene Fraser, who has had oversight of the program since its inception in March 2000 (see the timeline in Table I.1). The request for proposal (RFP) for the IDSRN program identified the following as being typical of the kinds of projects seen by the agency as candidates for a contract award:

• Performing needs assessments related to AHRQ’s goals

• Identifying research needs related to priority populations

• Assessing the feasibility of measuring and tracking quality at the national level using existing private-sector data

• Assessing the feasibility of linking integrated delivery system (IDS) databases with non-IDS survey data or other data sources

• Conducting timely research to answer questions such as what organizational structures improve patient safety, outcomes, quality, and access?

The IDSRN was to be built around partners, which were teams of collaborating organizations that included managed care organizations and the following other types of entities:

• Hospital-based IDSs (vertically integrated)

• Large, multi-specialty physician groups

• IDSs that include or are affiliated with safety net providers or providers caring for priority populations (low-income groups; minority groups; women; children; elderly; and individuals with special care needs, including individuals with disabilities and individuals with chronic care or end-of-life health care needs)

4

TABLE I.1

TIMELINE FOR THE IDSRN PROGRAM

Date Action Taken

March 2000 AHRQ issues RFP soliciting teams of partners/collaborators for the IDSRN

September 2000 AHRQ announces the award of contracts to 9 research consortia to establish the IDSRN

October 2000 AHRQ funds 10 research projects for the IDSRN for $4 million

December 2001 AHRQ funds 16 new projects for the IDSRN for $3.6 million

January 2002 First annual meeting of the IDSRN

December 2002 18 IDSRN project awards in 2002

February 2003 Second annual meeting of the IDSRN

June 2003 Internal changes at AHRQ – reorganization; new mission statement

August 2003 The 9 IDSRN contracts were renewed; 14 project awards in 2003

October 2003- March 2005

MPR evaluation of the IDSRN

2005 Year scheduled for IDSRN recompetition

• Universities, academic health centers, research consulting firms, or other research organizations working with the types of organizations listed above

In addition to traditional evaluation criteria like research capacity, experience, and adequacy of the management plan, the solicitation stressed that the collaboration, or partnership, should (1) have ready access to high-quality, IDS data integrated from a range of providers and (2) represent a diverse range of entities sufficient in size and experience to be relevant nationally.

The awards, announced in September 2000 after a competitive process that included an external review panel, went to nine consortia, each having a lead partner and multiple collaborators (see Table I.2 for list). The lead partners include both research organizations embedded directly within the IDS/managed care organization and independent contractors like Research Triangle Institute (RTI) and Abt Associates. Additional details on the characteristics of these entities is provided in Chapter V.

5

TABLE I.2

IDSRN PARTNERS AND COLLABORATORS AT TIME OF AWARD

Partners Collaborators

Abt Associates Inc. Geisinger Health System

Center for Health Care Policy and Evaluation (CHCPE), United HealthCare

Allina Health System RAND Health Systems Innovations

Denver Health University of Colorado Health Sciences Center Colorado Prevention Program

Emory Center on Health Outcomes and Quality Aetna Emory Center for Health and Aging

HMO Research Network administered through Kaiser Foundation Research Institute, Kaiser Permanente Northern California

Kaiser Permanente − Northwest; Southern California; Colorado; Georgia; Hawaii Group Health Cooperative of Puget Sound Harvard Pilgrim Health Care Health Partners Research Foundation Henry Ford Health System Lovelace Health System Fallon Health Care System

Marshfield Medical Research and Education Foundation Project HOPE Center for Health Affairs CODA, Inc.

RTI Intermountain Health Care UNC at Chapel Hill UNC Health Care CareOregon Summa Health System Providence Health System MayaTech Corporation University of Pittsburgh Medical Center

University of Minnesota, Division of Health Services Research and Policy

Institute for Clinical Systems Improvement Medical Group Management Association Allina Health System Blue Cross Blue Shield of Minnesota Institute for Research and Education HealthSystem Minnesota Delta Rural Health Consulting

Weill Medical College of Cornell University New York Presbyterian Healthcare System Joseph L. Mailman School of Public Health of Columbia University North Shore University Hospital Memorial Sloan-Kettering Cancer Center Health Watch

6

The original three-year awards were extended for two more years in fiscal year 2003. While all partnerships were renewed, AHRQ set conditions on the renewal of one Marshfield Medical and that contract is now inactive pending resolution of discussions on these conditions.

B. EVALUATION OBJECTIVES AND CONSTRAINTS

The IDSRN program has been operating for over four years, and AHRQ will soon be making decisions regarding its renewal and any changes that would be desirable if the program is continued. The purpose of the MPR evaluation is to support AHRQ both in this effort and in more general oversight of the IDSRN by addressing questions in three areas:

• Assessment Area 1: To what extent is the program’s infrastructure designed to encourage “real world” adoption of research findings? As AHRQ looks at program renewal, what types of changes in the infrastructure might be considered to clarify IDSRN goals and to promote “research to action” as an explicit goal?

• Assessment Area 2: What are the strengths and weaknesses in the way the program is administered? Are there ways in which administration could be improved? To what extent has the IDSRN’s use of contract awards supported or impeded the IDSRN’s goals?

• Assessment Area 3: Which major accomplishments achieved under task orders seem most successful? What can we learn from these selected task orders about successful examples of translating research to practice and the characteristics that advance or constrain the adoption and dissemination of changes in practice based on research?

This report includes findings in the first two areas and provides an overall assessment of outcomes across the task orders. Appendix A presents findings for Area 3 from case studies of a small but diverse set of IDSRN task orders that provide examples of work that is consistent with the IDSRN goals.

The evaluation is constrained by four factors. First, the IDSRN has been in place for only a limited time, which means that many of the

partnerships have only recently completed many of the task orders awarded them, and there has been little time to disseminate findings or, if disseminated, for the findings to have been used by others.

Second, the evolving nature of AHRQ’s priorities and goals for the IDSRN means that some

elements that are now of substantial interest (like the design of a formal infrastructure for dissemination and implementation) were not so initially.

Third, AHRQ itself is evolving in its approach to work with, for example, a major strategic

planning process built around substantive “portfolios” of interest areas. While this evaluation tries to reflect emerging processes in our recommendations, much remains unknown about how they will evolve and hence recommendations about the IDSRN will ultimately need to be reviewed in context of changes in AHRQ overall.

7

And fourth, the funding available for this evaluation is relatively limited ($200,000), particularly in light of the complexity of the program. As a result (and as described in the next chapter), our assessment is based solely on what we can learn from documents and from interviews with AHRQ staff and IDSRN partner/collaborator teams. These complex teams involve many people who are working on a variety of task orders. So while we tried to interview as many individuals as possible, we were constrained in that we were able to devote only about one day per IDSRN team.

9

II. DESIGN AND METHODS

This chapter reviews the theoretical considerations and the logic model underlying the evaluation as well as the overall analytic approach and data sources. We focus specifically on the methods that are relevant to assessment areas 1 and 2, leaving a description of case study methods (relevant to the assessment area 3) to Appendix A.

A. THEORETICAL FRAMEWORK: LOGIC MODEL FOR THE IDSRN

There is a sound multidisciplinary theoretical foundation for understanding not only how research is converted to practice (what we term here “adoption”) but also how these practices are disseminated and diffused. A number of recent, excellent syntheses of work in this area have guided our approach to evaluating the IDSRN [see Berwick (2003) and Fitzgerald (2002) for syntheses specific to innovation and diffusion in health care; see Wejnert (2002), Hage (1999), and Strang and Soule (1998) for syntheses capturing in a more general way the literature in sociology, organizational theory, and other social sciences]. In health services research and policy, Canadian researchers have played a prominent role in developing models of how to use these theories in translating research to practice (see, for example, Lomas et al. 2003; Lavis et al. 2002, 2003). Figure II.1 shows the basic logic model we developed for understanding the process by which innovations (of any type, including insights from research findings) get adopted and diffused throughout a larger body of organizations.1 The model is framed to describe adoption of findings from individual task orders—the basic unit of analysis we used in examining IDSRN outcomes. According to the model, program outcomes depend on several factors: whether and how well the task orders are conducted, whether they lead to actionable results that are adopted by the host delivery system or other immediate relevant players, and whether those results and/or the host delivery system’s experience with adoption are disseminated more broadly to its target audience, thus completing the process via diffusion and broader implementation. In evaluating process, it is not only what actions were taken but also why they were taken that are important considerations. We used the logic model to identify specific processes and questions that are critical to the evaluation, including protocol design.

1 After the project began, AHRQ was a cosponsor with others of a conference that focused on Accelerating Quality Improvement in Health Care Strategies to Speed the Diffusion of Evidence Based Innovations in Washington DC (January 27-28, 2001) convened by the National Institute for Health Care Management Foundation and National Committee for Quality Health Care. A paper at that meeting discussed “Complexity and the Adoption of Innovation in Health Care (Plsek 2003). A key point of the paper was that problems like quality are complex, not just complicated, and unlikely to be resolved with formulaic and expert knowledge approaches. Taking a systems approach, the paper examined the characteristics of organizations that innovate and the processes that lead to such innovations. The paper stressed the importance of decentralized innovations that are dynamic and establishing habits for change that are supported by new language and tools. The variables discussed share many similarities with those included in the model we developed as important to change, but complexity theory supports a much less linear and centralized process of change. Neither our model nor theirs addresses the relationship between research and innovation, particularly in terms of translating broad bodies of work into practice.

10

FIGURE II.1

A STYLIZED LINEAR MODEL OF ACTION (Task Order Level)

Project awarded

Innovation Research effectively and fully implemented

Generates results and findings—potential direct outcomes

Adoption Findings are actionable and add to existing knowledge Feedback

Findings are communicated to appropriate audience

Audience finds action attractive

Sufficient support/resources to implement exists

Implementation proceeds—adoption Improvements Results of implementation are positive and important—impact in delivery

Results are disseminated to other potential users—diffusion

Theory highlights the many factors that contribute to initial adoption, diffusion of practices

in more settings, and overall change. First, there is the innovation or the finding itself; the question it addresses; and the focus, products, and results of the task order. Task orders that result in innovations or findings that are new and “actionable” in a broad range of settings are the most likely candidates for adoption.

However, adoption will not occur unless the findings are made known to the appropriate

audience, which may comprise the local organization, the local community (similar organizations), affiliated organizations (other entities owned by the parent organization or collaborators), or others (e.g., national audience, similar providers nationwide). It will be important to determine the extent to which the audience was identified and the characteristics considered when individual task orders were devised, and if not, whether, how, and when it was indeed specified.

The audience can be reached in different ways. For example, there are broadcast (external)

and contagion (internal) models of diffusion (Strang and Soul 1998), either of which may use

11

informal channels such as social networks and ties or formal channels such as briefings and other organizational processes. In addition, third parties such as the media and information brokers can enhance awareness of the findings, though information communicated through social and other networks may carry more weight (Wejnert 2002). Some people and organizations are better connected than others—both horizontally and vertically—and thus are more likely to hear of innovations.

The literature on marketing, only recently applied to medical care (other than health

promotion/education), stresses the importance of shaping communication so that it addresses the audience’s values, its concerns, and the competing ideas for change (see, for example, McDermott 2000; Hornik 2002). On a more mundane level, the literature reinforces the importance of using the communication channels also used by the audience and therefore considered trustworthy. These concepts reinforce the importance of defining the audience and examining the strategies used to reach it, especially since it may be a diverse group.

However, while the dissemination of knowledge may be essential to change, it is by no means the only ingredient. Potential adopters must find the innovation attractive. They must also have enough organizational and political support to advance its adoption and enough resources and other support to implement it. Theory highlights a number of not necessarily independent factors that can expedite or impede the process of change.

First, as Berwick (2003) emphasizes, there is the perception of the change—its benefits,

risks, and uncertainty; its compatibility with the values, beliefs, history, and needs of potential adopters; and its overall complexity and ability to be simplified, or “reinvented,” for local adoption. In evaluating the IDSRN, it will be important to assess the extent to which communication addresses these audience concerns.

The second factor that can affect the process of change has to do with the principle that

“good evidence” does not necessarily translate directly into a desire to adopt. Though a “push” model of dissemination both assumes that good ideas will be implemented on merit and emphasizes the communication of good science, the “pull” model is driven by the needs of potential adopters and how they view both the innovation’s source and its ability to meet their needs (Fitzgerald et al. 2002). The “push” model and its more complex variant, the “communication and feedback” model, is driven by the notion that the ability to build consensus on evidence-based science is limited both when there is uncertainty about how to interpret the findings and when standards for judging evidence vary across disciplines. We will want to be mindful of how the findings and the way they are communicated relate to these criteria.

Third, change will be more likely to occur when the source of innovation is both locally

respected and congruent with local needs. While some individuals (and organizations) tend to be innovators and early adopters, most are risk averse to one degree or another, preferring to see the experience of early adopters, especially when they are close by or closely affiliated, or to wait until an innovation becomes the status quo (Berwick 2003). Spatial proximity between the innovation’s source and its final “destination” expedites communication. But other sources could decide to copy an innovation or to compete with it by putting a different strategy forward. These concepts can help in identifying and interpreting diffusion efforts.

12

Fourth, changes like those likely under the IDSRN typically take place in a context that is specific to an organization or even to a consortium of organizations. This context includes both the culture of the organization and the environment in which that organization operates (Fitzgerald et al. 2002). The variety of theories on organizational change highlight the importance of organizational structures developed specifically to respond to environmental uncertainty (structural contingency), dominant coalitions developed to respond to major environmental changes (political theory), and control of resources (resource dependency theory) (Hage 1999). How and the extent to which change will occur will vary and organizations’ willingness to change may be associated with certain features of its structure. For example, these include: whether an organization uses an organic structure involving decentralization, horizontal communication, and shifting leadership, each of which promote innovation; whether an organization focuses on pro-change values or high-risk strategies; and whether an organization has a complex division of labor that reduces bureaucracy and shifts an organization’s attention from the achievement of efficiency to the achievement of higher morale (Hage 1999). Most theories are based on organizational settings and interorganizational relationships such as joint ventures and strategic alliances, which have been studied only to a limited extent (Hage 1999; Sofaer and Myrtle 1991). Interorganizational settings and relationships are particularly relevant to the IDSRN program.

Fifth, adoption is not a simple accept/reject decision (Fitzgerald et al. 2002). It is a

multidimensional process that brings into play knowledge, persuasion, decision, implementation, and confirmation (Rogers 1995 as cited in Fitzgerald et al. 2002), though not necessarily in a linear way. The complex structure of most organizations means that adoption typically involves a protracted negotiation process between individuals and groups that is characterized by shifting alliances (Fitzgerald et al. 2002). Various types of opinion leaders will likely be important at different times in the process, including “node or focal points” who translate information from academia to practice and back; “experts” with local credibility who can move opinion; and “political leaders” with the management and political skills that can effect change. Depending on the context, each of these roles may be relevant.

The sixth and last factor affecting the process of change is that, despite the similarity of the

processes that underpin diffusion in different settings, there are some differences. While most IDSRN settings involve health care, the various clinical and administrative units operate under different professional norms and values. As a result, the decision makers in different units will weigh options differently. To the extent that policymakers are the focus of an innovation, the adoption process will be shaped by both the legislative process and administrative agency constraints on decision making (Kingdon 1995). Research, for example, may be more valuable at the stage when issues surface than, for example, when competing alternatives are being reconciled. A study that tracked how research is used in the policy process highlights the diversity of research findings and ways of measuring and defining use so clarifying concepts is important in assessing how research is used here (Lavis et al. 2002). Hence, if the locus of innovations varies, analysis needs to account for different types of applications.

All six sets of factors—alone and in combination—will influence implementation of the innovation. Given this complexity, it is important to clearly define both the innovation content and its audience. Moreover, in the interest of producing consistent results, it is also important in the analysis to focus on overall rates of adoption rather than on any one event (Hage 1999).

13

The extent of emphasis on various aspects of the logic model varies with the research questions. For many questions, the top panel of the logic model is most relevant, as it relates to understanding what work was conducted under the IDSRN and whether that work has been focused in ways that yield actionable results. But we also want to understand generally from delivery-based settings not only what makes results “actionable” and relevant to the setting but also the mechanisms for promoting adoption and diffusion.

Finally, the evaluation applies the same concepts relevant to the adoption of task order

products to assess the “fit” between the outcomes the IDSRN seeks and the overall structure of the program. For example, at the IDSRN program level, we seek to explore how the program structure and topics selected for work in the task orders account for both the interests of potential users and the mechanisms for reaching those users. Within the partner/collaborator entity, we will look at how those doing IDSRN work are linked to the organizations in which they function and the extent to which the program and the task orders issued under the partnership are organized to (1) reflect operational needs, (2) communicate results, and (3) facilitate adoption. We will also look at the dissemination vehicles available to partnerships and the larger organizations in which they are embedded.

Unfortunately, while the logic model proved useful in identifying questions to ask, its value

in explaining IDSRN outcomes was far more limited. The reason (as described more later on in the report) is that the IDSRN’s focus, to date, has mainly been at the “top box” on Figure II.1. That is, the IDSRN has awarded task orders and they have been completed, but the structure and resources available to take them the next step toward broad adoption have been relatively limited and few, if any, findings have as yet been implemented particularly outside the host institution doing the original work.

B. ANALYTIC APPROACH

The evaluation has two basic units of analysis: individual task orders and the IDSRN as a whole. We took this approach because, in reviewing the IDSRN’s funding and accomplishments, we found that task orders are the mechanism through which funds and other resources are allocated. On the other hand, the need to address evaluation questions that span the totality of the IDSRN experience suggests that the unit of analysis is sometimes the IDSRN as a whole and its partners/collaborators. We review here the research questions in the first two assessment areas and the data sources we will use to answer them.

1. Assessment Area 1: Moving Research to Action

The first set of research questions relates to providing formative feedback to AHRQ about the ability of the IDSRN to achieve what now appears to be its primary goal—moving adoption of research findings to practice by locating research in user organizations. AHRQ wants to know whether the IDSRN is structured to support this goal and, in considering program renewal, what changes the agency might make in the infrastructure to better support this goal. We have framed this question as follows: To what extent has the IDSRN’s infrastructure been developed to encourage “real world” adoption of research findings? In considering program renewal, what types of changes in IDSRN infrastructure might be considered to both clarify IDSRN objectives and promote the overall goal of converting research into action?

14

Addressing this broad question involves answering a number of other questions:

1. What does each major participant in the IDSRN hope to achieve through collaboration? Has this changed over time and if so, how?

2. What partner/collaborator teams formed, and which were selected? Have teams changed over time and why? How closely are the teams tied to their respective delivery organizations?

3. To what extent is there a program-wide identity and infrastructure?

4. What work has the IDSRN supported, and with what funding has it done so? How has this changed over time, why, and to what end?

5. How are funding priorities and task order topics determined?

6. What infrastructure exists at AHRQ to support adoption and dissemination of products from the task order among partners and collaborators, and in their associated delivery organizations and elsewhere?

7. What are the task order products, what type of real world applications are relevant to them, and to what extent have efforts been made to encourage adoption?

8. What do IDSRN participants view as the most critical strategic issues for future program attention? What are viewed as the program’s key strengths and weaknesses? Are there particular changes that should have priority? Is there consensus within AHRQ and among its partner teams on a focus on moving research to action? What constraints exist in moving task work toward this goal?

In addressing these questions, we relied largely on interviews with partner/collaborator staff but we also used information obtained as part of the now-completed program review and in interviews with AHRQ leadership and key program staff. In addressing questions 4 and 7, we relied primarily on document review to develop analyses built up from information on individual task orders.

2. Assessment Area 2: IDSRN Administration

The second set of research questions is related to the IDSRN administrative infrastructure. Often overlooked, this infrastructure plays a critical role in determining how well initiatives work. Given AHRQ’s limited experience with initiatives seeking real world applications of work and task order contracts versus grant-making as a funding mechanism, an assessment of the strengths and weaknesses of the administrative structure will provide AHRQ with important feedback on how the program can be improved. We have framed this issue as follows:

What are the strengths and weaknesses in the way the IDSRN is administered? Are there ways in which administration could be improved? To what extent has the IDSRN’s focus on contract awards supported or impeded program goals?

15

Addressing this broad question involves answering the following other questions on AHRQ processes for awarding and monitoring contracts, and on three program-wide administrative issues:

1. How are task order contracts awarded?

2. What role does AHRQ play in monitoring the task orders after award, including the respective contributions of the task order monitor, the IDSRN project officer, and the contracts office?

3. To what extent are the IDSRN funding and priorities opportunistic as opposed to strategic? What are the strengths and weaknesses of each approach, given the setting in which AHRQ operates?

4. What have been the administrative, political, and substantive implications of the IDSRN’s focus on contracts versus grants?

5. Are there ways in which AHRQ’s internal monitoring systems could be strengthened to support the development of information on efforts to adopt and/or disseminate task order products under the program, particularly after the initial contracts end?

This part of the assessment is based on information received from AHRQ staff and, where relevant, from partners and collaborators on program administration. Our analysis points out the differences and similarities between AHRQ’s views and those of the partners.

C. DATA SOURCES, COLLECTION, AND ANALYSIS METHODS

We used the four data sources described below in the evaluation of the IDSRN. Interviews and Core Document Review. At the start of the evaluation, we reviewed core

program documents and interviewed key AHRQ staff to describe how the program works. Documents on Individual Task Orders. We developed a database on the individual task

orders funded through IDSRN from its inception through the end of fiscal year 2003.2 For each task order, we reviewed the scope of work and, in most cases, the original proposal and the final report (or, in projects still underway, the most recent report submitted by the partner/collaborator team). We also prepared one- to two-page synopses of each task order indicating the title and the organization responsible for conducting the work (including involvement of collaborators), the topic explored, a summary of the work, the key findings/products, and any information on the adoption of findings within the delivery setting and on further dissemination/adoption efforts. We also used these synopses and the core program documents to develop a spreadsheet of basic data on each task order. Using this database, we constructed tables that profiled not only the characteristics of work funded over time but also the amount and sources of fiscal support and

2 In FY 2005, AHRQ expects to have available $6.3 million or more to fund task order work. Seven of these

were already awarded and the rest are under active competition, with awards expected by the end of the fiscal year.

16

how they changed over time. We also used the database as a starting point for interviews with AHRQ task order monitors and partner/collaborator staff, and for determining which case studies to conduct. Appendix B includes selected tabular data on task order characteristics that complement the information in the task order synopses.

Interviews with AHRQ Staff. We interviewed 26 AHRQ staff in the course of the evaluation (see Table II.1 for a list of topics). These staff included AHRQ program leaders, including core members of the Executive Management Meeting (EMM); individual task order monitors—about their role and their view of the success of each task order; and contract staff—about administrative processes. Most of the interviews took place in person at the AHRQ offices in February-March 2004;3 task order monitors working on related topics (e.g., bioterrorism) were sometimes interviewed together to conserve time. Two people from our evaluation team participated in each interview, a senior person who led it and a more junior person who took notes. The interviews were summarized within a few days of each interview and finalized after review by the evaluation team.

Confidentiality was a concern in the interviews. We told those who asked that we would not go out of our way to associate specific comments with specific individuals but that, because of the small number of individuals being interviewed, we could not guarantee that the source of information on some topics might not be obvious.

Interviews with Partners/Collaborators. To the extent feasible, we conducted one-day

site visits to each partner organization to collect data on the following: how the program is viewed by its outside participants, the rationale for forming a particular team and participating in the IDSRN, experience with particular task orders, experience translating results to practice, and other important issues. We aimed to spend about half the day talking with partner organization staff and the other half split between discussions with senior management at the partner organization and a group meeting with partners/collaborators, many of whom participated through a conference call.4 The structure of interviews varied from one team to the next, reflecting both differences in how responsibilities are divided among the team and challenges involved in scheduling group meetings with many busy individuals. Of the nine partners, seven were visited on site.5 Interviews with Marshfield and Abt/Geisinger were conducted entirely by telephone for logistical reasons. We followed up some of the on-site interviews with telephone calls to get input from selected collaborators or others not available at the time of the visit.

3 Interviews conducted at the start of the project as part of the program review covered some of the same topics

and were also used to support analysis. Some individuals interviewed initially were also re-interviewed to provide the opportunity to cover all the issues relevant to the project.

4 Specific partner interviews varied according to the team arrangement. For example, given that neither RTI not Abt is a delivery organization, we spoke instead with collaborator staff to get the insight into the delivery system.

5 For the RTI team, we conducted the interviews at the Portland collaborator site in order to talk with delivery system leaders.

17

TABLE II.1

TOPICS FOR INTERVIEWS WITH AHRQ STAFF

Interviewee Topics of Interest

AHRQ leadership Role of respondent in the process Perspectives on how well it is promoting agency mission Broader departmental context of relevance (e.g., secretarial priorities, changes in contract administration, constraints) Nature and limits on agency authority that affect the IDSRN process Perspectives on the range and mix of topics, the timeliness of results and outcomes, dissemination and factors influencing these decisions Key strengths, concerns, recommendations for improvements

Program officer and task order monitors

Respective roles and responsibilities How award topics are decided on and the budget distributed How competition is structured and awards decided on How partners/awards are monitored and assisted What is working well, what is a problem, and why How they think others perceive the process Key strengths, concerns, recommendations for improvements

Contract staff; internal AHRQ administration

How the award process works and timeframe Contract boilerplate and rationale Award history and any associated protests, problems Relationship with program staff and how well that works How they think others perceive the process Key strengths, concerns, recommendations for improvements

Each interview was conducted by a senior member of the MPR team. All telephone

interviews and all but two of the site visits also involved a second individual who was responsible for taking notes. As in the AHRQ interviews, either that person or the interviewer (as applicable) summarized the notes from each visit or interview within a few days of the visit or interview. In total, 65 individuals were interviewed for an average of seven per partner team, though the number varied substantially across teams depending on their organizational complexity. Table II.2 shows the topics covered for each set of interviews.

18

TABLE II.2

TOPICS FOR INTERVIEWS WITH PARTNERS/COLLABORATORS

Interviewee Topics of Interest

Lead partner staff Nature of immediate partner entity and reasons for interest in IDSRN; where partner fits in parent organization

Rationale for selecting collaborators; how partners/team work together

History of IDSRN involvement: generally, annual meetings, task orders

Experience with specific task orders where partner was lead; perspectives on others through collaborators; how decisions to bid are made within the team

Involvement in topic selection and consistency between organizational needs and interests and those topics

Way IDSRN work/findings are communicated within the parent organization; experience on tasks awarded to date; history of organizational outcomes and adoption

Who is viewed as the appropriate audience for dissemination/diffusion

Vehicles for broader communication of IDSRN findings and results to this audience, experience to date, and perceptions about both these goals and the process

Indepth review of 1-3 task orders (this may occur either with partner or collaborator interview as appropriate)

Perspectives on the grant award and monitoring process, AHRQ staff’s contribution to the IDSRN objectives

Delivery or operational setting (CEO, medical director or the like)

Relationship with partner unit/entity

Knowledge of and involvement in IDSRN activity

Perspectives on the value of the specific task order work to date and IDSRN generally to the organization

Relevant questions to task orders chosen for in depth review

Reasons for diffusion/lack of diffusion of results; reasons for any limitations in implementing specific projects (to the extent these involve operational issues relevant to them)

Personal perspectives on how best to diffuse products developed under the IDSRN; any constraints that are important to convey

Partner and lead collaborator contact from each affiliate

Nature of each organization and how they came to the partnership/collaboration

Process of internal team communication and decisions on activities; views of AHRQ award and monitoring process

Targeted (brief) review of history of specific awards to collaborating organizations, their findings, and dissemination

Key collaborating delivery systems: important elements of delivery setting context that affect performance, vehicles for dissemination

19

III. FINDINGS ON IDSRN FINANCING AND TASK ORDERS

Findings from the analysis of IDSRN financing and task orders cover the first four years of program operations (FY 2000–FY 2003 awards). The analysis was purely descriptive, and the findings are based largely on a review of documents and a statistical analysis of task orders awarded and their progress. However, we do take advantage of selected insights into the task orders developed in the interviews with AHRQ or IDSRN partners and collaborators.

A. IDSRN FINANCING AND SOURCES OF SUPPORT

Fifty-eight task orders totaling $14.2 million were awarded through the IDSRN from its inception through FY 2003 (Table III.1). Less than half of this amount ($6.4 million) came from core AHRQ funds. The rest came from three sources: funds authorized by Congress for AHRQ to spend on addressing patient safety ($4.0 million), funds from DHHS earmarked for addressing bioterrorism ($3.0 million), and, to a much more limited extent, funds from other sources. The availability of these unanticipated sources of support has expanded the IDSRN and influenced how it is perceived, an issue discussed later in more detail.

TABLE III.1

IDSRN TASK ORDER AWARDS AND FUNDING SOURCES, FY 2000-FY 2003

Funding Source

Task Order

Awardsa

Total Awarded (in 1000s)

Average $ Per Award

Awards Made in FY 2000

(in 1000s)


(in 1000s)


(in 1000s)


(in 1000s)

All sources 58 $14,198 $244, 791 $2,415 $3,507 $5,327 $2,949

AHRQ core funds

30 6,419 213,968 2,140 1,103 949 2,227

Patient safety funds 16 3,990 249,376 — 1,929 1,467 594

Bioterrorism funds 9 2,951 327,723 250 — 2,701 —

Otherb 6 1,050 139,583 25 475 210 128

Source: MPR Analysis based on data provided by AHRQ staff a58 awards were made though numbers add to 61; some task orders had multiple sources of support bIncludes $0.5 million paid for through long-term care funds available to AHRQ, $0.25 million from CMS, $0.23 million from the Office of Minority Health and $50,000 and $25,000 respectively from Health Services Advisory Group (HSAG) and ASPE. The Long-Term Care funds supported two task orders and the others one each.

20

IDSRN funding levels have varied over the years. Task order awards totaled $2.4 million in the first year of the program (FY 2000), rising to $3.5 million in FY 2002 and to $5.3 million in FY 2002 before declining to $2.9 million in FY 2003. (In FY 2004, AHRQ expects funding to rise again, possibly to $6 million or more.6) Only part of this variation is a result of the fact that the availability of funding from specialized sources has varied over time.7 Though a little more than $1 million per year has typically been dedicated to the IDSRN from the start, total funding has varied with AHRQ’s annual budget and how much of that is spent on grant awards in any fiscal year. Thus, while about $2 million in awards were made from AHRQ’s core funds in FY 2000 and FY 2003, only about half of that was awarded in the intermediate two years.

Our interviews with AHRQ staff suggest that the reasons for such variation are complex,

reflecting factors within and beyond AHRQ’s control.8 For instance, Congress often does not finalize appropriations until late in the fiscal year, so while AHRQ may be reasonably confident about receiving at least a certain amount of funds, it has been hesitant to spend until decisions are final. Beyond this uncertainty, AHRQ typically reports to Congress on the amount of its available research funds that will be paid out in grants rather than contracts because this issue is important to many of AHRQ’s core constituencies. It could be close to the end of the fiscal year before AHRQ knows how much it will spend in that year for grants and what that expenditure means for its ability to either award contracts and or use its full budget appropriation. As a result, the IDSRN (which has a more rapid award cycle because of contract task orders) may get approval to spend additional funds toward the end of the fiscal year. Conversely, if the agency feels it is “squeezed” given, for example, congressional uncertainty or the potential for competing demands on funds, it can delay decisions on IDSRN task order awards until closer to the end of the fiscal year. The ultimate impact is to delay the awards themselves until late in the fourth quarter, an issue viewed as undesirable by both IDSRN staff and partner/collaborator teams (see Chapter VII for more detail).

Readers should note that the allocation of funds within AHRQ is very important to diverse

external constituencies particularly in the research community. AHRQ is the major source of funding for investigator-initiated grants in many areas relevant to health services research. Such grants are particularly valued by academically based researchers. Within those institutions, leadership is very familiar with the much larger National Institute of Health (NIH) grant

6 The IDSRN June 2004 newsletter indicates that about $2 million has already been awarded for emergency

preparedness (5 task orders), and about $350,000 in the area of organized systems and Health Insurance Portability and Accountability Act (HIPAA) work (2 task orders). Solicitations in 11 other areas were yet to be awarded, with 1 cancelled because no proposals were received.

7 Most of the bioterrorism funds were awarded in FY 2002, for example, and patient safety funds first became available in FY 2001 with the heaviest funding in that year and the next.

8 The issue of appropriations affects awards in many ways. Program staff, for example, may defer using core funds until they determine which projects can be funded through specially targeted funds. Some funds come through HHS and while they may be targeted, it may take some time to learn how much will be spent through AHRQ and by the IDSRN. External and largely academic-based researchers that provide AHRQ with political support tend to favor grants and AHRQ has responded by providing Congress with estimates of the share of funds to go to grants. Task order contract awards under the IDSRN program may be delayed until the agency is more certain about whether it has met this grant commitment. Alternatively, the agency may find itself with year-end money that needs to be spent.

21

processes. Assessment of research productivity and quality give great weight to the ability to secure investigator-initiated grants via processes like those developed through NIH’s RO1 awards.

B. AWARDS PROFILE

1. Size

For the most part, the contracts awarded through the IDSRN are medium-sized, or less than $300,000. None has covered a period of performance longer than two years; the shortest has been seven or eight months. Awards to examine bioterrorism tend to be somewhat larger than average (see Table III.1). Awards paid for through core funds tend to be smaller than those paid through other finds, but this partly reflects some small contracts, especially in FY 2003, awarded to help disseminate information that was previously developed under other task orders. For example, in FY 2003, 12 awards paid for by core AHRQ funds averaged about $187,000. In each of the other years, the average award was $220,000 or more. Though task orders are for distinct scopes of work, some (particularly in the most recent years) involve a follow-on to previous activity, a topic considered in more detail in later chapters of this report.

2. Topics

The task orders, which cover a diverse array of topics, have been categorized into 12 priority areas by AHRQ, though the distinction between some areas is not entirely clear, at least in our minds. Appendix B provides a summary of each task order, and Table III.2 shows the 12 priority areas, though the amounts and areas of interest have varied over the years.

The topics addressed in the task orders have been determined largely by the funding source.

As mentioned, work in information technology, patient safety, and bioterrorism has been funded exclusively though the additional funds earmarked for these issues. Core AHRQ funds supported work in each of the nine other areas, and some areas also received other funding support.

In its first year (FY 2000), the IDSRN was funded almost exclusively by core funds. Priority

areas in that year—data development, the structure of health care delivery, and minority health—appear to reflect the origins of the IDSRN and the priorities of the then-current DHHS leadership. We were told that the speed with which task orders were executed that year limited the care with which topics or proposals could be reviewed. In some cases, awards were made opportunistically, i.e., taking advantage of the IDSRN vehicle and the capabilities of particular contractors associated with each partner.

After the first year, the availability of other sources of support is also reflected in awards,

particularly to explore patient safety and bioterrorism. Topics addressed in later years have also has been influenced by AHRQ staff interests and greater awareness by staff of the IDSRN capabilities. For example, AHRQ staff focused on long-term care were instrumental in drawing on entities affiliated with selected IDSRN partners to develop projects relevant to long-term care.

22

TABLE III.2

IDSRN TASK ORDER AWARDS BY PRIORITY AREA, FY 2000-FY 2003

Priority Area Total dollars

($1000s) Total Awards:

All (Core Funds)

FY 2000

FY 2001

FY 2002

FY 2003

Information technology in IDS

$ 613

3 (0)

--

3

--

--

Organization/ coordination of care

$1,599

6 (6)

--

--

4

2

Patient safety $2,994 11 (0) -- 4 5 2

Bioterrorism $2,701 8 (0) -- -- 8 --

Knowledge transfer/use

$ 346

5 (5)

--

--

--

5

Data and measurement

$1,020

3 (3)

3

--

--

--

Health care for minorities, women/children

$1,427

7 (4)

2

5

--

--

Structure of health care delivery

$ 999

5 (4)

5

--

--

--

Long-term care $ 798 3 (2) -- -- 1 2

Quality improvement

$ 770

3 (3)

--

--

--

3

Working conditions and quality

$ 498

2 (2)

--

2

--

--

Financial incentives and cost

$ 432

2 (1)

--

2

--

--

Source: MPR Analysis based on information provided by AHRQ; priority areas are defined by AHRQ as are the

classifications of individual task orders

3. Portfolio Areas

AHRQ has recently moved to a matrix form of work planning. Though centers remain the basis for the formal organization, AHRQ has developed a list of 10 topical portfolios that they believe spans the areas of work important to AHRQ regardless of where it is carried out in the organization. The intent is to use this as a vehicle for decisions on priorities for funding across the agency. Staff from diverse entities across AHRQ have been assigned to work on planning for each topical area. Teams are led by senior staff with extensive involvement in work related to that area. Each team is charged, as we understand it, with identifying what is known through research in that area and where priorities for future research lie.

23

To understand how the work of the IDSRN relates to this evolving strategic planning, we classified each task order into one of the 10 portfolios that, to our knowledge, have been defined by the agency (see Table III.3). Our classification may not precisely match how AHRQ views particular task orders in terms of the portfolios but it provides a good general indication of where IDSRN work fits within the general portfolio strategy. While there has been at least one IDSRN task order in all but two of the areas (prevention and training), task orders have been most heavily concentrated in five areas—quality improvement and patient safety (17 awards), system capacity and emergency preparedness that includes work historically referred to as bioterrorism work (11 task orders), cost, organization and socio-economics (9 task orders), health information technology (8 task orders), and data development (7 task orders).

4. Deliverable Requirements

To facilitate adoption and dissemination, AHRQ has modified the structure of task orders over time to place more emphasis on products that facilitate converting findings in ways useful for adoption and dissemination and to structure deliverables so that they support this function. Although all but 9 of the 58 task orders specified a final report, most also required various forms of presentation, manuscripts, or, increasingly, tool development (see Table III.4). Unique products were particularly common in FY 2003, when several of the contracts (mostly small ones) involved work that specifically helped the research team move one step beyond the findings to their application. However, as discussed in later chapters, the task order structure, even as revised, is still perceived by many participants as being less than supportive of this important step.

5. Distribution of Awards by Partner/Collaborator Team

The IDSRN task order award process is competitive for the most part. AHRQ staff characterize the initial selection of partner/collaborator teams as a “precompetition” that allows them to compete for task order contracts. In most cases, all teams are eligible to compete and encouraged to do so except in rare cases when a sole source award can be justified.9 Although the number of applicants for any given task order has fallen somewhat over time, several compete for most contracts. Their applications, typically due a few weeks after the request for task order (RFTO) is released, are reviewed by AHRQ and moved through an expedited award process. This feature differs markedly in internal control and speed from the more traditional processes AHRQ uses to award grants and cooperative agreements.

Given the task order award process, the distribution of awards among partners depends on how many RFTOs they respond to and how successful their applications are. The mix of topics covered in the solicitations is broad, and the interests and capacity of some partner/collaborator teams are narrower than others, discussed later.

9 Partners have been encouraged to add to their collaborating entities as needs emerge. Smaller partners have

been encouraged to expand—Denver Health, for example, has added a number of collaborating safety net providers to its team.

24

TABLE III.3

TASK ORDER AWARDS FY 2000-FY 2003 BY AHRQ PORTFOLIO AREA

Portfolio Number of Task Orders

Total

Total Amount of Awards

All Areas 58 $14,197,881

Care Management 2 499,833

Cost, Organization and Socio- Economics

9

2,090,638

Data Development 7 1,706,243

Health Information Technology 8 1,710,232

Long Term Care 3 798,209

Pharmaceutical Outcomes 1 190,000

Prevention 0 0

Quality Improvement and Patient Safety 17 3,952,962

System Capacity and Emergency Preparedness

11

3,249,764

Training 0 0

Source: MPR classification of AHRQ data on task order awards using portfolio designations provided in August

2004. Classifications may not in all cases be the same as those perceived by AHRQ though an initial review of the classification by the IDSRN project officer at AHRQ indicates that they are relatively consistent with how AHRQ views them.

Each of the IDSRN partner/collaborator teams has received some task orders (Table III.5).

The HMO Research Network, which includes many research affiliates, received the largest number of task orders (12). Marshfield and Emory each received the fewest (3). For the most part, teams received awards each year, especially after the first year. Marshfield is an exception, having received its awards in the first two years. Weill, affiliated with New York Presbyterian, received only one award before FY 2002, and the partnership’s involvement in bioterrorism research has been an important, though not an exclusive, source of contracts.

C. IDSRN ADMINISTRATION WITHIN AHRQ

The IDSRN does not have an extensive infrastructure. A single AHRQ staff member, responsible for technical oversight of the program, is supported by the contracts office and nondedicated staff who oversee particular task order contracts.10 While an annual meeting of

10 Kelly Devers was the original person responsible for the IDSRN but she left the agency before the IDSRN partners were selected. Fred Hellinger headed the program for some time afterward. For the last three years, Cynthia Palmer has headed the program.

25

TABLE III.4

IDSRN TASK ORDER AWARDS FY 2000-FY 2003, BY TYPE OF DELIVERABLES

Specified Task Order Products

All Task Orders

Awards in FY 2000

Awards in FY 2001

Awards in FY 2002

Awards in FY 2003

Final Report 49 10 15 15 9

Final Reporting 10 3 3 4 0

With presentation and manuscript

11

1

8

1

1

With other presentation 7 3 2 1 1

With tools 8 1 0 3 4

With manuscript 10 1 2 4 3

With AHRQ Presentation 3 1 0 2 0

No final report—other deliverables only

9

0

1

3

5

Source: MPR analysis of individual task order contracts

TABLE III.5

DISTRIBUTION OF IDSRN TASK ORDER AWARDS AND FUNDS FY2000-FY2003 BY TEAM

Partner Team

Total Funds Total

Awards Awards FY 2000

Awards FY 2001

Awards FY 2002

Awards FY 2003

Abt/Geisinger $1,474,962 6 0 1 4 1

CHCPE/RAND 1,672,237 8 3 2 1 2

Denver Health 1,856,645 7 0 4 2 1

Emory (previously Aetna/Prudential) 816,652 3 1 0 1 1

HMO Research Network 2,873,109 12 1 4 5 2

Marshfield/Project Hope 436,810 3 2 1 0 0

RTI (various systems) 2,120,001 8 2 2 1 3

University of Minnesota (MGMA, others)

1,086,965

4

0

2

1

1

Weill (NYPHS, VNSNY) 1,860,500 7 1 0 3 3

Source: MPR Analysis of information on task orders provided by AHRQ

26

IDSRN participants is funded and some other communication exists, AHRQ management tends to focus on task orders as opposed to the program as a whole.

Decisions on IDSRN task order topics, like other key agency decisions, are made formally

and centrally through the Executive Management Meeting (EMM), a weekly meeting of the agency’s center and office directors. Historically, IDSRN staff at AHRQ have solicited ideas internally and externally, including reaching out to partner teams by e-mail and newsletters. Internally, program staff “package” concepts for proposed projects along with a brief description, sending about three such packages to the EMM each year for approval. We were told that three criteria are used for approval: Is the concept a high priority for AHRQ? Is the IDSRN uniquely suited to study it?11 And does someone in one or two IDSRN partnerships have the data necessary to do the project? After a project is approved, a RFTO is developed and sent to the contracts department, which works with program staff to execute the competition. From our interviews with them, we believe EMM participants differ in their awareness of the process and how central they view the EMM in establishing priorities. In some cases, important decisions may have been made outside the meeting or via the director. The awards process and task order topics could change, given AHRQ’s new mission statement and the agency-wide focus on discrete portfolios.

Contracts are monitored by AHRQ task order monitors. Many are monitored by CDOM

staff, where the IDSRN is located, but staff from other centers also play significant roles in task orders that fall within their areas of responsibility. Of the 58 task orders awarded in program history, 32 are monitored within CDOM, although this is the case for some because an original project officer left, and the IDSRN project officer took it over (see Table III.6). Most of the others (18) are directed from the Center for Primary Care, Prevention, and Clinical Partnerships (CP3), which reflects the Center’s work with bioterrorism and informatics as well as primary care. The rest are managed by the Center for Quality Improvement and Patient Safety or CQuIPS (4), the Center for Outcomes and Evidence or COE (3) and the Office of Extramural Research, Education and Priority Populations or OEREP (1). The task order monitor has typically been involved in developing the work, and his or her job responsibilities include substantive work related to that area. The amount of direct involvement with the contractor varies by task order and task order monitor, according to their interest, as discussed further in Chapter VII, Section D. That is, monitors define their role differently and some are more substantially involved in the project work than others who oversee it but view their role as more of a facilitator.

While AHRQ holds an annual meeting for the IDSRN and communicates with the partner

teams through newsletters and e-mail communication,12 there are not enough funds for agency

11 Agency staff distinguish the IDSRN from other vehicles by the nature of the work, the involved partners, and

the contract mechanism. Primary Care Practice-Based Research Networks (PBRNs), they say, are anchored in academic settings and involve grants rather than contracts, though some objectives overlap.

12 We understand that annual meetings were not envisioned at the time the program was created so it will be valuable to learn about the impetus and desired intent of these meetings, which we assume relates to encouraging cross-network collaboration. Such collaboration and information sharing could be impeded if grantees view themselves in competition with one another.

27

TABLE III.6

DISTRIBUTION OF TASK ORDERS BY AHRQ CENTER

Center Total Task Orders

Managed Total Core Funded Task

Orders Managed Total Other Task Orders

Managed

All 58 30 28

CDOM 32 23 9

CP3 18 3 15

CQuIPS 4 2 2

COE 3 2 1

OEREP 1 0 1

staff to visit partner/collaborators, so staff have little insight into how the teams work—either program-wide or on individual task orders that involve collaboration with operational agencies. Often, few incentives or mechanisms exist to foster dissemination or adoption of research. Task orders may include products designed to encourage dissemination but these typically are produced when most available funds have already been expended. In addition, the contract typically does not include funding to do presentations of work. Non-academic researchers and those in practice may have little incentive or ability to subsidize this on their own. The agency has a particularly difficult time once a task order ends, since there is no continuing mechanism for monitoring post-contract activities, and partners have no direct financial incentive from AHRQ to promote implementation.

In Chapter VII, we review in more detail the strengths and weaknesses of this infrastructure

given AHRQ’s experience with it and make recommendations for improvements.

29

IV. TASK ORDER TYPES AND OUTCOMES

Given the diversity of the IDSRN task orders, we categorized them by type in order to make the findings more valuable to AHRQ as it thinks about the IDSRN and its future. The categories are defined by differences we perceived in how the task orders link research to practice. This chapter discusses both the distinguishing features of each task order type and what can be learned about outcomes of each type from the information available to us.

A. IDSRN PROJECT STRATEGIES

Table IV.1 profiles the seven categories that, in our mind, capture the diversity of types of task orders, or projects, conducted through the IDSRN.13 Most types of projects could probably not have been conducted without the infrastructure created by the IDSRN and its associated delivery systems. In different ways, each type of project represents work that is relevant to linking research with delivery systems. Beyond this common ground, however, the particular focus, strategy, and balance varies across categories.

The work supported by the IDSRN seems to fall into two main categories, although a few

projects do not fit neatly into either one. Projects in the first category build on the IDSRN tie to delivery systems to conduct research that can only be done with that linkage. Forty-eight percent of the IDSRN’s task orders and 50 percent of the FY 2000-2003 budget has supported this type of work, which includes the following types of projects:

• Operational data assessment and validation involves assessing the capacity of delivery systems to develop data and measures, which are one facet in organizational readiness to assess performance or identify improvements. An example is a study intended to validate AHRQ’s quality indicators in specific operational settings.

• Clinical intervention and assessment involves implementing clinical interventions based in the delivery system and evaluating their outcomes. An example is testing whether electronic order entry reduces medical errors.

• Research using IDS data involves health services research that is relevant to health systems and probably could not be addressed without the operational linkage. An example is using delivery system data to examine racial differences in health outcomes.

13 In their presentations on the IDSRN, AHRQ staff have distinguished four stages of the research cycle: needs

assessment, knowledge, translation and dissemination, and evaluation. They also have sought to examine the link to the delivery system (e.g., whether system data was used, how project results relate to operations). We developed the typology used here because we perceive it reflects the kinds of distinctions AHRQ staff have found relevant and conveys the range of diversity inherent in IDSRN in substantive terms that may be more meaningful to diverse stakeholders.

30

TABLE IV.1

IDSRN AWARDS FY 2000-FY 2003 BY TYPE

Type of Project

Description

Total Projects

Total Funding

Examples

Potential Link between

Research and Practice

Challenges That Influence

Value

Operational Data Assessment and Validation

Assess the capacity of systems to provide specific data, develop specific measures

4 $1,083,674 Capacity to conduct studies of race, ethnicity (Denver Health, Emory), operational validation of HCUP quality indicators (RTI); private sector data for NHQR (CHCPE)

Assesses one facet of infrastructure readiness to assess need for or make operational improvements

Uniqueness of individual systems; ability to move beyond assessment to make change or take appropriate action.

Clinical Intervention and Assessment

Patients in the IDS are involved in intervention; outcomes assessed

12 $2,769,120 Electronic order entry (Denver Health); otitis media practice guidelines (HMO Research Network); falls management tool (Emory)

Identify promising delivery interventions that work in practice

Evidence base for interventions; ability to generalize or bring to scale results

Research using IDS data

Take advantage of IDS administrative, claims, or other data to carry out applied health services research

12 $3,191,558 Racial differences in care outcomes (CHCPE), impact of payment policies on care (U of Minn), provider group characteristics and medication errors (HMO Research Network)

Enhances the knowledge base for understanding how health systems work and access to data otherwise not available for research

Identifying questions for research that have potential for ultimate operational value; ability to generate findings that build on evidence base and are taking the “next step”

TABLE IV.1 (continued)

31

Type of Project

Description

Total Projects

Total Funding

Examples

Potential Link between

Research and Practice

Challenges That Influence

Value

IDS Systems Analysis

IDS systems and flows are analyzed prospectively to identify performance, needs, or potential areas for improvement

8 (7 bodies of work)

$1,958,126 Modeling link between care transitions and iatrogenic injury (Abt/Geisinger), Assessing factors influencing diffusion of IT (RTI); assessing reasons for pneumonia hospitalization by Evercare patients (CHCPE)

Using delivery base to better understand problems or constraints and ways of intervening

Ability to generalize beyond a single system or point in time; follow-through on findings to identify and test improvements

Tool Development

Develop web or other tools for care delivery or public health improvement

17

(12 bodies of work)

$3,957,230 Electronic order entry (Denver Health); otitis media practice guidelines (HMO Research Network); fall management tool (Emory)

Identifying promising delivery interventions that work in practice

Evidence base for interventions; ability to generalize or bring to scale results

Dissemination Infrastructure

Projects that aim to support infrastructure in various ways to encourage dissemination

2 $594,310 National network of medical group practices (U of MN/ MGMA); Leadership conference on patient safety (Weill)

Improves channels of communication to get information out

Strategic importance of particular effort; relevance of infrastructure to other IDSRN work, AHRQ or field

Organizational studies using data outside of IDSRN

Projects that take advantage of IDSRN vehicle and participants to study issues relevant to IDS but otherwise not built on IDSRN unique qualities

3 $643,863 Quality provisions in MCO contracts (Marshfield; Project Hope); Hospital-Volume link (CHCPE); nursing home policies and quality (U of MN)

Addresses research questions that shed light on health care delivery organizations

Relative priority vis a vis work that builds more directly on IDSRN capacity

Source: MPR classification based on information on awards provided by AHRQ.

32

These types of projects appear to fall securely within the bounds of traditional health services research if one views such research as being focused on “building knowledge” through the use of established methods to describe and explain various aspects of health care delivery in a way that has the potential to contribute to improvements in policy, practice, or operations. A key issue in this kind of research relates to whether the “right” research questions have been asked and what vehicles exist to move the findings to practice, or at least to the point at which the next steps needed to move them in that direction are identified and acted upon. While topics for such research have historically been investigator-initiated, there is growing interest in involving users in the process. The mechanisms for such involvement do not exist for the most part. Projects in the second category aim, in various ways, to stretch the boundaries of traditional research to work that is more directly and immediately linked to an operational issue or need. The IDSRN appears to include two categories of projects of this type (described below), which correspond respectively to the needs assessment and translation parts of the research cycle. About 43 percent of IDSRN task orders and 41 percent of the FY 2000-FY 2003 projects fall into these two types.

• IDS systems analysis involves assessing IDS operations to identify the need for improvement and appropriate areas for intervention. An example of this type of project is the study of the reasons for hospitalizations for pneumonia in Evercare patients to identify how hospitalizations might be reduced.

• Tool development involves developing tools to improve delivery by providing a way for organizations to take action or change in a specific way. A very prominent example of this type of project is the group of IDSRN projects focused on planning tools to aid in responding to bioterrorism events at the local level.

In assessing the relevance of such tools to the translation of research into practice, the main issues relate to the quality of the research on which the tool is based, i.e., the extent to which research exists that is relevant to the operational needs of interest, how well the tool itself is developed, its applicability in a diverse array settings, and how easily and readily it can be or is adopted. For instance, studies of IDS operations are based in particular systems. Deriving findings that are actionable in that system is a key issue as is the potential generalizability of those findings to other settings. Two other categories of projects do not appear to fall neatly in the two preceding categories. One is research that uses data outside of the IDSRN and that really does not require the IDSRN for support. While such projects may both benefit from the experience of researchers affiliated with the IDSRN and focus on questions relevant to delivery systems, they do not use system specific data or take advantage in other ways of the unique link between the IDSRN and delivery systems. The second kind of project aims to reinforce and build infrastructure for dissemination rather than substantive development of ways of linking research with practice. In both cases, while the work is related to translating research to practice, it appears to heavily emphasize one or the other. Though these projects are few, they each account for about 9 percent of both task orders and funds.

33

As we discuss later, the categories of task orders in which the IDSRN should focus is an issue worthy of consideration.

B. PROJECT OUTCOMES

1. Measurement of Outcomes

A key question for AHRQ is whether the task orders are generating findings that lead organizations to make changes in how they operate (a measure of translating research to practice). AHRQ program staff distinguish three levels at which change can occur as a result of task orders: (1) within the partner/collaborator entity, (2) across the IDSRN network (to other partner/collaborator teams), and (3) outside the network.14 Most findings, or products, developed to date are viewed as having a geographically or otherwise limited audience, although some products, tool kits, for example, may have broader appeal. However, research on how adoption occurs suggests that the reference group for particular organizations and products varies, which needs to be considered in determining the likely diffusion of these tools and the kinds of organizations likely to adopt them.

We reviewed information on outcomes from IDSRN projects that were in the IDSRN

quarterly newsletter, other program documents, and interviews with both AHRQ and partner staff to assess the extent to which IDSRN task orders have led to change in any of the three areas described above. We applied the following definitions to determine whether a change in any of the three areas was made as a result of the task order:

• Task orders influencing interventions within a delivery system of the partner/ collaborator entity. This kind of change was operationally defined as reports that the task order had led to some operational change in the delivery system, regardless of the type or intensity of the change. Solely briefing managers on the work was not enough to lead to a positive report of change. Diffusion across organizations in the individual contracted team was included in the category rather than the next but such diffusion was very rare.

• Task orders influencing other IDSRN members. This kind of change was defined as a task order finding resulting in some active involvement or change by teams other than the one under contract.

• Task orders influencing actions external to the IDSRN. This kind of change was defined as reports that the work had been “used” or at least “considered” by operational entities apart from the IDSRN partners/collaborators. We did not assess actual implementation because information to do so was lacking evaluation.

14 More recently, AHRQ staff have also begun to distinguish between outside impact at the state/local, national

and international levels.

34

We also identified two additional outcomes that could signal that the task order was having or could have an impact on practice.

The first is whether project results are published in peer-reviewed, referred journals.15 This

is the traditional measure of how research adds value and is conveyed to the field. While IDSRN seeks to go beyond this outcome, it is still a useful criterion, especially for projects for which other outcomes are not noted or for which there is no follow-up. One executive noted, for example, that while research had to be “operational,” he also tended to look critically at the peer-reviewed literature for any evidence indicating that they should be doing something different from what they are doing now. Hence, publishing can support the translation of research into practice by adding to its credibility and visibility.

The second outcome is work that leads to a follow-on task order. The fact that AHRQ

judges the work to be worthy of a next step suggests that the results are viewed as potentially useful. The follow-on work is typically intended to make the results more useful in a practical sense. While it would be useful to include follow-on work funded outside the IDSRN as well, AHRQ has no systematic way of tracking this. We did occasionally hear of examples of follow-on work evolving outside of the IDSRN. The efforts about which we did hear typically involved research funded through the traditional RO1 process, not necessarily support for moving that research to practice.

2. Findings

Table IV.2 shows the outcomes of task orders on the measures defined above. In sum, 50 of the 58 task orders have been completed, so their outcomes could be assessed—at least to date. Of the 50, 30 were judged to have had some direct effect on outcomes, and 20 were not. Operational changes typically occurred at the level of the individual delivery system except for tools, which typically had a broader audience. Of the 20 task orders with no known direct outcome on delivery, five did result in a paper that had been published or accepted for publication, and one was judged to be important enough for a follow-on task order.

While our study was not designed to identify all the reasons for individual task order

success, we can speculate somewhat on why the 20 task orders had little operational effect. First, three of the 20 task orders involved research that was loosely linked, at best, to IDS systems and did not rely on their data. None of these three were adopted operationally, suggesting this kind of loosely linked model is not very effective. Second, research studies often need an additional step to move them into practice, which could explain the relatively high rate (8 of 11) at which research studies with IDS data that were not clinical interventions were not adopted. And third, programs of innovation need to anticipate some degree of failure as an inevitable part of the risk of innovation. The more fundamental question is probably not what

15Another potential outcome involves making presentations relevant to particular applications to a nonacademic audience. This probably excludes presentations at the IDSRN meetings themselves, since while they are relevant to dissemination within the IDSRN, they generally do not reach others outside the IDSRN. Because this outcome is harder to track and distinguish from academic reports to research colleagues, we do not include formal measures of it here.

35

TABLE IV.2

IDSRN TASK ORDER OUTCOMES BY PROJECT TYPE, FY 2000-2003

Impact of Task Order on

Delivery System*

Other Outcomes

Type of Project

No. Awards

No. Complete

Nonea

Local

Other IDSRN teams

External

Peer-Reviewed

Paperb

Follow-on Task Order Awarded by

AHRQ

Total 58 50 20 19 1 10 12 9

Tools 17c 15 3 4 1 6 1 4

Research with IDS data

12

11

8

2

0

1

5

1

Clinical Intervention

12

9

2

7

0

1

1

0

IDS Systems Review

8

7

2

3

0

1

3

1

Data Capacity 4 4 2 2 0 0 0 3

Research, no IDS data

3

3

3

0

0

0

2

0

Dissemination Vehicle Support

2

1

0

1

0

1

0

0

Source: MPR Analysis of available information aReflects projects where there was no explicit evidence of impact. Because site visit time was limited, we could verify many but not all the outcomes for each task order with IDSRN partners/collaborators. bNumber of tasks with 1+ publication. Only publications that are known to be published or accepted for publication are included. cThe 17 task orders reflect 12 separate bodies of work. The 17 include two sets of projects with an initial and follow-on task order and one set of four sequential projects.

these 20 task orders could have done to make them more successful, but what is needed to lead the other 30 to be more replicable in other locations. That, we believe, requires a focus on less idiosyncratic approaches designed to single systems than efforts to pilot and replicate interventions that could be more broadly applicable in a number of diverse settings.

3. Operational Effects by Task Order Type

The types of operational effects produced by the task orders also varies within task order category. We briefly describe the outcomes that we could identify from the information available to us for the major categories of task orders generated through the IDSRN.

36

Clinical Interventions. Clinically based interventions were more likely than other types of task orders to have an operational effect in local systems. This is not surprising, since the work is directly focused on a topic that has the potential to influence practice. For the most part, operational effects stemming from these types of task orders tend to be in the specific clinical areas targeted by the intervention. Both positive and negative findings had operational effects. Clinical interventions influenced practice in the following ways:

• Abt/Geisinger applied negative findings on the effectiveness of automated reminders about diabetes care in its decision to move toward a system of less-intrusive reminders across multiple conditions. (Automated reminders for very high-risk patients were maintained.)

• Abt/Geisinger applied negative findings on the use of an electronic medical record by patients (“My Chart”) to make the system more patient friendly and to retrain clinical staff on how to use the record.

• Denver Health determined that it was feasible to use electronic order entry via a vendor-developed system and that the alerts had a modest effect on the medication-ordering process. The organization applied negative findings on physicians’ reaction to a computerized order entry system to make system modifications.

• Denver Health found that although it was feasible to incorporate CDC’s guidelines for tuberculin testing and treating latent TB into its automated system, the guidelines identified a proportion of the patient population that was too large to treat. The organization used these results to improve screening and identification.

• HMO Research Network/Lovelace. On the basis of positive findings on the effectiveness of implementing otitis media guidelines for children, Lovelace decided to continue this intervention. (Moreover, follow-up work was funded by AHRQ to disseminate results.)

• Cornell/Weill adopted an electronic discharge order set to improve care for heart failure patients as they make the transition from hospital to home care; the team is now pilot testing the feasibility of this order set more generally with a large home care agency.

• Marshfield used results showing positive effects of coumadin clinics on adverse outcomes to support congressional testimony advocating pay for performance projects; the findings led to consideration of broader use of the clinic within the system, although we do not know if this occurred.

To date, the findings from most of these clinical intervention task orders have not been moved beyond the local system. This should be of some concern because broader adoption is important to AHRQ’s success in moving research to practice. Interventions geared to idiosyncratic system features may not be appropriate for more general application. To the extent that this is so, the situation raises issues about whether these types of interventions are consulting projects that benefit mainly the single provider system and if so whether this is an appropriate use of federal funds. To the extent that the lack of dissemination reflects a failure to link

37

interventions closely to the existing state of the art also raises concerns. In other cases, findings adopted locally may have the potential for broader application, but this potential either may not have been fully exploited (e.g., feedback on CDC guidelines) or there has not been enough time to do so.

Other Health Services Research Studies. The task of translating other kinds of

operational health services research into practice is more challenging than clinical interventions because the applications are not always as obvious as they are for clinical interventions. Nonetheless, IDSRN’s support for these other kinds of research related to IDS has resulted in some operational effects:

• Each of the two task orders that assessed the capacity of systems to generate race/ethnicity data led to changes in operational data collection so that these two elements would be captured better and more accurately in the future (Aetna/Emory, Denver Health).

• Analysis of adverse effects of medication errors in HIV care has led to obtaining pharmacy and provider feedback on how such errors can be reduced (Kaiser No. California/HMO Research Network).

• Findings on medication error and laboratory monitoring that were generated in studies of the effects of medical group practice on quality and patient safety led to system improvements in laboratory monitoring (HealthPartners/Lovelace of the HMO Research Network).

• Findings from research on the influence of the structure and culture of group practice and financial incentives on costs have been shared broadly with medical groups via MGMA. The information was the impetus for follow-on work intended to create a more formal collaboration with MGMA that would also help the organization make its data collection more nationally representative (University of Minnesota/MGMA).

Each of these projects involved data from the delivery systems and they related to operations and clinical processes. More abstract findings (such as on organizational structure) have not led to short-term change though provider systems do appear to be interested in the findings. Of the few task orders involving research that did not depend on data from the IDSRN, none produced findings that were used operationally, although some resulted in peer-reviewed publications.

Process Analysis. Task orders in which system processes were analyzed in order to better understand both their relationship to outcomes and the operational implications of these findings tend to vary widely. Outcomes of these task orders are difficult to identify given the information available to us, although three of these projects appear to have had operational effects:

• Analysis of the reasons for hospitalizing nursing home patients with pneumonia documented large variations across settings. This variation seems to reflect in part on nursing home leadership and on staffing differentials between weekend and weekdays. Evercare identified the leadership problem on its own and moved on it

38

before the research had been completed. We were told, however, that later findings confirming this step were valuable internally. (Evercare/CHCPE).

• Analysis of potential measures of emergency department overcrowding that could provide an early warning of emergency capacity problems and needs for diversion was applied to develop a data measurement and tracking system that is being implemented in the IDS and outside it. (The inability to secure follow-up funds limited follow-up analysis and further system testing and diffusion of the system elsewhere) (HealthPartners, Kaiser Colorado/HMO Research Network).

• Systems of transferring information on medications as patients are transitioned across settings of care were reviewed with a view toward reducing medication errors. The delivery system hired a “transition” pharmacist to help implement and test the system. An unrelated hospital in Seattle reports having been influenced by these efforts and changed its system of discharge planning and transitions of care, though it is not using a transition pharmacist. (Providence/RTI).

Tools. Operational effects from task orders involving tool development varied by tool type. Overall, however, findings from these types of task orders had practical applications, which is not surprising given that “practice” is implicit in tool development. The bioterrorism area accounts for a disproportionate number of task orders of this type, probably reflecting the strong demand for information in this area and the relative scarcity of existing tools. The following tools have been adopted:

• A web-based bioterrorism inventory and mapping tool developed for parts of Pennsylvania has been presented at multiple meetings of local public health officials. Users seeking to apply it outside of Pennsylvania would have to adapt it to show area-specific resources (Abt/Geisinger).

• Rocky Mountain Regional Care Model tools for selecting alternative care sites in the event of a bioterrorism threat will be used to train public health workers across the country. The tool was shared by the military with the 2004 Olympics Planning Committee, and was adopted for local use in Denver. (Denver Health)

• Spreadsheet-based tools for hospital administrators to respond with antibiotics to a bioterrorism threat, including resource materials and national guidelines, best practices, and development of a clearing/warehouse database were used to train personnel in strategic national stockpile workgroups. The bioterrorism emergency response model (BERM) has been widely disseminated. It was posted on the AHA and AHRQ websites, and distributed to all counties in New York State and elsewhere in the U.S. (Weill).

Outside of the bioterrorism area, the following tools have been adopted:

• Adapted software that was developed to examine racial differences in cardiovascular diseases to pilot test and develop a tool that delivery systems can use for this purpose.

39

The tool is being disseminated through the learning collaborative that AHRQ and RWJF are sponsoring with health plans in the area of racial disparities (Rand/CHCPE).

• How-to-guides were developed to support Medicare’s quality assessment and performance improvement (QAPI) projects. The guides, which help managed care organizations provide culturally and linguistically appropriate services for Medicare+Choice beneficiaries, have been the focus of a number of workshops. Also disseminated to non-Medicare plans, the guides are available on the CMS and AHRQ websites (Lovelace/HMO Research Network).

• Tools have been developed for hospitals to generate patient safety indicators from statewide hospital discharge data, using Utah as a target state (Intermountain/RTI).

In many cases, the development of tools required a series of IDSRN task orders in which the early work focused on reviewing the existing research, data, and needs in the area to develop an initial crude application. Subsequent work involved refining the methods, making the tools user-friendly, and providing support for adoption. The audience for the tools was not necessarily the partner’s local delivery system so much as a broader set of entities whose needs were prompted by external factors (e.g., terrorism threats, CMS requirements on quality improvement, interest in patient safety). Support for development often came from outside AHRQ. Given the limitations of funding available within AHRQ, how much of AHRQ’s core funding should support tool development that is not closely linked to translation of research is an issue for debate. The tools that have been developed under the IDSRN seem very useful and appear to have benefited from the involvement of a research perspective. They have also been highly visible, which raises AHRQ’s profile. But the connection between these tools and the rest of AHRQ’s research or the broader body of research is not consistently strong, so it may be worthwhile for AHRQ to consider what kinds of tools should have priority for the IDSRN.

4. Conclusions

The outcomes of these task orders can be viewed in at least two ways. The fact that over half of all the task orders produced findings that have had some practical application is a positive sign that the IDSRN has partly succeeded in developing work that can be translated into practice. On the downside, these applications tend to be somewhat narrow as well as locally limited, at least to date. This raises a concern about whether the IDSRN is structured to take full advantage of its operational links to develop findings and tools that are universal enough to move the field forward and create a more solid connection between research and practice.

41

V. CHARACTERISTICS AND PERSPECTIVES OF IDSRN PARTNERS/COLLABORATORS

In this chapter, we review the structure and characteristics of the IDSRN partners and their collaborators, how they work, their reasons for participating in the IDSRN, and how they regard the program. Table V.1 profiles each team, the rationale for participation and team development, how the partners work with their collaborators, and the focus of the task order work in which each team has been involved in to date.

A. CHARACTERISTICS OF IDSRN TEAMS

IDSRN partner/collaborator teams are diversely structured (Table V.1). Of the nine teams, five are based in entities that have a connection to an insurance or health services delivery system. AHRQ staff and many outsiders originally thought teams with these types of partners would be more dominant in the IDSRN. The five include:

• The HMO Research Network, a long-standing network of research affiliates consisting of large integrated and prepaid systems

• Denver Health, a large integrated safety net provider system

• Weill/New York Presbyterian, a large urban medical system

• Marshfield, a well-respected rural group practice

• United Healthcare, a major national health insurer (under which CHCPE falls)

Except for Denver Health, the mission of the unit within the five entities under contract with the IDSRN is to conduct applied research (based in organizations whose mission is insurance or service delivery). Denver Health, the exception, is led by the CEO with support by an internal staffer with research expertise.

The other four teams are based in universities (University of Minnesota, Emory) or research

firms (Abt, RTI) that have developed affiliations with service delivery systems to support the IDSRN. The Emory team was originally based within Aetna, a large insurer with whom it retains a relationship.

Teams vary in the breadth, diversity, and geographic scope of the delivery system they

cover. Abt/Geisinger, Denver Health, Marshfield, and Weill/New York Presbyterian are essentially based around a single delivery system in a locality. The Denver Health and Weill/New York Presbyterian systems include a broader range of integrated services, whereas Abt/Geisinger and Marshfield are essentially well-established multi-specialty group practices. Emory (as originally formed) and CHCPE (under United Healthcare) are essentially based around a single, though large, national insurer, which means that they have data on use of health

42

TABLE V.1

CHARACTERISTICS OF IDSRN PARTNER/COLLABORATOR TEAMS AND WORK

Partner Entity Name

Type of Entity

Rationale for Participation

and Team

Process of Working with Collaborators

Involvement of Collaborators in

Task Orders

Substantive Areas of Focus

Comment

Abt Independent contract research firm linked with Geisinger, a single delivery system in Pennsylvania

Business and professional opportunity for applied research; Built on staff contact with Geisinger and perception that Geisinger’s rural location and EMR would be of interest.

Materials shared and discussed with contact at Geisinger. Extensive collaboration with Geisinger having some independent contact with AHRQ.

Each task order has included Geisinger but Abt has generally been responsible for analysis and the final product. May change in the future as Geisinger has been developing its data analysis capacity.

Bio-terrorism response (2), patient safety, quality and information technology (4)

Reylon Media included as service vendor (web developer) and worked on one task order.

CHCPE (United HealthCare)

Research center embedded in large national managed care company with some research support from independent research firms (RAND, Health Systems Innovation) both of whom had a prior history with the Center

Hoped to secure stable funding for Center and have an impact. Initial developers were involved with AHRQ in the broader goal of a national data depository. Viewed IDSRN as a more limited effort to support the agency and quick results. Viewed Allina as bringing information from a vertically integrated delivery system and RAND bringing additional analytics.

Materials sent by IDSRN director to those in United Healthcare and in partners that they perceived might be interested.

Center views itself as only doing task orders in which it has some involvement; has led 6 of 8. RAND has had a large role in 4, leading two (on disparities) and having major tasks on others. Allina was only involved with 1.

Racial and ethnic disparities (3), LTC issues building on Evercare (1), Private sector data for quality (1), hospital/ volume link (1), impact of M+C limits on pharmacy coverage (1), identifying at risk enrollees (1)

Allina Health System had to drop out. Center’s role within United has changed, with extensive staff turnover and less IDSRN commitment.

HSI is a small firm focused on technology whose head has a history with other participants.

TABLE V.1 (continued)

43

Partner Entity Name

Type of Entity


and Team



Task Orders


Comment

Denver Health

Large IDS safety net provider with CEO leading team supported by internal research lead

Viewed their integrated system and safety net focus as unique, with a CEO head adding the message that this is important work. CEO had historical ties to John Eisenberg. Team focused around Denver Health, bringing in collaborators for specific task orders as vendors.

Denver Health is in the lead on all task orders. Materials are circulated widely within the organization. Specific collaborators and associated vendors elicited for individual task orders (5 of 7 have involved collaborations).

Plays a strong coordinating role and is responsible for the analysis. Uses vendors (e.g. Siemens) as needed Bioterrorism work has brought in state public health organizations in the region. One of few partners to work with systems in other teams (Weill, RTI).

Race/ethnic disparities (2), regional bioterrorism response (2), use of technology to improve care (2), hospital redesign (1)

University of Colorado is a fixed collaborator but the two institutions also compete in the research field as the university is said to want access to systems data while Denver Health aims to build up is unique research identity. Thus, the university has only been involved in one task order and potentially a new one).

Emory University- based center working with Aetna (where researchers were located in a center at award until 2001)

Center was embedded in Aetna during the solicitation and being encouraged to get outside funds. IDSRN seemed “a perfect fit”. Aetna had data and wanted to be a good public citizen. No consideration was given to collaborators because, center staff say, they were part of Aetna.

When at Aetna, task order requests and other information were circulated and posted. IDSRN staff still call Aetna as appropriate but the relationship is less interactive.

The partner was at Aetna for the first task order (race/ethnic data) and at Emory for the other two. Aetna was involved in one of the latter and the other was an AHRQ initiated task with the Emory Center on Health and Aging.

Data to study race/ethnicity (1), modeling care use in bioterrorism (1), fall management in LTC (1)

There was a year’s hiatus during the transition. Researchers continue to have a relationship with Aetna but find it is harder to know the needs. They would like a broader set of systems and data now that they are independent and have struggled to find Emory/Atlanta collaborating systems with data.


44

Partner Entity Name

Type of Entity


and Team



Task Orders


Comment

HMO Research Network (With KP Northern California as administrative entity)

10-year-old research network focused on collaboration in research involving systems. 12 of 14 members participated. Members are affiliated with Kaiser (6 regions) or independent and local HMOs and delivery systems (6)

Strong interest because IDSRN seemed what the network does. All but United and Aetna joined in. KP took lead because it was their turn to do so. Limited involvement of outside researchers (despite requests) because that was not viewed as needed given internal resources.

Task orders circulated to all collaborators with a date for response. If multiple entities want to respond they either work it out (sometimes including several) or (more recently) have the option to submit two bids. Proposals are prepared by those leading task order and Kaiser is intermediary.

Task order leaders include KP N. Cal (4), Lovelace (4), Group Health Cooperative (2), Health Partners MN and Kaiser GA 1 each.) 7 of 12 organizations have received some funds and all but one of the others remains interested in future work.

Quality and patient safety (4), cultural competency (2), maternal and child health (2), working conditions and patient safety (2) ER overcrowding (1), synthesizing knowledge of care coordination (1)

Some collaborators have been more active than others though all but a few have been involved in one or more task orders. Say budget precludes multi-site interventions.

Marshfield Rural group practice affiliated research center collaborating with Project Hope. (Center is small and embedded in larger more clinically oriented research organization.)

Interest spurred by prior contact with John Eisenberg and AHRQ. HSR director got support from the Center’s medical director. Project Hope involved based on prior experience and initiation of contact.

Task orders circulated and discussed among team. No recent action. Contract has been on hold, with AHRQ perceiving performance issues.

Marshfield has led two of the three task orders with some help from Project Hope. Project Hope led one early on, which had a minimal Marshfield link.

Quality incentives in provider contracts (1), quality/patient safety (2)

CODA, a collaborator for survey work was not used because these projects do not involve primary data collection.

RTI Independent contract research firm collaborating with geographically dispersed and independent provider systems

Lead staff attracted to potential of IDSRN for operationally useful research and data. Systems identified through existing work and contacts to generate a team with what they viewed as having a varied mix.

RTI sends out task orders to collaborators with request for a fast response. Response depends on team interest. RTI has secured Partnership funds from AHRQ and convenes two team meetings per year. RTI takes lead on developing proposals with systems support.

RTI has led each task order with one or more system collaborators (typically 2 or 3 per task order—one only RTI).

Intermountain, Providence health system, and UMPC have been most active systems.

Validating HCUP (2) and related training (1), information technology in IDS (1), risk reduction and drug information (2). Regional bioterrorism tools (2)

Two of the original collaborators—CareOregon and Summa, have fallen out as one disbanded and the other got less research focused. UNC has also had leadership changes. They and consultants have not been as involved as anticipated. Baylor (TX) has been added recently.


45

Partner Entity Name

Type of Entity


and Team



Task Orders


Comment

University of Minnesota

University-based research center collaborating with provider and insurer systems in Minnesota and with the MGMA for medical groups nationally

Opportunity for funding in areas of interest. Collaborators come from existing relationships and bring data (BC/BS Minnesota, Allina, Park Nicholett) and information on medical group characteristics and financials (MGMA, contacts with medical groups).

Task orders are reviewed by the director and only sent on selectively to team members as they view appropriate. Decisions on bids generally made based on the defined research priorities of the lead staff, which include building up a set of relationships with operational people to address issues of mutual interest.

University researchers have taken the lead in managing and analyzing work, with BCBS Minnesota, MGMA and other collaborators supporting for data collection.

Financial incentives and medical group practices (2), establishing a network of medical group practices (1), LTC quality and patient safety (1)

Team is mostly focused on developing a body of data, applied research and relationships with group practices nationwide. From a base originally in Minnesota it is expanding. Highmark of Pittsburgh is being added. Focus has meant some local collaborators have been used less than they’d hoped.

Weill Medical College of Cornell University

Medical school/hospital system (NY Presbyterian) based research department drawing on its system and other resources in the NY area

IDSRN attractive in light of merger to form New York Presbyterian IDS for which department does system-wide research on quality. Opportunity for funding also attractive.

Weill reviews task orders and sends them to team members or others on an ad hoc basis as they see appropriate. If an RFP is not of interest, they sometimes send it to the team. Columbia submitted a (failed) bid this way.

Weill has led all the task orders. The bioterrorism work has involved public health organizations in NYC. The quality work has been within the Presbyterian health system but the VNSNY collaborated on the work involving home health.

Statewide planning for bioterrorism (4), quality improvement (4, including one involving transitions to home health and classified as LTC)

Many original collaborators in the Presbyterian system have not been active and Weill instead has relied on others in NY appropriate to particular work.

46

services and some ability to influence providers but fewer direct links to them. In contrast, the HMO Research Network and RTI teams include collaborators in multiple, geographically dispersed organizations, and the University of Minnesota capitalized on its historical links to state insurers and providers and the Medical Group Management Association (MGMA) in the effort to reach out nationally to medical groups.

Team Evolution. To some extent, the IDSRN teams evolved similarly. In each case, the

impetus for the decision to form an IDSRN partnership and apply for the work came from a “champion” within the partner organization who identified the IDSRN as an important opportunity and led the work to develop the team and the proposal. The champions in each organization except for Denver Health are researchers with an interest in applied systems research. While the nature of the potential task orders was a draw for those championing proposals, the potential to garner financial support was relevant for almost all teams. Even partners with some “hard” money support from their associated systems, were also expected to bring in outside revenue to support them. In addition to funding, historical business and other connections were important in team development, as champions drew upon contacts and relationships with researchers and delivery systems to form the team. AHRQ—and—particularly John Eisenberg’s outreach to those in delivery-based settings—helped motivate individuals to respond to the opportunity presented by the IDSRN. Indeed, a number of the teams cited earlier discussions they had with AHRQ about the potential for the IDSRN or similar research/delivery collaborations. AHRQ connections to delivery-based researchers also contributed to the team-building effort via the relationships the agency had developed through its work to “build bridges” between practitioners and researchers.

Infrastructure. Infrastructure is especially relevant in large, diverse, geographically dispersed, multi-organization teams like the HMO Research Network, RTI, and the University of Minnesota. The HMO Research Network’s informal, but highly developed, infrastructure pre-dates the IDSRN. Having evolved over a 10-year collaboration among delivery-based research centers, the network already had multiple grants for collaborative work, well-established processes, and experience working together. At the other end of the spectrum, the RTI team’s infrastructure is unique to the IDSRN in that the firm built its team from scratch, using the contacts and experience of those championing the bid. The University of Minnesota, on the other hand, having no direct link to a delivery system, could not use direct relationships with provider systems to build a team infrastructure. Instead, it relied on linkages with well-positioned intermediaries like the MGMA and Minnesota Medical Society that had the means to reach out to providers. Both the HMO Research Network and RTI viewed their ability to convene periodic meetings of a diverse set of collaborators as important. The HMO Research Network has done this for many years via well-developed conferences, whereas RTI, perceiving meetings to be valuable, elicited support for them through an ultimately successful application for an AHRQ partnership grant.

Infrastructure is also important in less complicated systems. Research entities embedded

within a service delivery system vary in both sophistication and history. Whereas individual collaborators within the HMO Research Network and CHCPE at United Healthcare have a long history of system-related research and established ways of communicating with system executives, both research sophistication and communication links with operations were substantially less developed elsewhere. Abt/Geisinger, Denver Health, and Marshfield, for

47

example, all have relatively undeveloped research centers that they hoped to strengthen through participation in the IDSRN, as did some of the RTI collaborators.

When research entities are not well established and embedded in a delivery system, the

support of senior executives on the clinical or administrative side appears to have been particularly important to IDSRN participation and performance. Denver Health, for example, benefited from the fact that the research focus for their IDSRN reported directly to the CEO, who led the IDSRN team. Weill/New York Presbyterian benefited from the fact that the IDSRN team director reported to top leadership of both the delivery system and the medical school. Marshfield’s team, in contrast, was weakened by a number of structural and other barriers that limited communication with and support from top clinical management.

When the leaders of the IDSRN partner were outside the delivery entity entirely, it was

essential to have a link between the outside leadership and a well-positioned manager in the collaborating delivery entity. RTI staff we interviewed noted, for example, that a good test of such a position was whether the individual with whom they were working could actually commit the organization to participate in the IDSRN. However, RTI staff also said that being able to effectively do research within a system means that the system itself must have some commitment to a research mission and some capacity to support the study. As RTI learned about these features and responded to changes in the orientation of the delivery system over time, the identities of the active participants in their IDSRN network also changed over time.

Several individuals we spoke with made the point that IDSRN success depends, to some

extent, on a delivery system’s commitment not only to research but also to marrying research with clinical enterprise. Interestingly, the three IDSRN teams with the fewest task orders also appear to have the weakest link with delivery systems: Emory (which was “spun-off” of its Aetna base); Marshfield (where communication and strong links with the delivery system appear weak because of organizational issues); and the University of Minnesota (which, by design, worked through “broker” organizations that mediated links with providers). But not every large delivery system embodies or is suited to the concept of marrying research with clinical enterprise. At the same time, interviewees also stressed that the major mission of delivery systems is to provide medical care, not to conduct research. This means that while systems might be supportive of research and even underwrite some of its costs, the research cannot interfere with medical practice and ultimately has to make sense within the dominant mission of the organization. Thus, there always are tensions inherent in the two sets of objectives.

B. TEAM STABILITY

The organizations in which the IDSRN partner/collaborator teams are based have been markedly stable for the most part despite a highly fluctuating health care market over the period since IDSRN was launched. By stability, we mean continuity in team leadership and in mission support from the delivery system for the research entity in which the lead staff are based. Obviously, such stability is important in sustaining collaborations such as the IDSRN.

Stability of Partner Leadership and Organization. Stability was the rule in the partner

leadership and organization except in the insurer-based teams at Aetna and United Healthcare. The research centers where the IDSRN was based in each system received substantial fiscal support from both organizations. Within each system, however, core funding for the centers

48

eroded as the economic environment became tighter. In addition, both organizations also became more mission driven, creating increased interest in assessing how IDSRN activity related to the core business mission and overall organizational priorities. In Aetna’s case, the center where the IDSRN was based was originally part of Prudential. Bill Roper, then a key executive at Prudential, championed the center and viewed it as a means of enhancing the scientific basis and consistency of the organization’s work in quality monitoring and HEDIS measurement, particularly with respect to prevention, disease management, and information for decision making. After Prudential was acquired by Aetna in 1999, support for the center dwindled as multiple executive leadership changes reduced knowledge “at the top” of the research center’s history and people. When Jack Rowe became head of Aetna in 2001, the firm decided to focus more heavily on its core business. In addition to receiving 75 percent of its support from core funding, the center where IDSRN was based also duplicated, from some perspectives, the pre-existing Aetna unit developed to conduct internal data analysis (USQA, Aetna Integrated Informatics). Jack Rowe negotiated a financial arrangement with Emory University to transfer the center and its staff to Emory “with a dowry.” From an Emory base, the center remains within the IDSRN (after a year hiatus when its status was being determined) and continues to have an affiliation with Aetna and a business agreement, giving it access to Aetna’s data. Maintaining “insider knowledge” and relationships with Aetna has been more difficult under the new arrangement because of changes in staff, procedures, and issues important to Aetna that Emory researchers are less familiar with because they are no longer integrated into the company. The transition also means that Emory researchers are under more pressure to be fully grant funded. As an Aetna-related entity, the center did not seek other collaborators, but at Emory, it has more of an incentive to do so, and there are fewer constraints on this process.

In CHCPE’s case, the instability revolved around changes in staff and in changes in where it

is placed within United Healthcare and what the organization views as its mission. United Healthcare has six business divisions. The original heads of the center—Sheila Leatherman and, later, Eileen Peterson—believed that it was important to the center’s mission to be viewed as an independent entity. To achieve this, they negotiated an agreement in 1998 with United Healthcare’s CEO through which they would report directly to him rather than to an operating division.16 United Healthcare would subsidize CHCPE for five years (at about 50 percent of total costs, including overhead) so that it could pursue grant-centered investigator-led research. Since the center was formed, there has been extensive turnover in top leadership. As a result of a recent reorganization, the center is now embedded in one of United Healthcare’s major business divisions (Ingenix) in a unit responsible for Applied Health Care Informatics, most recently headed by Cary Sennett. As a result of the change, CHCPE is expected to bear the full United Healthcare overhead with no subsidy; in addition, there has been more staff turnover and vacancies in some leadership positions. Although support for IDSRN is still intact, the effects of the changes on IDSRN participation and interests are uncertain.

16 Both Aetna and United Healthcare at that time were members of the HMO Research Network though each

decided to participate in the IDSRN separately rather than through that Network. Aetna no longer is eligible for membership given the organizational changes but United Healthcare (CHCPE) still belongs. Similar to many of the centers that are part of the HMO Research Network, researchers at these insurer-based centers viewed themselves as doing quality research relevant to systems and operations with an external as well as internal focus.

49

Stability in the Partner/Collaborator Arrangement. Given the variation in the IDSRN partner/collaborator teams, the task of judging stability among the collaborating entities is complicated because collaboration is structured differently from one team to the next (see Table V.1). In general, there appear to be three major sources of instability between partners and collaborators. First, when teams involve multiple delivery systems that have never worked together, collaborations can erode as it becomes apparent that the capabilities of individual collaborators are less robust than they may have seemed (e.g., RTI collaborators like Summa). Collaborations can also erode if the IDSRN task order scope and funding make certain kinds of collaborations less relevant or less feasible (e.g., Weill’s collaboration with Sloan-Kettering to pursue cancer-related topics). Second, organizational fluctuations and staff turnover can be a source of instability. Although the former were markedly low in most of the IDSRNs, it affected a few like CHCPE and the University of Minnesota, which had proposed working with staff at Allina. Third, particular collaborators chosen for a particular purpose—survey or website development, for example—were not always used if that skill was not needed in task order work for one reason or another. Surveys are not popular because they require OMB approval, and website development is relevant to only some projects.

While instability in collaborating entities was not extensive, the nature of the task orders

awarded and the dynamics in particular teams meant that some collaborators received little work. Some of them remain interested and supportive of both the IDSRN and the team. Others are disappointed that the effort involved in the initial IDSRN procurement did not lead to more opportunities. Most current IDSRN partners are likely to compete again for the awards, but some are likely to modify the composition of their teams.

C. HOW TEAMS WORK

1. Task Order Decisions and Response

IDSRN partner/collaborator teams appear to have communicated for the purpose of developing a task order response, a process that took one of two forms. The most prominent was collective decision making, but some teams used a more centralized process.

Collective Decision Making. In this approach, used by at least five teams, the lead partner

circulates each task order request to the all the affiliated organizations requesting an expression of interest by a certain date. Decisions about responding to a task order and how to structure a bid are based on the level of team interest. Partners that are not delivery systems typically require a show of interest from one of their associated delivery systems in order to respond, since task orders are structured to take advantage of the delivery base. In most cases, the individual task order response involved only one or a few delivery systems even in teams with more systems. According to those we spoke with, this choice was driven by limits on the available funds.

Collective decision making among a diverse group of entities appears to have been an issue

mainly for the HMO Research Network, as the other teams were not structured in ways that depended on multiple expressions of interest from delivery systems. Because the network had a history of working together, interviewees said that the process of determining who would be involved in a response was a collegial one in which previous decisions were considered. The need for this process has been reduced because AHRQ will now accept multiple bids from the

50

HMO Research Network. Although collaborating entities may often be competing against each other, multiple awards are sometimes made. The submission of multiple bids reduces the time the HMO Research Network needs to respond to task orders.

Centralized Decision Making. In the centralized approach to decision making, the partner

is seen as both the core of the IDSRN team and the dominant decision maker. However, the way in which task orders are distributed by the lead partner among collaborators varies on the basis of task order content and the partner’s interest. In Denver Health, for example, task orders are circulated widely among staff in the delivery system to determine their interest before involving others. Although Denver Health had a few named collaborators on its team from the start, the director prefers to add collaborators as the team moves forward and as needs and roles become apparent via particular task orders. The more academic-based organizations seek first to determine investigator interest, although these partners also work to maintain ongoing collaboration with particular organizations linked to delivery. Like Denver Health, Weill/New York Presbyterian seeks to identify collaborators appropriate to specific projects. The Weill bid originally included many care delivery organizations within the New York Presbyterian system as well as a few others. In conducting task orders, Weill has relied on a range of system-affiliated providers that has been narrower than anticipated in the original proposal. This reflects not only the nature of the task orders to which Weill/New York Presbyterian responded but probably also a growing knowledge of the challenges in working more closely across diverse, even if associated, delivery entities in the New York Presbyterian system.

Regardless of the decision making approach, the entities that would be involved in carrying out the work are involved in developing the proposal. In teams led by delivery-based investigators, their use of research support from a collaborating organization not involved in service delivery varies. The HMO Research Network rarely uses such support, as it sees its internal capacity (across the systems) as strong. CHCPE often partners with RAND, although it sees itself as having a central, substantive role in the work. Marshfield has relied on Project Hope to enhance its research skills, but competing demands on Project Hope investigators’ time and their lack of detailed knowledge of Marshfield’s data is seen by Marshfield as constraining Project Hope’s involvement in some task orders. Weill/New York Presbyterian tends to internalize the research in most of its task orders but draws extensively on operational experience from outside. The composition of its team, like that of Denver Health, tends to vary. In teams led by research contractors (Abt and RTI), the partner plays a major role in responding to task orders because the delivery-based collaborators tend to have less experience with the bidding process. RTI also believes that its role as the point of communication with AHRQ and its hands-on control of the task order process are important to team success.

The way the IDSRN was set up resulted in some degree of competition across teams. As teams learned about one another, they seemed to get a better sense of where to focus their energy so they could be successful, leading to fewer proposals for each task order. We heard (often not directly from the named organization itself) that competition has made some participants reluctant to share their ideas with others, since funding is limited and awards would be unlikely to include multiple partners. In some interviews, we were told that competition is a particular concern for those based outside of delivery systems. One participant suggested that if AHRQ wants to encourage more cooperation and collaboration among teams, they should somehow signal this interest in developing their RFTO.

51

2. Focus of Task Order Work

The IDSRN partners vary in the kinds of work they have taken on. However, the diversity in the task orders and the general interest of partners in working with the program make it difficult to clearly distinguish between the types of work in which each is involved. All but one of the partners has conducted at least some task order work in quality and patient safety for example, an area for which AHRQ has extensive and dedicated funding support. The one exception is Emory which has had no work in the quality or patient safety issues. In addition, while the University of Minnesota had one task order in this area, the lead partner views itself focusing more on the administrative and economic side of health care rather than on the clinical side. The clearest differentiation between partners in terms of the topics they have addressed involves bioterrorism and long-term care—two areas that had not been anticipated when the IDSRN was developed.

Work on bioterrorism has been a challenge for many teams because, as mentioned, the topic

was not envisioned when the IDSRN was put together, so the IDSRN teams were not structured with this topic in mind. As one HMO Research Network interviewee noted, “Bioterrorism wasn’t the thing we were thinking of when most of us started our careers.” While the network observed that it had “fallen down” in this area despite having submitted one unsuccessful proposal, it also perceived that it made more sense for other IDSRN partners to take the lead. Although five partners received at least one bioterrorism award, the most prominent work was performed by Weill/New York Presbyterian and, to a lesser extent, by Denver Health. Weill’s activity here has three sources: location (being based in a large medical center in New York City), serendipity (previous work becoming more relevant since 9/11), and an ambitious young investigator who saw the opportunities in the work and seized them. Denver Health’s involvement in bioterrorism stems from its role as the major large safety net provider in the region and the fact that the IDSRN head also heads the system and thinks broadly. In both cases, new collaborations with a broad set of stakeholders in public health and emergency response had to be built to support the work. Abt, Emory, and RTI also did some work in the bioterrorism arena.

Like bioterrorism, long-term care was not a focal issue when the IDSRN was set up. The emphasis was on integrated care systems involving acute care delivery, the focus of most health insurance. However, AHRQ staff working in long-term care services viewed the IDSRN as a potentially valuable vehicle for working with long-term care providers as well, since it had links with delivery systems. These staff therefore sought to identify people or units within the networks that provided relevant links for exploring issues important to long term care. As a result of their interest (and because they also had a little funding for their interests), the IDSRN has been used to support three task orders in long-term care.

Often sole source, the long-term care task orders were built around the capabilities of

individual IDSRN teams. One contract went to Evercare, a United Healthcare subsidiary focused on providing medical care under contract to those needing long-term care services. Originally serving nursing home residents, Evercare now has as many patients out of nursing homes as in them. A second contract, which went to Weill/New York Presbyterian, was focused on better information exchange between home care providers and hospitals about heart failure patients in transition from the hospital to their home. This work has involved the Visiting Nurse

52

Service of Greater New York. A third contract, which went to Emory, concerning falls among the aged, takes advantage of expertise in the Emory Center for Health and Aging.

AHRQ’s work on long-term care within the IDSRN is unique relative to the work

represented by most other task orders in that AHRQ staff have drawn on their own knowledge of the field—both to identify collaborators that are appropriate to particular partners and to proactively use them to encourage IDSRN work in the long-term care area. While teams have generally viewed such proactive efforts in a positive light, the suggested collaboration has sometimes caused concern for a partner because of competition or a history with the suggested collaborator. AHRQ may want to be sensitive to these concerns and provide a way for partners to raise their concerns about proposed collaborations freely. IDSRN’s “re-orientation” toward long-term care is an indication of the program’s flexibility and the fact that some AHRQ staff view it as an opportunity to pursue topics of interest.

3. Conducting Task Order Work

The process for conducting task order work varies with the nature of the work and its requirements. The process mirrors the process of responding to a task order. Each task order typically has a lead investigator who directs the work. The selection of the task order team is typically negotiated as part of the proposal development. Because of a relatively limited budget, only a few people are part of the funded set of investigators, although the work itself might bring in more people such as in the case of some bioterrorism task orders.

Some IDSRN partners led each of the task orders in which they were involved. Others had

collaborators lead some task orders. The latter model is likely to have been used mainly in two cases: (1) for projects led by investigators at research contractors and universities (i.e., in entities not directly affiliated with delivery systems) who used data from delivery systems and (2) for projects headed by a delivery system lead partner who was also a strong, independent research collaborator. For example, RAND and CHCPE worked together on two sequential projects involving race/ethnic and gender differences in cardiovascular outcomes, adding RAND’s methodological expertise to the skill of CHCPE’s staff and bringing United Healthcare’s access to and knowledge of data from their system. The work resulted in a tool to examine disparities, which CHCPE brought to its operations staff in order to get their input on its feasibility.

In interviews, we heard diverse reports of the relative ease of obtaining access to both systems data and systems people depending on whether the researchers are located within an entity affiliated with the system. The responses indicate that people with a formal system affiliation can more easily access data because they are likely to have a history with the organization, an understanding of the system’s capacity to respond to certain needs, and more ready access to decision makers. If the lead investigator on a project is not based in a delivery system, the lead investigator needs a strong person well-positioned in that system to provide access to data and supporting analysis. An ongoing, trusting relationship between outside and systems collaborators is also important.

As a result of the IDSRN experience, relationships within the teams have matured, and

access to data for collaborating systems and researchers has improved. The value of maturity is something that AHRQ staff may want to consider in deciding whether to give existing partners any preference on the recompete.

53

4. Disseminating Results and Translating Them into Action

The task orders have varied substantially not only in focus but also in the types of findings they generated and in the “readiness” of findings to be translated into practice. Some task orders, for example, may generate findings that are not very interesting or worth disseminating. In others, the work is relevant but needs to be followed up or integrated with other work before the findings are ready to be broadly disseminated or translated at all. Here, we focus on the infrastructure and incentives for disseminating findings and translating them into practice, assuming that the findings are suitable for this purpose. (The previous chapter provided analysis on actual task order outcomes to the extent that they can be assessed from the information that is available.)

We consider two dissemination mechanisms—publishing and presentations—and three

ways in which research may be translated to practice—within the IDSRN partner/collaborator team, throughout the IDSRN, and from the IDSRN to organizations outside the IDSRN entirely. The way in which each IDSRN partner/collaborator team was positioned to support these activities differs, as do the incentives for pursuing these activities. Furthermore, the IDSRN’s infrastructure provides very little organized support for these activities within the partner teams or across the program. Thus, while the kinds of people attracted to the IDSRN have an interest in pursuing dissemination and translation, the incentives and support to do so are limited, and the constraints are many.

Publications, Presentations, and Other Research Products. The IDSRN attracted investigators who are interested in applied research. As such, virtually all of those we interviewed expressed an interest in both presenting and publishing their results. The settings in which the IDSRN is based provide some support for publishing results in peer-reviewed journals and presenting them at professional meetings. While this is obviously true for teams connected with academia, such environmental incentives are also common in some other settings, including delivery systems. Even when projects had been completed and no publications had been drafted, researchers still indicated that in the long run, they intended to develop a paper, although whether they did so is not clear.

Products other than articles and presentations are often needed to translate research into

practice. These can include tool kits, manuals, best practices guides, and so on. Because such products are not, however, an entirely natural “by-product” of a project, their development within the IDSRN has depended heavily on whether they were made explicit in the task order structure. Nontraditional products like these appear to have been more common for IDSRN task orders supported by outside funders whose constituency involved operational entities such as public health officials, Medicare+Choice plans, and so on. If a collaborator represented operational organizations (e.g., MGMA), this sometimes provided a dissemination outlet leading to specific user groups.

Within the IDSRN, time and money are formidable constraints on the development of task

order products structured to reach out to an audience that might use the work. Except when required in the contract, the competition for investigators’ time has been a major constraint on their efforts to publish and present their work. This is particularly true when incentives in an organization are structured to encourage movement from one project to another as funding is

54

expended (contract research) or when new operational issues arise (delivery-based staff). [Survival on “soft money” is a fact of life for many, including university-based researchers.]

Task orders under the IDSRN have increasingly included papers and other products among

their deliverables (see Chapter III.) However, these alone do not appear sufficient to encourage dissemination of results to operational entities. As we discuss later, the IDSRN partner/collaborator teams almost universally see the task orders as demanding a large amount of work for very little money.

In addition to money, the period of performance itself has also constrained publication and

presentation. Because rapid turnaround is a priority in the IDSRN, projects are typically slated to run for 12 to 18 months. Until projects are completed, however, it often is not possible to identify the findings clearly enough to determine which ones are most worthy of disseminating, how, and to whom. Completing the project and preparing the dissemination strategy and products within the period of performance has been an issue for some teams. While some partners said they were able to get a no-cost extension to support dissemination, others said that the contracts office refused them this opportunity. They also said that it was frustrating to have unexpended funds in their budget while being denied the opportunity to use them to draft an article or to present a paper at a meeting. The frustration was magnified by the fact that the IDSRN was focused on dissemination, particularly in later years.

As we observe the situation, an incomplete understanding of what makes an effective paper and how to get it published has also limited dissemination activity under the IDSRN. Our interviews with both IDSRN teams and AHRQ staff suggest that some participants are more experienced with the publishing process than others, and that an understanding of how to “place” articles effectively has sometimes been lacking. Unfortunately, there are few formal mechanisms in the IDSRN to teach those who know less, especially if these individuals are not in systems that provide such support naturally through a mentoring function. The exception we noted were AHRQ task order monitors who took it upon themselves to offer such support in the interest of helping others to develop products that could be published or disseminated.

Applying Findings in Delivery Systems. The IDSRN is structured around delivery

systems based, in part, on the assumption that tying research to such systems will make the work more relevant to system objective and the results more accessible. In our interviews with IDSRN partners and those in delivery systems in which partners are embedded, we found support for this assumption. The IDSRN is not uppermost in the minds of those in delivery organizations, but at least some key executives in each system tend to know that their organization has been participating in the IDSRN and take pride in this participation—both with regard to their affiliate and in terms of the prestige afforded by the link with AHRQ. In addition, the IDSRN helped to fund work that, according to some IDSRN partners and collaborators, might otherwise be unfunded because of its applied nature and thus difficult to perform.

The fact that many IDSRN partners are based in an operational system seemed to “open the door” more readily to both formal and informal communications about project needs or results and what that may imply for delivery. One executive noted that because the IDSRN staff are part of the company, they had automatic access to him, whereas it would be more difficult for an outsider to get an appointment. While this executive is open to collaboration with outsiders who

55

can deliver relevant work, he said that given the time required to develop the relationship, it makes more sense to try first to work within the system.

In another operational system, the IDSRN is based in a large medical group where the

IDSRN partner has a long and established history of collaboration in projects begun well before the IDSRN. The head of the IDSRN holds a position in the group that leads him to meet at least monthly with top medical management, prepare updates for the medical staff four times a year and meet periodically with the staff to brief them on specific projects both within and outside the IDSRN.

In a third system, the IDSRN director runs the health system and views this function as a

signal to staff that the work is important. In the fourth of the five delivery-based systems, the project head is also a department head in the medical school, who reports to the head of the hospital. These pre-existing relationships and roles expedite communication. We heard that while formal meetings are part of an organization’s operations, the informal communications tend to be important as well. Therefore, looking only at time the IDSRN staff meet formally with system leadership reveals only part of the relationship.

Despite these channels, however, there are many barriers to using them to move research findings to practice. In some cases, the research may not be operationally relevant. In other cases, size presents a problem. United Healthcare, for example, has over 100 medical directors throughout its divisions, making information-sharing across units a formidable undertaking. Relevance and size aside, there may be other barriers to change. For example:

• One system leader noted that staff are overwhelmed by many requirements and constrained by limited funds and information technology. These circumstances constrain how they approach implementing change. If a product is to be used, the process for doing so needs to be broken down into steps. There must also be a business case to support adoption.

• In another system, an executive pointed out that getting a process or a product adopted means not only to “find the answer—what to do now” but to determine how to get systems to “buy into it.” The approach to change in this system is incremental because of concern for adverse consequences. Management and other staff are very concerned with local adaptability, so they like being involved as a test site. Within their organization, some units are also better equipped than others to use research because of the characteristics of their leaders. Thus, in this organization’s view, translating research into practice is possible if “the major benefit of the project to the facility is identified, clinical champions exist, the change has a minimal impact on physicians, and credible data in useful formats are available.” Both the scientific and business cases need to be made.

• In a third system, a senior executive pointed out that feasibility and cost are major barriers to implementation. If the benefit is not that clear or compelling, it is difficult to make the case for change. As they see it, the biggest barrier is that the large volume of work to be done makes it necessary to set priorities, which, although useful in some ways, may postpone some worthwhile actions. A cultural change must also

56

take place in terms of getting people to do something in ways other than what they are accustomed to. This, in turn, means overcoming inertia.

Other than these barriers, the element that needs to be acknowledged is serendipity. Weill’s work on bioterrorism has been described as “being in the right place at the right time with the right people.” The fact that a young ambitious researcher was involved in bioterrorism work and took advantage of each opportunity as it came along was also critical to developing what is generally regarded as a successful series of projects in this area under the IDSRN.

Some system-based researchers also expressed caution about trying to conduct research that is too heavily focused on immediate utility in the system. The original leaders of CHCPE said that they thought it important to convince management that there was a good business reason to support the center’s work in public policy that drew on their system base rather than to take on narrower projects that might have more immediate business appeal but that could also raise proprietary issues that might be limiting. In addition, producing results quickly enough for management might conflict with good research by forcing the work into a tight schedule (e.g., answers in a month). On the other hand, bringing results “to scale” might be even more important to AHRQ, they suggested, than finding specific applications for research within systems. As discussed later, some participants see some IDSRN work as idiosyncratic (i.e., unique to one system), thus diminishing its value in a broader setting. Redundancy, they believe, would make system executives more confident in applying findings because it would provide evidence that an intervention would work in different settings. They also questioned the public benefit of funding work that they see as consulting work, which tends to work to the benefit of a single system.

Applying Findings Throughout the IDSRN. The IDSRN is also built on the principle that

the partners might learn from one another and, through these relationships, help to enhance dissemination and broader adoption of findings. To our knowledge, this end was generally not achieved over the course of the program despite a few task orders involving multiple organizations. We were told that each team found it sufficiently challenging just to keep internal communications moving. Task order funding was not judged as being sufficient to support cross-system work even when such work might be useful in bringing the results to “scale.” In teams that include multiple systems, communications seemed stronger. The HMO Research Network, for example, has a history of regularly scheduled conferences to facilitate communication, and RTI seems to be trying to use that model with a partnership grant. But even within individual teams, funding typically limited the ability to support collaborative work across systems. This disappointed one system collaborator who hoped that participation would make merging data across systems more feasible.

IDSRN partner/collaborator teams do not necessarily view one another as the most important reference group with whom to share learning. For instance, the HMO Research Network involves systems with a number of unique properties. The issues relevant to Marshfield, because of its rural location and group practice setting, are different from the issues relevant to the large, diversified urban systems. Even organizations seemingly similar make distinctions between themselves that diminishes the value of that system as the “go-to” entity. For example, both Denver Health and Weill/New York Presbyterian have safety net responsibilities, yet when they tried to work together, they found that their systems really are

57

quite different, in both length of stay rates and data systems. In addition, the competitive nature of the task order process does not necessarily support cross-partner cooperation, especially because the small size of task orders makes it difficult to support multiple-site interventions. In some areas, like bioterrorism, the IDSRN is not the core audience so sharing is more limited.

Given the diversity of people and organizations participating in the IDSRN, it is perhaps not surprising that the partners are not necessarily viewed as the most important players from whom to learn. As one partner said, they did not view the IDSRN as having very much of a program identity and saw it more as “putting together nine groups.” In several interviews, the partners and their collaborators said they did not have a good idea about who else participated in the IDSRN or particularly about what work they accomplished.

The IDSRN’s infrastructure appears to reflect an interest in encouraging cross-system

communication, though it is limited in scale and scope. Two main structures are designed to support communication across the system as a whole—the quarterly newsletter and the annual meeting. Of the two, the annual meeting is most visible to participants, and many of those we interviewed attended at least one such meeting. While opinions on the meetings and their importance varies, most people said that their knowledge of the other IDSRN participants grew as a result of having attended the meeting. But the meeting participants varied over the years, and there were few attendees from each team. As a result, some collaborators in large systems may not be very involved in or knowledgeable about the IDSRN as a whole. Further, the staff attending the meeting did not always include the kinds of people within the IDSRN organization who would be most effective in encouraging their systems to adopt change based on the program. While AHRQ has made an effort in recent years to include some system leaders, the meetings are typically a more appropriate forum for involved IDSRN participants and AHRQ staff. In addition, some attendees say that, historically, much of the work presented at the meetings has been weak, so it is not clear whether participants believe that there is something there to apply. In any case, delivery system-based staff noted that results needed to be translated into tools or other utilitarian vehicles before they were ready for application. Many of the task orders were not focused on work that went that far.

Applying Findings Outside the IDSRN. There do not appear to be any formal vehicles inherent in the IDSRN that would encourage communication outside of the IDSRN (e.g., external newsletters with substantive reports of findings). Moreover, the responsibility for dissemination does not seem to fall systematically on AHRQ or the task order team, but varies instead by project on an ad hoc basis. The most direct support within the IDSRN for dissemination beyond the program is through the specification of deliverables such that they are oriented to an outside audience and, through help from AHRQ’s co-funders, useful in forging links outside the program.

Other than its regular advisory processes, AHRQ has no formal way of gaining input from a

broad cross-section of potential users of program output on what they would find useful. In recent years, AHRQ’s IDSRN project officer has encouraged partners to share ideas for projects, and the partners and collaborators noted that they very much appreciated this change in interaction. However, it is not clear that this solicitation provides an effective vehicle for obtaining input from end users, since such communication is mediated by the IDSRN’s base which, as we noted, is in research despite its practical focus. At best, the sharing of ideas

58

between partners provides a vehicle for obtaining input from IDSRN systems, not necessarily from the full set of potential users who might ultimately benefit from the work.

Whether IDSRN results become known outside the program appears to depend heavily on

existing mechanisms of communication confined to the partners and collaborators, and those within AHRQ. Within the agency, there appear to be few formal processes for communicating IDSRN findings to others in the agency, although this may change as AHRQ moves to portfolio management. Historically, AHRQ’s work on user liaison (now including knowledge transfer) has provided a route through which findings in key areas can be shared with subsets of organizations within the user community. The collaboration with HHS on bioterrorism work has provided some opportunities to draw upon the CDC, although the ties between AHRQ and other agencies appear to be relatively weak.

Communicating the findings externally depends very much on informal processes, that is, on the connections and interests of the AHRQ staff involved in the IDSRN and the investigators working on a given task order. Theory suggests that opinion leaders are important to effective communication and adoption. Because the IDSRN is based on respected systems, the administrative and clinical executives in these systems are probably involved in ways that would support disseminating work outside the IDSRN and getting it adopted. However it is not clear that IDSRN work typically rises to a level that it is relevant enough to one of those executives that they are likely to disseminate it. With the exception of Denver Health, for example, IDSRN participants have more connections in the research community than in the operational one.

Funding also constrains the ability to take advantage of the links to a potentially interested

audience. We heard, for example, that some investigators involved in particular IDSRN task orders had been offered opportunities to speak at meetings via contacts that we believe probably came from AHRQ or other funders. However, IDSRN participants said that the number of these invitations they could accept was constrained by the lack of funding available to support the costs of attendance and their time.

The ability to raise an awareness of IDSRN results outside the IDSRN is also limited by the

team structure. Some teams involve entities that naturally provide access to broader dissemination. Examples are the HMO Research Network with its member systems and the University of Minnesota with medical groups via their MGMA link. Task orders awarded to other teams have resulted in other connections that could support dissemination, particularly the bioterrorism work and its relevance to a variety of organizations with links to the public health community. For the most part, however, the teams are based in individual systems or groups of systems. In this context, IDSRN participants may have an interest in getting work out, but the ability to do so depends on serendipity and individual initiative more than anything else.

Because research tends to be cumulative, AHRQ faces the decision about whether it should

claim “credit” for research it produces, or will doing so seem self-serving and myopic? The latter is a risk that AHRQ runs given that the research generally builds upon multiple contributions that probably lie well beyond not only IDSRN support but also that of AHRQ in general. Further, AHRQ’s limited budget will force the agency to make a tradeoff between how much of its resources to put into dissemination and translation at the expense of building original knowledge, or vice versa.

59

D. IDSRN: PERCEPTIONS VERSUS EXPECTATIONS

1. IDSRN Partners/Collaborators

In sum, the IDSRN attracted a set of organizations and people that are both similar to and different from one another. All have an interest in operations and are more involved in applied research than traditional academic researchers. On the other hand, they represent different points on the applied research continuum in terms of (1) their interest in the “big picture” versus “specific local needs” and (2) how comfortable they are with research that is less generalizable or, in some cases, with what some might characterize more as analysis than research in that the work does not rest on an evidence base and was not necessarily designed to contribute to the development of the knowledge base.

IDSRN partners/collaborators said they did not have a strong sense of what to expect at the

beginning of the collaboration but that they feel relatively satisfied with their roles. Large and more experienced groups expressed a bit more disappointment about the resources being, in their minds, very limited. The emerging work in some areas, especially bioterrorism, was a surprise for many. Although some perceived this work as both fitting well with what they do and presenting opportunities, others perceived that they were not very well positioned for this work.

The partners/collaborators generally see applied research as the centerpiece of the IDSRN,

which is what attracted them to the program to begin with. But just how this key element and the program overall is viewed differs across teams. For those not based in operational settings, the IDSRN is an opportunity not only to learn generally how research is applied but also to understand more precisely where research fits in and how to enhance its use. In contrast, Geisinger, Abt’s collaborator, sees the IDSRN as an opportunity to learn more about the effect of an electronic medical record (which they use) on health outcomes. The University of Minnesota and many of its key collaborators see the IDSRN as a means to improve data on medical groups and apply the results of an analysis of these data to efforts intended to improve medical group administration across the country. For the HMO Research Network, the interest is more general: to improve the health of members and work together on that topic.

IDSRN partners also look outside the IDSRN for gains through the IDSRN. Even if the partners looked for results that would be applicable within their system, many of them hope that participation in the IDSRN will enhance their credibility and prestige, both generally and in the research community. In their minds, participation has a market value—that it is something akin to a commodity they could use in responding to clients (like purchasers of care who might appreciate, for example, that they were investigating racial or ethnic disparities) or to attract other resources and opportunities. For many outside researchers, the ability to secure funds to study health systems is important, especially giving the few alternative sources of support. IDSRN participants do not necessarily see the audience as being made up exclusively of systems. For example, the original leadership of the team at CHCPE viewed the IDSRN as an important vehicle for developing data from systems that could support analysis of what they termed “mega trends” in health care delivery. Their interest, they said, was less in meeting immediate operational needs than in helping generate analysis from operations that would be of strategic interest, both to public policymakers and leaders in private systems.

60

As discussed later in more depth, clarity on what it means to turn “research into practice” does not exist at AHRQ and, even if it did, IDSRN partners and collaborators hold many different views on this issue. Some IDSRN partners/collaborators see the IDSRN as a natural “laboratory” that makes room for real world data. By that, they seem to mean that the IDSRN is valuable in providing a way to embed research in systems so that policymakers, managers, and practitioners generally have access to information that provides more sensitive measures of the way these systems work. While findings from an operational laboratory could be useful for practice, the intent was less to immediately address an operational question or need than to develop work that would ultimately improve the health care system and health care. Others, in contrast, view the IDSRN more as a vehicle for “pushing results out into the real world.” They seek more immediate application for the work via a “rapid cycle” process that would be beneficial for health plans and delivery systems. But some also cautioned that the real value of such work lies in its ability to move findings “to scale,” something that could conflict with the business time frame in which answers are typically sought. They also see as prominent the question of how to handle the fact that evidence might be lacking for important applied or operational questions. They asked, “What is ‘good enough’ research?” and “How do needs relate to a body of work, not just a single study?”

2. Congruence Between IDSRN and AHRQ Views

Perceptions of the IDSRN within AHRQ are inconsistent. The EMM characterized the IDSRN diversely as providing quick studies in non-academic settings where “the rubber hits the road;” a potential test bed to see if something that appears promising in research can work in the real world; a “learning lab” to get real input from real people in “real-ish time;” a vehicle through which research can better account for system characteristics; an opportunity to grow “centers for excellence;” and a way to complete cutting edge projects quickly. Thus, although all AHRQ management is supportive of the IDSRN and buys into the goals of translating research to practice, we found no consensus on what this means.

As we see it, there is one important difference between how AHRQ staff see the IDSRN and how the partners/collaborators see it. AHRQ staff tend to describe the program in global or conceptual terms rather than in concrete terms. We are not clear on the AHRQ staff’s view of translating concepts to reality, particularly through specific operational program components that would be consistent with that view. Their global focus on concept seemed to us to differ fundamentally from that of the IDSRN partners/collaborators, who might also care about the big picture but in reality are much more focused on what the IDSRN means for them and on the concrete opportunities made available through task orders or other funding.

How best to translate a concept to “on the ground work” and the focus of the IDSRN are

issues worthy of consideration.

61

E. PARTICIPANTS’ ASSESSMENT OF THE IDSRN AND SUGGESTIONS FOR THE FUTURE

1. Overall View of the Program

IDSRN participants tend to view the IDSRN favorably and think that it has allowed them to conduct work that would not have been possible without the program. In the words of some participants: “They’ve done five projects that wouldn’t have happened without the IDSRN” or “It was good money” or “We’re working with really neat people and have created strong relationships. It’s a wonderful opportunity.” For entities having less experience with soft money or research, the IDSRN has been especially important. Both Denver Health and Geisinger, for example, said the IDSRN has allowed them to build their research capacity within the system, an important goal for them. More broadly, several participants noted the potential in the IDSRN.

Some are particularly enthusiastic about the potential for the IDSRN to apply research to practice. For example, one partner said:

It feels like the IDSRN is on the verge of making an enormous contribution to translating research into practice. People have been giving lip service to this for a long time. Before AHRQ there was no one that has really understood how different this is. . . AHRQ is leading the charge on this. They are right at the edge. It’s messy now. It doesn’t always go the way you want, but that’s what it means to be on the cutting edge. The IDSRN is a great model for trying to do this.

Echoing this theme, another participant said, “The IDSRN is at an early stage relative to its potential,” while another said, “We are at the stage where we can go to the next level, to address the culture of medical groups, medicine, and research organizations as one.”

But not all participants are this enthusiastic even though they perceive the potential in the IDSRN and support its goals. One interviewee, for example, said that she would sometimes see a task order and say ‘I wonder what’s driving this’, with some task orders seeming like “lobs over the fence, for example on ‘safety’ leaving folks to twist things to make it fit.” Another observed, “Rapid cycle is good, but they need to know how leaders and mid-level managers operate and what influences them, how research fits in. We need a national research agenda.” One interviewee put both perspectives together.

It’s so exciting that someone heard us talking about the relevance of research in practice—that’s the strategy emanating from the IDSRN. IDSRN isn’t really providing funding support, but philosophical support. It’s important to see this evolve. But it’s not just about IDSRN funding, but about agenda setting and how medical research should evolve. Traditional research culture is not set up to do relevant research. You have to think about cultural changes—what you have to do to make a difference.

Few of those we interviewed suggested major restructuring of the program, although they had some suggestions for marginal improvements and some concerns about particular aspects of

62

the program. In part, this response probably reflects the interest of many participants in using the IDSRN to pursue work and individual projects they believe are important in ways that are consistent with the IDSRN. Many of those we interviewed do not, in our mind, want to “rock the boat.” Through experience they have learned to work within the IDSRN and would like to continue to benefit from this experience. Major changes in the IDSRN could dilute the value of the knowledge they have gained and change the nature of opportunities available to them, which limit funding opportunities. As a result (and perhaps because they came at the end of our interview), the responses to our questions about overall program performance and suggested change tended to be limited and focused mainly on IDSRN mechanics—that is, administrative changes that would make it easier for them. However, some interviewees recommended larger changes, and others raised concerns that could, taken to their logical conclusion, promote larger change as well. We review below the dominant “big picture” and more specific kinds of issues or recommendations that emerged from our interviews with IDSRN partner/collaborator teams and those in the organizations within which the teams are embedded.

2. Issues for Future Attention: “The Big Picture”

Four main themes appear to dominate the “big picture” concerns of the IDSRN participants. The first involves money and the size of task orders relative to the IDSRN’s objectives and the scope of work involved in the task order. The second involves the lack of a clear identity for the program and the diversity of topics in the task orders. The third involves the lack of continuity of topical interest on the part of AHRQ. The fourth involves the absence of funding to support the “infrastructure and dissemination” objectives of the IDSRN.

Task Order Size in Relation to Scope of Work. The view that AHRQ task orders are small, both in relation to other funding sources familiar to IDSRN partners and especially in relation to the scope of work, was raised in most interviews. One project director characterized the situation as “champagne ideas on a beer budget.” Another said that AHRQ is getting “a lot for its money.”

Some IDSRN partners said that they have subsidized (some heavily) the low budgets. Some

delivery systems, for example, said that they are making in-kind contributions, such as underwriting the costs of data collection or project management. One system sets an internal limit on how much support it can give, and it has met this limit in all but one task order but said it was able to do this because of a familiarity with the data. In carrying out some task orders, some interviewees reported that the actual costs exceeded the contract budget. For example, the organization doing the work to develop culturally and linguistically appropriate guidelines said that it lost a lot on the project. But because they believe in the project, a few investigators are putting in time that never is billed in order to support their network. We did not learn what, other than individual interest, motivated these subsidies, though we expect that it is some combination of good will, desire to maintain good relationships with AHRQ, fear of lost opportunities, and self-interest. The magnitude of the subsidies, however, appears to be limited by how much organizations feel they could afford, and how much individuals are willing to contribute in the way of time.

Experience differs across task orders and with the situation of the individual partners.

Those who are more expansive and flexible in their interpretation of the scope of work may have been more likely to encounter problems. Partners and collaborators that need to contribute to

63

overhead (which includes all nondelivery based entities and some based on delivery systems) have, in effect, fewer funds to support direct costs than it would appear from the total budget. This is most obvious to those more familiar with RO1 grants (where overhead is added as an additional cost after award) and to those who once benefited from subsidized overhead and who are now expected to cover those costs.

In addition, while many appreciate the quick-turnaround nature of the task orders, there is also a concern that the schedule is not always realistic. For example, one very operationally oriented IDSRN leader observed:

“Some of the task orders may need to be longer; we know the tradeoff with quick-turnaround but 12 months is a little tight, especially if you need to hire new people or create new relationships. Also the IRB takes time.”

Additional comments reflecting these concerns were made in interviews with multiple and very different teams:

“If I was to change one thing, I would tell them to increase the award amounts and increase the duration of the projects….” “There is not enough funding…The amount of money was not at a level to justify the full project. It is hard to attract attention to this project due to low funding even though it has a huge amount of potential.” “Funding levels are very, very tight. It’s champagne ideas with a beer budget. But that’s AHRQ for you. The one (task order) I was involved in gave a lot of in kind services.” “You need longer projects with better dollars.” “There is not as much funding as we thought though there are exciting opportunities. IDSRN doesn’t lend itself to multiple site initiatives [because there is not enough money.]” “The dollar volume is pretty low. Most of the kind of work we do involves big data volume and patient volume. There are intrinsic costs associated with this type of work—you have to hire programmers and they don’t come cheap. It’s a little hard to work within the resource constraints. This is a bit of a rub with IDSRN. The small dollar volume has some downsides on attractiveness, feasibility, and fit.” “It is a small amount of money, less once overhead is out. . . . We are able to do it because our collaborators have been good citizens and given us things for less than they cost (e.g., in kind resources, computer time.)” “We struggle with this (loss of money). We want to be part of the IDSRN, but we have to be careful that participation is not putting us in financial jeopardy.”

64

These concerns, particularly with regard to AHRQ’s expectations about the products that could be produced for the size of the projects, are important and widely shared across the IDSRN partners and collaborators even though some appear to be in a better position than others to handle small budgets.

Lack of Program Identity and a Clear Task Order Focus. None of the partners appeared to perceive the IDSRN as having a solid identity, and many commented on the diffuse nature of the task orders that have been issued under the program. One interviewee observed, for example, that “AHRQ is not sure what the IDSRN is all about, and there is not buy-in to this concept. They are under a lot of pressure for translation and adoption, but they’ve got a ways to go.” Another said “I would change the name of the IDSRN. Nobody recognizes it. It isn’t well known. There hasn’t been any branding process.” One interviewee speculated that AHRQ’s failure to develop a unique identity for the IDSRN is connected to the fact that the agency has been concerned about the possibility that outsiders might see the IDSRN as a “welfare system” for the nine entities. From the partners’ perspective, few outside the IDSRN have any idea of what they have done. Also from their perspective, a clear program identity could make their work more visible and more credible.

The diversity in task orders and changes in the topics funded over time have contributed to

the difficulty in establishing a program identity even though the addition of certain topics has brought more dollars to the program. To complicate matters, the IDSRN program infrastructure also has done little to support a coherent program identity in that it is not set up to make the most of task order findings. For example, one participant perceives that:

“The way IDSRN is structured they want a quick idea, want to have an answer, and then move onto the next thing. It’s a bit odd because the research question comes out of the depth of AHRQ. They are interested in it initially, and then we provide the data, but then there are no clear policy implications. It’s hard to be helpful if we don’t know the use. The bioterrorism and safety push got racial disparities pushed aside.”

Another interviewee observed:

“You can’t keep tossing out a seemingly random set of research questions that people have to respond to in a month.”

Several interviewees commented on the limiting effects of task order diversity on outcomes, especially when dollars are limited:

“In addition to the lack of funding, the effect of IDSRN on translating research to practice is lessened by the number of fairly diverse topics which are attacking no cohesive single minded direction . . . (you) can’t tell me what this is—the investment in a lot of things minimizes input.”

65

“There is the potential to be a larger program, to get other funders to become partners in this. It’s hard political work to pick up on the people who are funding this work and develop a mechanism to partner…It feels like the feds are not coordinated in their approach to different topic areas.”

As noted before, the issue of task order identity is an important one many interviewees commented on. Most see the IDSRN via its task orders. Therefore task order content and IDSRN identity are highly intertwined. Fragmentation in the focus of task order topics and lack of continuity in work were therefore of concern to those interested in IDSRN identity. For example, comments included:

“We expected follow-up on tools and also seeing how they work. In general, they are pretty small projects that don’t stand on their own.” “But the purpose should be clarified. Is it transfer of research knowledge to the field? Is it working with the field to get them to do research? Is it to do some low-grade consulting? The task orders also are all over the place, and the focus changes over time. There has to be some continuity. Teams should have to outline a three-year stream of research that will take you somewhere, from here to there. It could be the same amount of money but more focused. The task orders look almost random. They are not streams of projects.” “The funding mechanism is not here for replication by others. . . . Initial projects are “loss leaders” though they are doing a better job of building on them.” “Focus it more. Find centers for excellence that line up with AHRQ’s interests and have it become a place to go for this research. . . . If you want to get things implemented, you need to build on prior work (which means you can’t always be addressing new topics).”

Continuity and Scale of Work in Particular Areas. Many of those individuals we interviewed sense that translating research to practice requires a long-term perspective that extends beyond single task orders. One of the IDSRN’s biggest critics argued that:

“The IDSRN is . . . a rush to disseminate ill formed ideas. It represents a major paradigm shift from health services research before—the idea that you’d do good research and accumulate (a body of research). . . . [In contrast the IDSRN is] largely descriptive research, leading to invalid conclusions that people rush out to implement. The partners don’t appreciate the limitations of their research. It’s two cultures.”

While this observation could be characterized as a “hard core traditionalist” research perspective and therefore dismissed, similar concerns were raised by those firmly based in the translation camp and having a long history of applied research. For example, established practice-based researchers in the IDSRN who are strongly committed to the program commented

66

on their disappointment that the IDSRN is about single-site interventions with no provision for follow-up on pilot projects through which tools could be applied more broadly in practical settings to see how they work. Although these researchers believe in the value of working directly in delivery systems because such work is easier to integrate in a practical setting, the applicability of findings will continue to be limited unless they are taken to the next level.

In another system that also has a long history of using research, an executive expressed

concern that the limited resources of the IDSRN underscores the importance of focusing on task orders that are strategic. He said that when he looks at moving research into practice, beyond the general methods issues, the secondary questions that come into play are: Is it transferable/translatable to clinical scenarios? Is it replicable? Because resources are constrained, he believes that the research findings must be widely relevant and that the work should have “clinical champions.” Redundancy can make the case even better. In the future, he said, “AHRQ needs more continuity across projects and more predictability” within the IDSRN.

To address these concerns, one IDSRN participant suggested, for example, that instead of

focusing on task orders, the program should work around three-year streams of research in which the partners pursue a related stream of task orders designed to generate movement from point A to point B. Another suggested creating centers for excellence, which would work with AHRQ on an ongoing basis on a body of work that is congruent with AHRQ’s priorities and the participating organization’s capabilities. These concerns are worthy of consideration.

The suggestions also raise critical issues about resource allocation at AHRQ. With a fixed

pot of money, the more focus and continuity, the less the opportunity to spread funding around or to have the flexibility to take on new topics or respond to emerging needs. On the one hand, it may be more effective for purposes of dissemination and impact to focus on developing a body of work that has broad application versus attempting to move research to practice by expecting each task order to rest on its own, since that could give idiosyncratic needs or transitory questions center stage. But focusing on fewer areas for a longer period also makes the stakes higher and the risks of failure greater. Further, unless work is targeted to areas that are already close to being “ready for use,” AHRQ may end up with very little to show in the short run in the way of concrete applications.

Funding for Infrastructure. IDSRN partner/collaborator teams are funded solely through

individual task orders, as described earlier. In addition, AHRQ’s resources are limited in the way of dedicated staff and fiscal resources that could further the objectives of the program. As noted, only one person runs the program, and while the budget can support a newsletter and an annual meeting, the task order process appears to be the only vehicle to support partner/collaborator efforts designed to move research into practice. In addition, limited resources also prevent IDSRN leadership from visiting with the IDSRN teams and their associated delivery systems, depriving them of the opportunity to become more knowledgeable about team needs and operating processes.

In our interviews, the IDSRN partners/collaborators raised three main infrastructure-related

problems, though they did not necessarily categorize them in this way. First, partners with multiple collaborators found that the management task of

communicating within the team to keep people up to date and to coordinate responses to RFPs

67

was a substantial effort. They have had to subsidize some of the costs of this effort and have been unable, unless they had other vehicles to support them, to do very much with their collaborators beyond what was called for in the scope of individual task order. The level of knowledge about the IDSRN and its work among collaborators is quite varied, even among those who are actively involved in task orders. Undoubtedly, this reflects, at least in large part, the lack of funding and external incentives for partners to spend time bringing their collaborators “into the fold.”

Second, as already described, funding for dissemination and translation work built on

individual task orders is often not available. Follow-on task orders have helped to address this concern, but the issue of how best to encourage and support broader dissemination of the task order results and experience remains an issue for the IDSRN.

Third, the current administrative success of the program appears to depend heavily on the

person who now runs it (Cynthia Palmer), although her resources for this task are limited. While other AHRQ staff share in the monitoring of task orders, some staff are not interested in these duties, and fiscal as well as staff support for work needed by the IDSRN outside of individual task orders is limited.

Those we interviewed believe that the IDSRN was running much more smoothly under its

current project officer than it had in the past. As one staff person noted, “Cynthia is A+++. She is queen of the IDSRN. We often call her. She is very straightforward, very savvy, and a great facilitator.” We heard many positive comments about improvements in the IDSRN infrastructure in the past few years, including more advance warning and better spacing on task orders, more concentrated efforts to solicit input from IDSRN partners/collaborators on what they want, the potential for small follow-on task orders to support the translation of work into forms that are more readily accessible to users, and better communication. But there are still shortcomings in how these tasks are handled. Furthermore, it is probably not healthy for an organization to depend so exclusively on one person for the success of a key initiative.

3. Issues Needing Future Attention—Narrower Concerns

Partners and their collaborators made very concrete comments about how to improve the program, citing specific changes that would make it easier for them. While the issue of the IDSRN’s administrative infrastructure will be considered in more depth in Chapter VII, we discuss here some of the areas that produced the most commentary.

Master Application Requirements. If IDSRN participants commented on the initial

application process at all, they noted that it was very demanding, particularly with regard to demonstrating data capabilities and, for some, with regard to developing the full constellation of collaborators and consultants. In discussions with one delivery system serving as a partner, we heard that the process of showing the quality and range of data available to support the system took two months and that the five resulting volumes cost $2,000 to copy. In one very large organization not generally focused on research, getting support for the high up-front costs required to apply was a challenge and the requirements to show access to data raised proprietary concerns that had to be addressed. Proposals spanning more than one system obviously had the additional burden of integrating multiple system responses. As one participant that included multiple networks said of the application process, it was “rough” and “not much fun,” though not

68

particularly unpleasant, particularly given the support received from all the other organizations on the team. While the application process was burdensome, IDSRN participants said that they later recognized later why it was important to the IDSRN to have them demonstrate their capabilities to generate data. With the work they already had done, they perceived that they would be in a better position to respond next time.

Individual Task Orders. As noted, IDSRN partners/collaborators want as much advance

communication as possible on the types of task orders that will be released, and they want the work spaced to avoid a rush at the end of the fiscal year. They appreciate the improvements made to date, although our interviews preceded FY 2004 procurements, so we do not know how they view the most recent year’s experience. Generally, they wanted more time to respond to task orders if possible; having the time is especially important to teams comprising multiple organizations. IDSRN partners very much like the quick-turnaround on awards, however, seeing that as a worthwhile trade-off for the short proposal schedule, and they would probably be substantially more unhappy with the program if it involved a protracted awards process. No one we spoke with described the application process for individual task orders (versus the master IDSRN agreement) as burdensome. In general, we too had a sense from the responses that participants generally thought that their efforts were rewarded with what they saw as a high probability of success, especially as they became more experienced in working with AHRQ.

IDSRN participants had mixed feedback on the content of individual task orders. In general,

they like the open-ended nature of the task orders, since it gives them the freedom to shape their response, but they would welcome more of a chance to provide input on the topics. In some cases, partners were concerned that the ambiguity of the task orders led applicants to promise more than they could realistically provide, and more consideration given to this issue by AHRQ would have been appealing to them. They also believe that AHRQ could do a better job of clarifying its expectations in the RFTO. For example, one participant perceived that AHRQ did not make it explicit that work was expected to be translated into practice, and that if that is AHRQ’s intent, then the agency should say so. Similarly, because task orders encourage competition among teams, AHRQ should make it clear if it wants to encourage collaboration among partners—regardless of whether limited funding may make it difficult to achieve collaboration.

Contract and Monitoring Issues. Interviewees had a variety of different experiences with

their AHRQ technical staff assigned to monitor specific task orders but few expressed substantive issues. For the most part, those we interviewed focused their comments by taking issue with reporting requirements and with the way contracts staff interpreted requirements. Several expressed concern that reporting monthly on progress was burdensome and not necessarily of value for these kinds of projects; they feel that they have been micromanaged. Over time, many said they learned better how to interact with the contracts staff and felt that the process was going more smoothly.

A frequently cited and continuing concern appears to be flexibility (or lack of it) in the

contracts office. IDSRN partners want the office to be more sensitive to the fact that there are sometimes reasons for delays that are outside their control. They also want to be able to get extensions for the work more easily and to use any remaining funds in the task order contract to support dissemination. Concerns over contract flexibility appear to go beyond lack of prior experience with contracts to the way that AHRQ interprets the contract vehicle.

69

A few partners also took issue with the minority subcontracting requirement because they feel not only that it is unnecessary but also that it does not work well in these kinds of collaborations. For example, a safety net IDSRN partner believes that its entire system is focused on minority issues, and another partner who made such an arrangement felt sorry that the collaborator spent a lot of time on the proposal but ultimately got no work because the task orders did not use their skills.

In sum, IDSRN partners and collaborators feel that the IDSRN has matured and that its

administrative procedures are improving, but that these procedures and the broader goals and processes of the IDSRN could be better and clearer still.

71

VI. ASSESSMENT: IDSRN INFRASTRUCTURE TO SUPPORT EFFECTIVE TRANSLATION OF RESEARCH TO ACTION

This chapter reviews what has been learned in the first assessment area of the evaluation of the IDSRN, which is related to how the IDSRN is structured to effectively support moving research to action. Our work plan frames this area of assessment as follows:

To what extent is the program’s infrastructure designed to encourage “real world” adoption of findings from research? In considering the program renewal, what types of changes in the infrastructure might be considered to clarify program goals and promote moving “research to action” as an explicit goal?

The chapter begins by discussing what we view as the four components of IDSRN infrastructure needed for the IDSRN to work. We then review the findings on each of these dimensions and the areas that AHRQ may want to change or at least consider as it looks to the IDSRN’s future.

A. FOUR COMPONENTS ESSENTIAL TO THE IDSRN INFRASTRUCTURE

As explained in Chapter II, theory suggests that research is effectively translated into practice only when several elements are present:

• First, there is the innovation or finding itself; the topic it addresses; and the focus, products, and results of the task order. Projects that result in findings that are new and “actionable” in a broad range of settings are the most likely candidates for adoption.

• Second, these findings have to be made known to the appropriate audience by using formal and informal communication channels in a variety of ways. However, while this kind of dissemination may be essential to change, it is not sufficient.

• Third, potential “adopters” must find the innovation attractive.

• Finally, it must also have enough organizational and political support to advance its adoption and enough resources and other support to implement it.

These concepts from theory can be translated as guidance for the IDSRN. Specifically, they suggest that for the IDSRN to be successful, AHRQ needs to be clear on the kinds of results it wants to achieve, make sure these are meaningful to its target users, establish partners and collaborators who are positioned to make the necessary translations and reach the target audience, and use a well-defined process of priority setting to identify where best to focus its efforts.

72

We believe that there are at least four elements of IDSRN infrastructure that can be shaped to address these concerns and best support the IDSRN in translating research to action. These are: (1) clearly stated goals that support an effective strategy for translating research to practice in ways consistent with specific operational goals; (2) partners and collaborators who can advance/support goals and strategies; (3) appropriately scaled and focused work that moves research to practice in ways consistent with overall IDSRN goals and strategies; and (4) vehicles for making IDSRN findings known to the appropriate audiences and for taking project findings the next step closer to adoption and diffusion in practice. We review below the strengths and weaknesses of how the IDSRN is structured now on each of these elements and make recommendations about where improvements might be made in the future. The analysis builds on what we learned from interviewees and our assessment of task order outcomes, though the interpretation of such facts and our conclusions from them reflect our own judgments based on experience.

In brief, our analysis suggests that the IDSRN did not have a well developed infrastructure

for promoting diffusion. More fundamentally, the program lacks a clear vision that links what research has to offer and what practice seeks to learn. For the most part, traditional research seeks “general truths” based on an accumulation of evidence and in the course of doing so draws on a way of analytical thinking that helps to identify generalities from the specific. Practice, in contrast, seeks answers to very specific questions that typically require insight on dynamics that may cross multiple disciplines (e.g. organizational behavior, economic theory, operations research) and can be applied to specific settings on a “real time” basis. Arguably, the key challenge for IDSRN is to identify how best, with limited resources, research can be used to make a difference in practice.

To accomplish this end, AHRQ will need to be clearer on how it conceives of the concept of

translating research to action. Right now, within the IDSRN, many projects generate results that are of interest only in a single setting. More successful ones seem to be those that use the analytical powers of research to develop tools and models that can be applied in a variety of settings. These examples typically apply research’s power of analysis, not particular research findings per say, to real world problems. They can reflect appropriate views of application of research to practice if the intent is to incubate innovations but they may not necessarily lend themselves as easily to spread of evidence-based innovation.

An alternative or additive model would be to create a dialogue among practitioners and

researchers that can identify important issues in the medium term that classes of practitioners face and where research might be able to contribute to addressing these issues. If AHRQ could better identify areas of potential synergy between bodies of research work and practical needs, IDSRN potentially could have a broader impact than it does now on translating research to address practical needs.

B. CLARITY IN GOALS AND PRIORITIES

As discussed in Chapter V, translating research to action means different things to different people. Within AHRQ, top leadership appears to “buy in” to the concept, but what this means in practical terms for the agency generally and for agency expectations of the IDSRN specifically is not clear. Among IDSRN partners/collaborators, there is a similar sense that the IDSRN is something “out of the ordinary” and “cutting edge.” Its link to delivery systems and to the

73

concerns of those systems makes it valuable to AHRQ. But this general agreement masks what seems to be considerable ambiguity in and inconsistency around what it means to translate research into practice and what the role of the IDSRN should be in doing so. As a result, some formulations of success on the goal of translating research to action appear to be achieved quite well by the IDSRN as it is now and others, not so well at all.

Because AHRQ’s goals for the IDSRN are ambiguous, it is impossible to judge the extent to which the IDSRN has been successful in meeting them. Clarifying goals in operational terms, if possible within the constraints of the political environment in which the agency works, is important and particularly relevant given the limited resources available not just to IDSRN but to AHRQ as a whole or even to the HHS. For example, what kinds of work are not appropriate for the IDSRN, who sets the boundaries, and on what criteria are they set. To facilitate consideration of this issue, we review the operational definitions of what translating research to practice could mean for the IDSRN, where the IDSRN appears strong, the measures on which it appears weak, and, finally, our recommendations for change or issues for consideration.

1. Areas of Strength

First, the IDSRN has enabled AHRQ to move beyond the university-based health services research community toward a community of researchers whose interests may be in applied work as evidenced by their decision to conduct their research as part of or in affiliation with service delivery systems. Through these researchers, AHRQ also developed closer connections to leaders of organized systems who value research, having supported its development and that of the IDSRN from within their system. The most obvious and established examples of this strong point are the organizations represented within the HMO Research Network, but the IDSRN also has developed links outside of that group in the other organizations affiliated with it. These connections can help build political support for AHRQ. Moreover, drawing in researchers whose interests are more diverse can also pose a positive challenge to the culture within both AHRQ and the academic research community. Funding from the IDSRN also reinforces the research infrastructure now present in operational settings and can help to build it up even more.

Second, the IDSRN has enhanced AHRQ’s capacity to build research on “real world”

operational data that go beyond the secondary national data sets, which tend to be based on surveys (especially population surveys) or claims, typically from government programs. Of the 58 task orders, 50 made some use of systems data from either the partner or collaborator organization. And while systems data may not be as “clean” as conventional public data, and the research process somewhat “messier,” such work also probably brings more information on the system features that influence outcomes than do data that are more removed from operations. Research in real-world organizations yields insights into operational constraints that may influence applications of research findings, like organizational uniqueness in relation to generally stated applications.

Third, the IDSRN has encouraged AHRQ to collaborate with other entities in the department. The most prominent example relates to the work on bioterrorism, which was not even envisioned under the original IDSRN. While the collaboration with bioterrorism has not been without tensions, AHRQ’s ability to forge links with more operationally based partners has paved the way for it to reach users and created the potential for it to gain outside support for research that the agency views as vital to its needs.

74

Fourth, the IDSRN has made AHRQ more “nimble” in a desirable way. In comparison, AHRQ’s grant programs take a long time to process, are academic in nature, are run through study sections that may be less likely to support work on emerging topics that fall outside their bounds, and probably also consider most tool development to be outside the scope of health services research. With this new flexibility, AHRQ has been able to pursue new kinds of projects and kinds of work that bring more risk insofar as they may or may not ultimately prove successful. On the other hand, they may bring rewards as well.

And fifth, the IDSRN has prompted AHRQ to support work that has been used in a real-

world setting or that is targeted to such use.

2. Areas of Weakness

The main weakness we perceive in the IDSRN is that it is not clear what real world applications of research the IDSRN seeks to foster and how the IDSRN seeks to move from “research to practice” in concrete terms. To the extent that generating concrete change is intended rather than the other formulations of success in moving research to practice that we have already discussed, the IDSRN’s lack of specifying here is an important concern.

Lack of clarity to some extent reflects the complex nature of both research and practice and the limits of any program built around small task orders to bridge the two. Because research is cumulative, there rarely is a single seminal study that provides definitive evidence on a topic. Instead, knowledge accumulates and its findings need to be synthesized and some judgment made on which are worthy and ready to be translated. But while AHRQ has increased its emphasis on synthesizing research, there remain few formal processes that provide the IDSRN with a way to either identify the relevance of particular bodies of work for application or to disseminate them.

The complexity of practice organizations and types of real world needs they face for data to

support decision making similarly creates challenges in translating research to practice. Executives we interviewed noted that the most urgent matters they face often have a short time horizon (e.g., the product was needed “yesterday”). If the IDSRN is to support such urgent needs, the work will need to take place much faster than even the fastest researchers tend to work; it will also have to be fully customized to each individual system. Research on such topics may be challenging even with very relaxed standards of traditional “quality.” Being able to accumulate findings across such studies and extend them to other settings is also likely to be difficult to the extent that the work is tailored to short-term objectives in specific settings. Therefore, we are left with the question: When one speaks of applying research to practice settings that are large and complex and different from one another, what does it really mean?

To the extent that use of research in practice is to go beyond single setting applications, what

also appears to be missing from the IDSRN is some way of identifying what types of strategic applications of research findings are likely to be of interest across a diverse range of settings not just in one site. Right now, the IDSRN uses a “retail” approach to its work with individual systems.

The IDSRN’s impact on practice also is limited by the way funding currently is allocated. Currently the IDSRN is oriented to “breadth versus depth.” But limited resources mean that

75

trade-offs exist between concentrating work (1) in a few areas that might support multi-site applications or longitudinal progress or (2) in a broader set of smaller projects that could involve more players and cover more territory. Both diffuse and targeted strategies entail risks. The fewer the projects, the higher the stakes but the greater the probability of success for an individual project. But concentration also is likely to be more politically “charged” because it, in all probability, will concentrate funding in fewer institutions. AHRQ would benefit by greater clarity on which approach it prefers and why.

3. Recommendations

When we began this evaluation, we asked if the IDSRN was a “program” or a “funding mechanism.” At this point, we do not believe that the IDSRN is a distinct program because its work spans the totality of the agency’s interests and aims to build on a broad base of research connecting work to real-world applications. Yet, while the IDSRN’s focus is much broader than a typical program, the IDSRN also has content and can be much more than a mechanism for funding quick-turnaround projects. The IDSRN fosters connections, atypical of most other agency work, between researchers and executives in large delivery systems. The ability to fully exploit the potential that such connections bring, however, will depend on whether AHRQ develops a strategy that is both flexible and provides a roadmap to the IDSRN’s goals, priorities, and anticipated outcomes.

We recommend that AHRQ determine whether it is satisfied with the types of outcomes that

are strongly supported by the way the IDSRN functions now. The current structure provides a reasonable way of diversifying the field of health services research and linking it more closely to health care delivery, to public health, and to researchers who are more active in those areas. However, this structure also lacks the formalized processes that would help AHRQ to be more strategic. A clearer, more practical definition of what it means to translate research into practice and what the priorities should be in this regard is important to the future of the IDSRN and to AHRQ’s overall objectives as well.

If AHRQ desires the IDSRN to achieve generalizable change in practice, the current focus

of the program needs to be sharpened. Addressing operationally relevant questions within an intermediate time frame (one to three years) that are common to a number of settings could be one way to focus the IDSRN. The fact that questions are practitioner-initiated versus investigator-initiated could be one way to distinguish the IDSRN from other agency programs. We also recommend that AHRQ give serious consideration to establishing a formal vehicle for gaining strategic input from the user community on the kinds of questions that, in their minds, could benefit from analysis in the “medium term.”

A logical way to support this user input could be a meeting convened by AHRQ with some

of the leading thinkers in the organizations associated with current IDSRN participants, perhaps adding a few outsiders with a similar background as AHRQ advisors. Ground rules—set in advance—should define the kinds of input sought by AHRQ, the agency’s interpretation of the goal of translating research to policy, and any priorities it has established to provide a topical focus. A process of creative engagement of researchers in the process to identify targets of opportunity could provide a basis for two-way communication that would serve the IDSRN well.

76

Finally, we recommend that AHRQ review its planning process to determine the best way to meet the IDSRN’s need for input on the status of research in particular areas on a continuing and formal basis. We assume that having a mechanism that is consistent with the broader needs of the agency and its programs would be most effective and efficient.

C. PARTNER/COLLABORATOR TEAMS APPROPRIATE TO THOSE GOALS

The IDSRN seeks to develop teams that combine research skills with an operational focus. We conclude from our analysis of the IDSRN that it has been reasonably successful at structuring this element of the IDSRN. Our read of the IDSRN experience is that, using different models, all of the teams marry the skills of applied researchers to the operations in large delivery systems. The key issue is how to structure requirements for the recompete and how to evaluate bids such that the strongest teams and collaboration with AHRQ will be encouraged. The specific goals and priorities AHRQ identifies for the IDSRN in the future should have a large bearing on these requirements. However, the IDSRN’s experience provides some insight into the features it will be important to specify.

First, AHRQ’s emphasis in the initial procurement on a specified set of delivery systems that

meet certain requirements still seems to be the appropriate focus. The IDSRN experience suggests that the active involvement of those based in appropriate delivery settings is crucial to the IDSRN’s success. AHRQ will need to review its priority areas to determine whether other kinds of systems should be encouraged. Entities that allow the IDSRN to better address long-term care issues or public health infrastructure concerns are likely to be needed if these issues remain a priority for AHRQ. Participation requirements defined too loosely, however, would make it both difficult to distinguish the IDSRN from other programs and may contribute to lack of clarity in goals or priorities.

Second, existing IDSRN teams have invested on developing collaborations and

communication. Giving some priority to the teams that have a good record under the IDSRN would be valuable to both anchor new work and leverage the resources already invested.

Third, the IDSRN experience suggests that developing and maintaining specially formed systems is labor intensive. We would guess that including more than three to five collaborating delivery systems in a single team would be too unwieldy unless they are either already naturally connected (e.g., the HMO Research Network) or build upon existing affiliations (e.g., MGMA). However, AHRQ will need to decide whether this type of mediated collaboration (between a university-based researcher and an intermediary with provider connections) “fits” the priorities of the IDSRN. These collaborations have the potential to create leverage, but they also run the risk of diluting the emphasis on linking research more directly to practice.

Fourth, the issue of whether the lead partner needs to be a delivery-based entity is worth

considering. Many perceived that this would be the case when the IDSRN was established, although the successful teams ultimately included those linked to, but based outside of, systems as well. Where the appropriate delivery system exists and an affiliated research entity is strong, having that entity as the lead partner would seem to provide the strongest combination because it builds on established relationships and knowledge. But there are few systems that combine sufficient scale and research expertise. If AHRQ wants to open the IDSRN to a broader set of systems (as it probably does), it is likely to be necessary to involve other kinds of research-based

77

organizations. In our view, the most value is added when the lead partner outside the delivery system is willing to serve as the broker for three to five delivery systems. The success of these “mixed models” requires both trust between partners and collaborators (e.g., as might be established through a prior history) and a strong champion with the right skills. The system contact on the team should have enough senior standing to generate commitments and enough knowledge of what it takes to support good applied research that he or she can effectively mediate between research and operations.

Fifth, in structuring the RFP for the recompete, AHRQ may want to decide if it is interested in pursuing multi-site studies more than it has in the past. Such studies could lead to findings that have been replicated and may be more generalizable. If AHRQ wants to move in this direction, signaling such an interest could help potential applicants develop partnerships that would support such efforts, but the resources needed to develop them would not be a good investment if such projects are not likely to be funded. AHRQ, of course, always has the option to fund multiple partners to do the same work as an alternative way of accomplishing this goal.

Sixth, AHRQ will have to decide how many teams to accept. While there is an incentive

within AHRQ to provide maximum flexibility for attracting outside funds by contracting with as many organizations as possible either as separate teams or as collaborators on teams, we think the IDSRN experience to date shows that this approach could have some important disadvantages. Those participating in the IDSRN had to feel there was enough in it for them to make their participation worthwhile. They wanted both the opportunity to be involved in a number of projects and the knowledge that if they did develop a proposal, there would be a relatively high probability of success. Success rates are particularly valuable to two groups of people: those based in systems in which developing proposals is not a regular part of their business and those more familiar with grants who might view the greater certainty and timeliness of funding as an appropriate offset for work that had lower levels of funding, faster timelines, and less flexibility. If too many organizations are involved, these concerns are unlikely to be met.

Finally, AHRQ may want to consider asking applicants to qualify by area or sets of areas. We understand that AHRQ is considering tying substantive work under the IDSRN to particular portfolio areas or groups of portfolios with similar qualification requirements. Applicants could bid to be part of the IDSRN for all or any of these areas. Because of their experience in the IDSRN, many current teams would probably decide to compete in all or several of the areas. If the IDSRN needs vary across portfolio areas, the portfolio approach could provide a vehicle for ensuring sufficient participants of each type while keeping competition for task orders limited to a small number of organizations. The “area focus” also might move the IDSRN closer to being conceived as a “center of excellence” and make it easier to develop communication across teams that share a focus. This approach could also provide a way to develop more committed and long-term relationships for work in a given area while still including a variety of organizations. However, care is needed to ensure that the administrative costs do not become overly burdensome either for IDSRN partners/collaborators (especially if they handled multiple areas) or for AHRQ. Hence, AHRQ may want to limit the number of areas for separate competition in ways that accentuate true and important differences in the qualifications of contractors needed for each area.

78

D. TASK ORDER FOCUS AND STRATEGY

AHRQ’s use of the task order mechanism in the IDSRN seems to be appropriate to the kinds of projects that are funded via the IDSRN because it allows for fast turnaround awards in which performance can be monitored against the specific scope of work and set of products specified in the task order. However, we believe that the IDSRN experience shows the limitations of using task orders as the central unit of planning.

As indicated, a key gap in the IDSRN’s infrastructure seems to be the lack of a process to

encourage strategic thinking within AHRQ on task order awards and linkages across task orders and between diverse bodies of work. At a minimum, it would seem that the request within AHRQ for a particular task order should be required to justify how the work relates to translating research into practice, as this goal is ultimately defined. Those requesting support should not be made to argue that the work is immediately applicable—unless, of course, AHRQ decides to limit the IDSRN to this type of work. Rather, the justification should demonstrate why the question asked is or will be important operationally at some reasonable point in time for a reasonable segment of the user audience, what the current state of the research and the evidence is on this question, why the particular task order is needed now, and what outcomes and next steps might reasonably result from the task order.

The task order process could be improved in other ways, which are discussed in the next

chapter on the IDSRN’s administrative infrastructure within AHRQ.

E. UNDERLYING INFRASTRUCTURE TO SUPPORT ADOPTION AND DISSEMINATION

The strength of the IDSRN infrastructure lies in the commitment of its partner/collaborator teams and of the person now running the program at AHRQ. Its critical weaknesses stem from the fact that AHRQ has very little other formal infrastructure outside of its partners and task orders to support the movement of research to practice. Without some effort to further develop this infrastructure within the agency and potentially outside it as well, the IDSRN is likely to be limited in what it can or should be expected to accomplish.

Several priorities for improvement have already been noted. First to the extent implementation across systems is a desired outcome, a formal process is needed to obtain input on IDSRN priorities from a diverse set of executives involved in both administrative and clinical capacities in the kinds of systems that the IDSRN is targeting. AHRQ could begin by including leading “thinkers” associated with its current partners and collaborators and others, of similar background, already involved with the agency. People based in operations who are also knowledgeable about what research can and cannot do might be particularly useful, as they will be better attuned to the questions that are both important and feasible to answer. Such input could serve a secondary purpose as well by giving the executives associated with the IDSRN a larger stake in the success of the enterprise and allowing AHRQ staff to get their help and support in disseminating findings to user audiences they can best reach.

Second, the IDSRN is too important an initiative to be limited to the kind of support that can be provided by a single dedicated staff person, regardless of the strength of her commitment. The IDSRN project officer oversees many task orders and manages all the other aspects of the

79

program. Overseeing the strategic direction and performance of the IDSRN as a whole is a full-time or near-full-time job to gather input on emerging priorities, needs, or even external funding opportunities in addition to monitoring, guiding, and encouraging the dissemination and application of findings from task orders. This function is not compatible with managing large numbers of task orders. It is therefore important to free up the project officer by developing a stronger cadre of AHRQ-based task order monitors. Further, a staffer who can help with the more mundane administrative matters would also be valuable. In addition, access to a limited amount of flexible administrative funds would help to improve both the quality of products and the ability to track and analyze the IDSRN experience as a means to learn from the past.

Third, AHRQ might want to rethink its current allocation of funds with an eye toward using

a small share of them to give partners and collaborators more flexibility to fund dissemination efforts both within and beyond their systems. As discussed later, these functions are not necessarily inconsistent with a contract mentality, and there are ways in which they could be shaped to ensure that the money does not just become a “slush fund” but rather the means through which teams can move research toward a practice. Diffusion depends heavily on personal contact and trust. AHRQ would be well served by identifying how to support and leverage the position and communication channels of operational system leaders to better foster diffusion of innovation.

Fourth, the link between the IDSRN and the broader AHRQ structure must be stronger and more explicit. As noted, translating research into practice means building on the cumulative body of research. Toward this end, the agency does some synthesis already and probably needs to do more to be consistent with its overall goals. What the syntheses reveal about targets of IDSRN opportunity should be more effectively communicated and more easily accessible within the IDSRN if it is to shape task order work and review. AHRQ also needs to look beyond its internal resources to support the dissemination of the IDSRN experience and products.

81

VII. ASSESSMENT: IDSRN ADMINISTRATION

This chapter reviews what we learned about administration of the IDSRN—the second assessment area. Our workplan frames this area of assessment as follows:

What are the strengths and weaknesses in the way the IDSRN is administered? Are there ways in which administration could be improved? To what extent has the IDSRN’s focus on contract awards (versus grants) supported or impeded program goals?

The previous chapters have described the way the IDSRN is administered and how it operates. Because they correspond to different administrative decisions, we first summarize the issues related to overall placement, funding, and decision making with respect to the IDSRN. We then focus on more specific elements related to procurements (master agreements and task orders), task order monitoring and oversight, and overall IDSRN program operations, including annual meetings and other AHRQ mechanisms to support the IDSRN and to further its overall goals. In each area, we summarize what we learned from the interviews about the strengths and weaknesses of these processes and our recommendations for future improvements. We end the chapter with a discussion of the issue of contracts versus grants as it is relevant to the IDSRN. As in the previous chapter, our assessment is based on the findings from the interviews and the program review, but also takes advantage of our own experience to help interpret these findings and develop conclusions or recommendations for change.

A. OVERALL PLACEMENT, FUNDING AND DECISION MAKING

1. Present Structure and Processes

The IDSRN is located within one of the core centers of the AHRQ (CDOM), but the IDSRN is expected to serve needs throughout the agency. Historically, the IDSRN has had a limited amount of funding it can count on ($1.165 million per year). The total amount available to the IDSRN depends on agency decisions on how to spend general AHRQ funds and AHRQ’s earmarks on specifically funded initiatives (e.g., patient safety); and the decision of staffers within the agency who are able to commit particular kinds of funds with departmental approval (e.g., bioterrorism or long-term care). The AHRQ director makes such decisions, typically towards the end of the year when both the budget and existing commitments, especially for grants, are known. While AHRQ is, in general, a flat agency that is not very hierarchical and reflects its academic roots, AHRQ also uses a centralized decision-making process to approve certain actions. One of these is spending, with authority for decision making highly centralized. The EMM and AHRQ director must approve particular task order topics (though not specific awardees), and the contracts office must certify that funds are available to be released to support the award.

While the evaluation project scope did not include a comparative assessment of the

implications of the AHRQ structure (relatively “flat” with decision making very centralized), our sense is that the IDSRN project officer has relatively more constrained authority than do leads on

82

some other initiatives. We base this on the funding mechanism (contract versus grant, which means the IDSRN project officer cannot use the outside authority of experts or their scoring to make the case) and the funding process (each limited size task order decided separately versus authority for overall initiatives or strategic thrusts).

2. Strengths and Weaknesses

The way the IDSRN is used is well suited to certain aspects of the way AHRQ operates. The speed and flexibility of the IDSRN is particularly valuable given the delays typical in appropriating AHRQ’s budget and the uncertainty of the ultimate and any specific earmarks each year. Further, at a time when matrix management is becoming more prominent at AHRQ (particularly through the movement to portfolios), the IDSRN’s ability to provide critical connections with delivery systems that are relevant to a variety of areas is useful.

But the way funds are committed to the IDSRN makes it difficult to strategically plan

around the IDSRN, as the amount available is not known in advance, and decisions are made on a task order-by-task order and year–by-year basis. Further, in an environment in which total funds are limited, some are externally earmarked for specific purposes, and authority to commit funding is divided unevenly, the IDSRN project officer, as we perceive it, is placed—at least structurally—in a difficult position. Securing resources so the IDSRN can achieve its goals—particularly from the limited AHRQ pot versus from outside—without being regarded as “greedy”—is challenging.

3. Recommendations

Recommending changes in present arrangements is challenging, since AHRQ has only limited control of decisionmaking on its budget and the issues raised here are not unique to the IDSRN. It would seem important, however, for IDSRN staff to have more predictable ways of knowing how much money will be available for the IDSRN from year to year and knowing in general terms what the priorities are for uses of that money. As we already have noted, we recommend greater clarity on IDSRN goals and priorities as a way of better identifying how success will be defined and what priorities will be pursued. To the extent this occurs, it may make it easier to establish the resources available to the IDSRN more definitely and earlier. In addition, we perceive it is worth considering better ways of providing the weight of the AHRQ director’s authority behind the IDSRN administration so that (1) the IDSRN project officer is better positioned to follow through on goals and (2) so that the process of IDSRN funding and decision making does not come to be viewed as a tussle between centers or center directors.

A second issue relates to where the IDSRN is best placed within the agency, and how it

should relate to the evolving focus on portfolios. Right now the center directors are responsible for individual AHRQ initiatives and programs. Because the IDSRN is based around integrated delivery systems, its current placement in CDOM makes sense. However, it could be useful to tie decisions about the IDSRN more directly to planning for portfolios and other agency priority-setting processes so that more meaningful buy-in occurs agency-wide. For example, portfolio leaders could assume more direct shared responsibility with the IDSRN project officer for carrying through on desired agency priorities that fall in that area and, and they could assume sufficient authority, within that, to meet out those responsibilities.

83

B. PROCUREMENT DECISIONS-MASTER CONTRACT


Authority for the IDSRN rests on federal contract procurement rules. Within AHRQ, contracts are the responsibility of the Office of Performance Accountability, Resources, and Technology (OPART). OPART recently was created by combining planning and evaluation with budgetary and management oversight for the agency. The OPART director, Kathy Kendrick, reports to the AHRQ director, and shares the portfolios initiative with the AHRQ director, Carolyn Clancy. Within OPART, Darryl Grant is responsible for IDSRN procurements, including both the original competition and individual task orders.

AHRQ staff told us in interviews that the original nine IDSRN partners were selected based

on a competitive RFP as described in Chapter I. Applicants were required to provide extensive information on their data capabilities. While we heard diverse reports of the number of applicants, it appears there were more applicants than the nine ultimately approved, possibly even 14 applicants, according to one AHRQ staff person. A panel comprising 25 percent Federal members and 75 percent non-Federal members (three to five non-Federal people, we were told), reviewed the proposals. Qualified applicants were scored and ranked. Though the panel’s recommendation was to fund the top five or six applicants, the concern that these applicants lacked the capacity to study rural issues and potentially other factors led the agency to award contracts to the top nine applicants.


Because it occurred well in advance of our evaluation and also because it represents privileged information, we reviewed the original procurement process only with a small number of AHRQ staff most closely affiliated with it. We did not have access to many of the details or any of the documents involved in the process (except the winning proposals).

The perception in AHRQ seemed to be that the process went well, but that expanding the list

of awards to those who had scored lower ultimately meant that one lower-scoring IDSRN partner was quite weak. Staff noted, however, that another partner who received a contract did not score well during the review but has proven to be one of the more successful partners.

As described in Chapter V, IDSRN partners perceived the application process for the initial

competition to be quite demanding, particularly with respect to demonstrating data capabilities. While burdensome, IDSRN participants said that they recognized later why it was important to the IDSRN to have capabilities generated, and they perceived that they would be in a better position to respond next time.

3. Recommendations

In developing the RFP for the master competition, AHRQ should be as clear as possible about what they want so that potential applicants can be well informed in deciding whether to apply. AHRQ should keep in mind the burden applicants face in responding to proposal requirements and limit the information required of applicants to that critical to the review.

84

Requesting more information could limit the applicant pool in ways that might be inconsistent with AHRQ’s interests. On the other hand, AHRQ needs to require information that is essential to determining capability and resources, an inherently resource intensive requirement. AHRQ staff may want to consider, if procurement rules allow it, differentiating in the recompete between entities whose capabilities have already been demonstrated and those that have not. IDSRN participants also suggested that clarity on the kinds of work likely to be forthcoming would help them focus their efforts better so that they did not spend a lot of time working up collaborations or consultant agreements that were unlikely to be used.

For the reprocurement, AHRQ also will probably need to decide how to assess whether an

applicant has the support of the delivery systems on their team (see Chapter VI for recommendations on team composition). This is a challenging area for assessment, particularly since the IDSRN provides the potential for external funding, and systems will likely want to support their affiliates or staff in whatever way they can to make the application successful. Further, as we have shown in our description of the IDSRN partners and their collaborating teams, the associated organizational arrangements are complex. Some of the more effective processes for communication with management may be informal, and the formal processes that make communication work are likely to vary across institutions. We recommend that in addition to the usual letters of support, AHRQ should request information on this topic that would vary with the role of the entity in the team as well as where the research component is located.

For research components within delivery entities, especially when based in a lead partner or

major collaborator likely to lead individual task orders, applicants could be asked to describe (1) how that unit fits into the operational organization; (2) its mission vis–à-vis the organization; (3) the maturity of the affiliation; (4) a history of analytical communication around particular organizational priorities or issues; and (5) how they view the IDSRN would work best to support that mission. The intent behind these questions should be to understand the role of the research entity within the system and the likelihood that the main people proposed are positioned and have sufficient history, stature, and respect to provide the liaison with top management that AHRQ seeks.

When the lead partners are not based in a delivery system, additional evidence should be

solicited as proof of a link with operational leadership. This is even more important when researchers are based outside of delivery systems. We are assuming that leaders not based in a delivery system will be mainly involved with systems whose own research capacity is too weak to support the IDSRN as lead; otherwise, the delivery-based entity should be out in front. Outside partners need to provide assurance that the delivery system(s) with whom they relate have sufficient research orientation and capacity to support the analytical work that is likely to occur under the project. In addition to demonstrating this capacity, however, the outside partner probably should be asked to demonstrate that they have actively committed liaisons within each delivery system who are very senior in the organization and who have decision-making authority to commit the organization. A history of working together on successful collaborations would obviously be an asset.

85

C. PROCUREMENT DECISIONS—TASK ORDERS


The task order process has two steps. The first step involves deciding which requests for task orders to approve for solicitation of proposals and the second involves the process of competing and awarding those task orders that are approved.

The process of identifying potential task order topics is relatively informal. Topics come

from multiple sources, including AHRQ staff ideas that are shared with the IDSRN project officer; ideas raised by AHRQ staff in response to conversations they have had with outsiders on topics of interest; and a formal call for ideas from the IDSRN partners/collaborators. For topics to be funded by “core” IDSRN funds, the IDSRN project officer reviews the ideas to identify those that are most promising, using what appears to be his/her judgment and advice from the CDOM director. A memo is prepared three or four times a year in which these concepts are packaged and brought to the EMM where they are reviewed and ultimately approved or disapproved (in whole or part) by the AHRQ director. Topics that are funded through dedicated sources whose use is approved externally (e.g., bioterrorism awards) do not go through this latter approval process, and exceptions may also apply for other kinds of projects (e.g., Health Information Technology initiative). In these instances, the staff that control the funds in question select the topics, present them to the EMM (only in the case of patient safety-funded awards), develop the RFTO, convene the review committee and select the final awardee(s).

Once identified and approved as necessary, the person who originally conceived of the work

and who will be responsible for overseeing it later generally prepares the substantive part of the RFTO. The proposed task order paperwork is sent to the AHRQ contracts office where, after certification that funds are available, an RFTO is completed and sent to the IDSRN partners. Those people working in contracts say that once they receive this information, they turn it around very fast, often in a day. Depending on how close it is to the end of the fiscal year, IDSRN partners have a minimum of 20 days, and up to 35 days to respond. Most solicitations go to all partners, but there is authority for sole-source procurements where unique qualifications apply; this authority has been used most often for work that builds on prior task orders. Responses follow a standard format, and while there is no page limit, most are said to be 10 to 30 pages at most. All submissions come through the IDSRN partner. When proposals are received, they are given to the AHRQ technical staff, where they are reviewed internally by a few specialists in the substantive area covered by the task order who assess the proposals. In 2004, the reviewers used a scoring template and prepared a summary with the strengths and weaknesses of each proposal, an average score for each, and a series of questions or comments for the contract officer to send to the applicants for revisions (if necessary). The panel submitted the summary with recommendations for funding to the contract officer who then sent the contract to the partner(s). The process used for review appears to vary depending upon the preferences of the task order monitor and the nature of the work. Task order monitors with a strong interest in the topic area appeared more involved in the process.


Both AHRQ staff and IDSRN partners view the major strength of the task order process to be its speed. In contrast to both grants and free-standing contracts, AHRQ staff perceived that

86

task order contracts were rapid and easy to award. Contracts staff also liked them, though they found it very burdensome to get a large number of requests late in the fiscal year. IDSRN partners/collaborators said that the fact that there was quick-turnaround on the awards was an important trade-off for the short proposal schedule. They probably would be substantially less happy with the program if the award process were lengthened. No one we spoke with described the application process for individual task orders as burdensome, which contrasts with that for the master IDSRN agreement. In general, too, we had a sense from the responses that participants generally thought that their efforts were rewarded with what they saw as a high probability of success, especially as they became more experienced in working with AHRQ.

From a procurement process point of view, the main weakness IDSRN

partners/collaborators pointed out was the bunching of awards toward the end of the fiscal year and the fact that requests came without advance warning. They want more advance notice, though they appreciated that AHRQ’s communication had been improving over time. Our interviews preceded FY 2004 procurements, so we do not know how they view the most recent year’s experience with what appears to us to be a large number of requests, again, late in the fiscal year. In general, IDSRN partners and collaborators liked the increasingly flexible way task orders were written but wanted more ability to give input on future topics and more realistic assessments of what could be done for the available money.

From the interviews, our sense is that the process of identifying which task orders to issue is

very informal, and its success depends very much on the insight and judgment of the IDSRN project officer. We heard of at least one case, early in the IDSRN, where the process of judging awards appeared relatively unstructured, which raises the potential that proposals would not necessarily be judged adequately or fairly. However, we heard that in recent years the process has been formalized, with the IDSRN project officer setting up review panels to evaluate applications for each RFTO, one of whose members includes the person who wrote the scope of work for the RFTO.

3. Recommendations

The speed with which the IDSRN task order process works is very central to its success, so any changes need to be assessed with consideration of the impact on administrative burden and speed. Within this boundary, however, it appears that there are ways in which the task order procurement process could be strengthened.

First, while informal processes have their advantages, it would be valuable to develop a

clearer description for all AHRQ staff on how ideas should be communicated and what criteria are important in judging a “good” topic. The content for this should be based on a statement of IDSRN goals and priorities that we recommend should be developed. Mechanisms for getting outside input from potential users should be strengthened in ways consistent with those discussed in Chapter VI. The IDSRN project officer should have authority to reject ideas that are inconsistent with this guidance. If AHRQ continues the current process of approving concepts for individual task orders, it would be valuable, as noted above, to tie recommendations to a broader analysis of priorities and their relationship to both user needs and the state of the art. The origins of particular ideas and why they were recommended over other approaches or ideas that might have been suggested could be a part of the submission. This might provide more

87

transparency that could be useful, especially if there is turnover in the IDSRN’s internal leadership at AHRQ.

Second, AHRQ staff needs to do whatever it can within external constraints to space out and

better plan for use of IDSRN funds. We were told that the agency can be fairly certain of at least a minimum amount of funding even if the full amount is not known or actual amount appropriated, and that staff are reluctant to commit for fear “something better will come up.” One interviewee also suggested “the budget delays are an excuse not to plan.” The department is encouraging more speed in spending across the agency by setting performance goals that include measures of the timeliness with which funds are released. Hence, the AHRQ director is being assessed on this dimension, and AHRQ has begun to respond with more incentives for early awards of grants. Perhaps the IDSRN could be rewarded for moving awards more quickly by providing “bonus funds” to each person responsible for relevant IDSRN contracts if certain early commitment goals and are met. A forward-planning process that is strategically focused on the “big picture” would obviously help as well.

Third, there could be merit in providing AHRQ staff with substantive mentoring to help

them think about how best to write a statement of work that leaves applicants with room for creativity but also provides a clearer statement of what AHRQ seeks in the work along with a scope of work matched to the likely funding available. Possibly one way to do this would be to invite someone who has had similar responsibilities for substantive task orders like this—in government or outside it but with no conflicts of interest—to give one or more seminars to staff as part of a panel.

Fourth, it is worth considering how best to assure equitable treatment in review and expert

guidance within the time constraints of the IDSRN task order review process. Right now reviews are internal and are apparently based on “general scoring.” We have seen RFTO proposals that include specific criteria defining how proposals will be judged. While AHRQ probably wants to provide some flexibility to accommodate diversity within the types of proposals different partners might submit, it should be feasible to be clearer with applicants about what AHRQ is looking for and how applications will be scored. In addition, it would be valuable, to the extent this does not exist now, to more formally establish within AHRQ how task order proposals are viewed and how review panels are constituted. In one interview, it was suggested to us that the IDSRN consider adopting processes used externally for gathering expert review efficiently. For example, AcademyHealth’s Health Care Financing Organization (HCFO) has an online expert review process that pays reviewers $100 for a review of a proposal. It could be worth considering the use of such strategies within the IDSRN to strengthen reviews. Obviously those reviewing need to understand the IDSRN and be committed to reviews consistent with its goals. If each task order were reviewed by two outsiders, both applied but one more focused on research and one on practice, it might enhance the credibility of the IDSRN awards, particularly in a research culture that puts great value on independent and external peer review.

Fifth, AHRQ needs to consider ways of providing the IDSRN project officer or other people

within AHRQ additional flexibility to make awards or modify contracts in a way that builds on each task order. Paralleling our recommendations that task order requests be justified with a solid analysis of their rationale and expected potential, the IDSRN project officer might be asked to require that task order monitors assess task order outcomes following the completion or the

88

near completion of each task order. The assessment would be substantive and would differ from the administrative review of performance for all contracts that HHS requires. The focus would be on assessing performance against the original rationale for the project and what was expected as outcomes. It would include a review of the importance of the task order findings and, where appropriate, what next steps would be valuable move the findings in the direction of application. A continuous quality improvement model that encourages frank analysis might be helpful. To encourage use of follow-up, a certain amount of funding within the IDSRN might be reserved for follow-on work, dissemination activity, or other kinds of support. In addition, there should be a process of assuring that the findings and lessons from each task order as well as the assessments of their potential are shared broadly within the agency so that they are linked to the variety of work they might support. This process would provide the IDSRN with the benefit of knowledgeable experts with different perspectives.

D. TASK ORDER OVERSIGHT

1. Current Structure and Processes

Each task order is assigned a technical contract monitor within AHRQ who is responsible for overseeing performance; certifying receipt of deliverables and assessing their acceptability; notifying the contracts office of any known problems in performance; and providing the AHRQ contracts office with approval for invoice payments (once reviewed by contracts for completeness) or other changes under the contract. Contracts typically require that the IDSRN partner report monthly or bimonthly on progress, in addition to any more formal deliverables.

When feasible, the task order monitor is the person who is responsible for work in that area

and is the author of the task order request. That means that the two staffers who work on bioterrorism at AHRQ, for example, handle most IDSRN bioterrorism task orders, and that others are similarly assigned. Before they can handle a task order, the monitor must go through project monitoring training within HHS.

Increasingly, task order monitoring responsibilities have become more concentrated, both by

chance and by design. When there is turnover in agency staff, the IDSRN project officer may assume responsibility for task orders either permanently or temporarily. Such turnover is not uncommon. Recently, for example, a task order monitor with responsibility for overseeing six of the task orders related to informatics left the agency. The current IDSRN project officer also has deliberately tried to concentrate management responsibilities for the task orders among staff who are likely to take their responsibilities seriously, and who have a history of good performance. In an environment in which intramural research is what attracted some AHRQ staff, interest in the oversight of research conducted externally by others varies substantially across staff in the agency. The strongest researchers have not always been the strongest task order monitors.

AHRQ task order monitors vary in how they carry out their functions. A few aim for very

substantive roles on a project. For example, they may be experts in that area and may have had a prior relationship with the team. Or, they may take responsibility for encouraging publication of the work and may contribute substantial time to helping a contractor develop an effective manuscript for a project that they feel has value. Task order monitors taking this strategy described their roles as “hands on,” “interactive,” and “demanding,” and at times served as a full member of the task order team. Others perceived themselves to be more a facilitator than an

89

expert. While they are committed to the work, their goal is to avoid barriers that detract from success and not micro-manage the work, unless that is called for. Still others are relatively laissez-faire, making little distinction between contracts (which imply accountability) and grants (which are less structured). And some combined multiple roles, for example, providing oversight and managing the mechanics of the task order but providing feedback and guidance on technical aspects where this drew on their skills and expertise.


Most AHRQ task order monitors with whom we met appeared to take their responsibilities seriously and seemed committed to the work. Those new to this kind of role were mixed about the guidance they received to be an effective task order monitor. AHRQ is viewed by many of its staff as an agency where you learn by doing, with few formal training or mentoring processes other than standard departmental processes. Some staff are active in seeking out the advice they need and in understanding the issues associated with managing this kind of work. Others are not. These individuals said they either had little trouble learning what they needed to do this work or wished for more “brown bags” or other types of mentoring. The department’s training was said to focus on mechanics, not on developing appropriate management strategies for task orders. Few were excited about it, although some were more critical than others.

The IDSRN partners and collaborators we interviewed had diverse experiences with their

individual task order monitors, but few were critical. A number noted the value added by the particular person at AHRQ with whom they worked. For these contractors, their main concerns dealt with contracts and administration. Monthly reporting requirements were viewed as burdensome and, from their perspective, not very useful given the speed of work. There were concerns that reporting monthly on progress was burdensome and not necessarily of value for these kinds of projects. They perceived that they were micromanaged. Over time, many said they learned better how to interact with the contracts staff and felt the process was going more smoothly.

As noted previously, a continuing concern appears to be whether the contracts office is

sufficiently flexible. At least some partners and collaborators said they wanted more sensitivity to the fact that sometimes there were reasons for delays in work that were outside the researchers’ control. They also wanted to be able to get extensions for the work more easily and to use any remaining funds in the task order contract to support dissemination. Concerns over contract flexibility appeared to go beyond any greater familiarity with grants but to the way AHRQ interprets the contract vehicle.

The structure of the HMO Research Network also created tensions with the contracts office,

though many issues have been resolved by now. The HMO Research Network is not a formal organization but a vehicle for its members to collaborate and communicate, not just around issues of the IDSRN but also more generally. Though Kaiser Northern California serves as the official partner for the IDSRN, they and others in the network view Kaiser’s role as an administrative conduit. (Such responsibilities are rotated across key leaders in the network over time.) While Kaiser staff take responsibility for leading communications and decisions about responding to task orders as well as transmitting reports, they and others in the network view each task order as the particular responsibility of the collaborating team working on that project. Thus, unless Kaiser Northern California has a substantive role in a particular task order, they

90

tend not to be involved in substantive reviews of either proposals or products. Because this departs from traditional methods of assigning accountability under contracts, the contracts office has been critical of the fact that the official lead partner in the HMO Research Network on each task order is not more involved, especially when they perceived the task order team to have weaknesses that such involvement would help to correct. In addition, the contracts office has noted that HMO Research Network is the only partner that uses a percentage of its task order funding to cover administrative costs, so they find it particularly distressing to observe poorer administration by this partner than by other partners who absorb the cost.

A few partners also took issue with the minority subcontracting requirement, because they

felt it was unneeded and did not work well with these kinds of collaborations. For example, a safety net IDSRN partner perceived that their entire system was focused on minority issues, and another partner who made such an arrangement felt badly that the collaborator spent a lot of time on the proposal but ultimately got no work because the task orders did not use their skills.

3. Recommendations

There is much about the process of monitoring contracts that appears to work quite well at AHRQ. Task order monitors seem to want to see work under their oversight succeed, and they try to do what they can to facilitate that success. While most of the time they are not very involved substantively, the roles they have played when they are involved seem highly constructive and valuable.

From the perspective of technical oversight on task order contracts, there are two main areas

that appear to need improvement. First and foremost, the current trend toward centralizing task orders with the IDSRN project officer probably is not best for either the agency or the IDSRN, even though having committed and qualified monitors is an important goal. As we discussed in the previous chapter, the IDSRN project officer is unlikely to have time for important strategic work across the IDSRN if she has too many responsibilities for monitoring individual contracts. Further, concentration lessens the investment and awareness (at least in theory) across the agency in the IDSRN. And, AHRQ task order monitors who spend their time working on issues that have some relevance to the topic of a given task order are more likely to be knowledgeable about the unique contributions of any project and how it can be applied or linked with work going on elsewhere. Task orders should not be awarded in specific areas unless the units within the agency most closely affiliated with that work are willing to invest in oversight.

Second, it would be valuable to develop more systematic ways of helping staff learn how to

be good monitors of task orders. Freed of primary responsibility for oversight of most task orders, the IDSRN project officer should be in a better position to mentor new staff and oversee the effectiveness of the monitoring process across all task orders. While AHRQ staff say the agency is too small to have formal training on task order monitoring for new staff, AHRQ should be of just the right size to support periodic (monthly or bimonthly) meetings of task order monitors—both experienced and inexperienced—to share insights, both substantive and process. If interests vary too much, some meetings could be organized around substantive areas of shared concern to a subgroup of monitors. Informal meetings also would facilitate communication across the agency on the capabilities reflected in the IDSRN, the results of task orders, and how they might relate to other ongoing work.

91

From the perspective of contracts, it would appear useful to develop more clarity on certain policies—for example, the granting of contract extensions, the ability to use remaining funds to support dissemination, and so on. We heard different reports from diverse IDSRN partners/collaborators about how easy or hard it was to get approval for this and about the circumstances under which approval would be granted. Though much of this probably reflects ignorance about the “bigger picture,” reports of such inconsistencies suggest that better communication and greater clarity of the criteria used for decision making is possible.

As part of that review, AHRQ also might consider whether it is using its contracting

authority most effectively to achieve the goals of the IDSRN. In our experience, contracts can be written in ways that provide substantial flexibility, and award of extensions is not necessarily inconsistent with the desire to motivate quick turnaround work. It is our impression that IDSRN partners/collaborators had learned, for the most part, to work with the contracts office staff but did not view such staff as being as flexible as they might be, even within the constraints of a contract environment. Those responsible for conducting task orders almost universally viewed themselves as trying to do the best job possible under sometimes challenging circumstances, even if AHRQ viewed their performance negatively. They took criticism personally and felt that it detracted from the kind of collegial partnership they wished to have with AHRQ.

E. GENERAL INFRASTRUCTURE FOR IDSRN WITHIN AHRQ

1. Current Structure and Processes

As noted previously, the IDSRN has very little formal infrastructure aside from annual meetings of IDSRN participants and a quarterly newsletter than had its first issue in April of 2002. The latest issue included an update on the number of awards likely this year and the total IDSRN funding, including a table listing awards made or solicited. The rest of the newsletter included a listing, by partner, of specific presentations and publications related to task order work.

2. Areas of Strength and Weakness

The IDSRN does its own internal evaluations of the annual meetings, and our findings probably are consistent with what we understand the IDSRN’s own evaluations to report. Perceptions of the value of the meetings vary, possibly in ways correlated with how much participants like meetings in general. Overall, the predominant view seems to be that the meetings serve a good function because they are the only time that participants can get together and learn about each other. Interviewees noted that the meetings were the main way they communicated outside their team and that the annual meetings were the main way the IDSRN was identifiable. The meetings also, because of their proximity to AHRQ’s offices, provided a way for AHRQ staff to interact with IDSRN partners and collaborators, particularly if their work relates to some of the areas presented.

The strongest criticisms centered around the agenda for the annual meeting. Partners and

collaborators wanted more ability to shape the agenda and provide input. The focus on presentations of ongoing IDSRN work was viewed as too limiting. In addition, some were very critical of what they viewed as either idiosyncratic or terribly flawed presentations that they

92

believed did not represent rigorous research building on the state of the art. Participants at these meetings wanted more time to talk to each other informally. They also thought more focus on topic areas that may come up during the next year or emerging issues was important. As one said, “Time should be devoted to focusing on future efforts.”

Fewer comments were made about the newsletter. Some collaborators appear not to know

about the newsletter. One, for example, who did not know about it suggested just such a mechanism. Whether this stems from AHRQ’s distribution system or that of the partners is uncertain. Among those commenting on the newsletter, one person said it was “dry,” though most felt it seemed to serve its communication objectives, even though they said it could be “punchier.” Those we interviewed also said that, in general, finding information on what projects others in the IDSRN were working on was challenging. Some suggested that the IDSRN web site be built up so that it was easier to access within the AHRQ web site (some viewed it as “hidden” now) and so that it supported a “library” of IDSRN information accessible online as useful ways to better provide information about the IDSRN.

3. Recommendations

In terms of meetings and the newsletter, we recommend that AHRQ continue them for now, but that AHRQ should solicit more input from users on what would be useful, both in terms of agenda topics and styles and in newsletter content. The meetings should not be viewed as a major vehicle to move research to practice, since the participants generally are applied researchers talking to each other from a base of systems whose shared concerns are not that great. But they provide a way for participants to learn more about AHRQ’s goals for the IDSRN as well as upcoming plans. If administratively feasible, more effort should be made to get participation from a broader and more consistent set of participants, including those in collaborating organizations. If face-to-face participation cannot be substantially expanded, perhaps the interests in web casting could be explored. The agenda also should be structured to provide more time for interaction both informally and through sessions focused on topics of shared concern to the IDSRN.

We are uncertain how the meeting should be viewed once the IDSRN is recompeted. If

partners are selected in ways that match portfolio areas, it may be possible to combine an overview session focused on topics of mutual interest with more targeted programming in specific areas. If based around portfolios, it could provide an opportunity to foster linkages within AHRQ and HHS across related activities and concerns. If AHRQ can combine its process of obtaining executive input with these meetings, it may be possible to build on the expertise represented there to promote broader discussions of how to move research to practice. Outsiders relevant to particular topics might be included as appropriate, especially in the targeted part of the meeting.

In terms of the newsletter, AHRQ might want to think about the audience and what that

means for content. As we see it now, the main audience for the newsletter is within the IDSRN. But feedback from users on what they would find useful could be valuable, since we are uncertain that a listing of presentations and publications best serves those needs, even though it could help AHRQ demonstrate activity under the IDSRN.

93

The review of the newsletter should be coordinated with planning for improvements in IDSRN web-based communications (which we believe are underway, at least to some extent). In general, better identifying the diverse audiences that the IDSRN wants to communicate with and the topics it wants to cover would help in determining periodicity and modes of communicating. With the growth in web-based technology, it could make sense for the IDSRN to better serve the needs of its partners and collaborators through periodic e-mails and a web site that included diverse kinds of information and resources. The newsletter could be recast as a more general way of highlighting particular activities or findings from the IDSRN for an outside audience of users. Resources could also be included that diverse audiences might find useful in better translating research to practice. Those resources could include, for example, articles, publications, and research syntheses, whatever their source, that are of particular interest to the key audiences that IDSRN seeks to reach. We assume these audiences are the same ones the IDSRN seeks to attract as partners and collaborators in the IDSRN.

As we have already discussed in Chapter VI, the IDSRN needs more infrastructure to

support its goals. We recommend that decisions on administrative resources to support the program be a core part of the process of clarifying goals.

F. THE QUESTION OF CONTRACTS VERSUS GRANTS

AHRQ wants to use the IDSRN to help foster certain objectives, namely the translation of research into practice (in ways to be defined), working with established partners and their collaborators. Politics aside, the IDSRN seems to be something whose pursuit is entirely consistent with a contract approach, particularly one based on the ability to develop specific products through the use of timely and targeted awards and to work with the private sector, including organizations with diverse business models, some of which may be harder to accommodate in grants.

There are four major potential arguments against a contracts focus for the IDSRN at AHRQ:

first, the organization has an historical focus on grants within an academic environment that regards such awards as signs of successful research. This means that the contract vehicle is, for the traditional research community, less prestigious, and this attitude carries over among some AHRQ staff. Second, support for the IDSRN both within AHRQ and externally, especially with NIH, is inhibited by the use of the contract model since both AHRQ and NIH have substantially more funds available to award using the grant mechanism than a contract mechanism. Third, contracts require OMB approval to collect new data (e.g., surveys), whereas grants have more flexibility. Fourth, contracts are viewed by AHRQ as requiring specificity that could eliminate their use for some objectives (e.g., core support to partners for administration, communication and similar functions).

Of these arguments, the second one appears the most serious, because an inability to tap into

grants could limit the agency’s ability to make the kinds of decisions it might want to about how to allocate resources. Further, the need to meet commitments on the share of funding that will be spent through grants may make it harder for the agency to avoid the unpredictable and uneven way in which funds have been made available to the IDSRN over time. It would be helpful to sort out how much of the issue is the value of contracts versus grants per se—as opposed to what the choice means for available funds. To the extent it is the latter, there may be creative ways of

94

structuring arrangements that keep the IDSRN task order focus but allow additional resources to become available, especially from outside the agency.

The OMB requirements argument is an issue only to the extent that it is important to the

IDSRN that the capacity to collect new data of the type subject to OMB is important to the success of the IDSRN. Depending on program goals, one could argue that new data collection is not central for most projects. There may be times, however, when it is – for example, when AHRQ wants to survey users to see if a tool works or to evaluate an intervention. When new data collection is necessary, there may be ways to structure work so that the IDSRN is able to get the data it needs more quickly than would be the case if OMB approval is needed. For example, funding for new data collection could be assumed by others or separated from the core task order work.

The other two arguments also appear less fundamental to us. Researchers, wherever they

work, are increasingly being held accountable for their work and many work partly or exclusively on contracted work. Properly administered and with the right attitude, there is nothing inherent in contracts that should lead researchers to experience them differently than many cooperative agreements. Writing contracts so that they provide flexibility and support for work that extends beyond an individual task order is more challenging than writing an individual task order, but there is experience with this, and contracts potentially can be written in ways sufficiently flexible that they allow for certain expenditures.

95

REFERENCES

Agency for Healthcare Research and Quality. “AHRQ Profile: Quality Research for Quality Healthcare.” Available at [www.ahrq.gov/about/profile.htm]. Accessed August 12, 2003a.

. “Fact Sheet: Integrated Delivery System Research Network (IDSRN): Field Partnerships to Conduct and Use Research. Available at [www.ahrq.gov/research.htm]. Accessed July 28, 2003b.

. “IDRSN Project Awards: Field Partnerships to Conduct and Use Research.” Available at [www.ahrq.gov/research/idsrnproject.htm]. Accessed July 28, 2003c.

. “Quality Research for Quality Health Care: A Report from AHRQ on Recent Activities and Future Directions.” AHRQ Publication #01-0018, March 2001.

Berwick, Donald M. “Disseminating Innovations in Health Care.” Journal of the American Medical Association 289(15): 1969-1975, April 16, 2003.

Clancy, Carolyn M. “Testimony on the President’s Fiscal Year 2004 Budget Request for AHRQ.” Before the House Subcommittee on Labor-HHS Education Appropriations. from Available at [www.ahrq.gov/about/cj2004/cjtest04.htm]. Accessed August 12, 2003.

. “Testimony on Technology, Innovation, and the Costs of Health Care” before the Joint Economic Committee, July 9, 2003.

. “Perspective: Back to the Future.” Health Affairs. Web Exclusive W3-314-316, June 25, 2003.

Eisenberg, John M. “Globalize the Evidence, Localize the Decision: Evidence-Based Medicine and International Diversity.” Health Affairs 21(3): 166-168, May/June 2002.

Fitzgerald, Louise, Ewan Ferlie, Martin Wood, and Chris Hawkins. “Interlocking Interactions, The Diffusion of Innovation in Health Care.” Human Relations 55(12): 1429-1499, 2000.

Gladwell, Malcolm. The Tipping Point: How Little Things Can Make a Big Difference. Little, Brown, & Company, 2000.

Gray, Bradford H., Michael K. Gusmano, and Sara R. Collins. “AHCPR and The Changing Politics of Health Services Research.” Health Affairs Web Exclusive, W3-283-307, June 25, 2003.

Hage, J.T. “Organizational Innovation and Organizational Change.” Annual Review of Sociology 25:397-622, 1999.

Hornik, R.C. (ed). Public Health Communication: Evidence for Behavior Change. Washington, DC: American Psychological Association, 2002.

96

Kingdon, John W. Agendas, Alternatives, and Public Policies. Addison Wesley Longman, 1995.

Lavis, John, Suzanne E. Ross, Jeremoiah E. Hurley, et al. “Examining the Role of Health Services Research in Public Policymaking.” Millbank Quarterly 2002: 125-154.

Lavis, John H., Dave Robertson, Jennifer M. Wooldridge, Christopher B. McLoed, Julia Abelson, and the Knowledge Transfer Group. The Milbank Quarterly 81(2): 221-247, 2003.

Lomas, Jonathan, Naomi, Fulop, Diane Gagnon, and Pauline Allen. “On Being a Good Listener:

Setting Priorities for Applied Health Services Research.” The Milbank Quarterly 81(3): 363-388: 2003.

McDermott, R.J. “Social Marketing: A Tool for Health Education.” American Journal of

Health Behavior 24(1): 6-10, Jan-Feb 2000.

McGinnis. “The Case for More Active Attention to Health Promotion.” Health Affairs 21(2): 78-93, March/April 2002.

Plsek, Paul. “Complexity and Adoption of Innovation in Health Care.” Prepared for a conference on Accelerating Quality Improvement in Health Care: Strategies to Speed the Diffusion of Evidence Based Innovations” January 27,28, 2003, Washington DC.

Sofaer, S., and R. Myrtle. “Interorganizational Theory and Research: Implications for Health Care Management, Policy, and Research.” Medical Care Review 48(4): 371-409, winter 1991.

Sorian, Richard, and Terry Baugh. “Power of Information: Closing the Gap Between Research and Policy.” Health Affairs 21(2): 264-273, March/April 2002.

Strang, David, and Sarah A. Soule. “Diffusion in Organizations and Social Movements: From Hybrid Corn to Poison Pills.” American Review of Sociology 24:265-390, 1998.

Wejnert, Barbara. “Integrating Models of Diffusion of Innovations: A Conceptual Framework.” Annual Review of Sociology 28:297-326, 2002.

APPENDIX A OF FINAL REPORT CASE STUDY ANALYSIS

A-3

As part of Mathematica’s evaluation of the Integrated Delivery System Research Network (IDSRN) for the Agency for HealthCare Research and Quality (AHRQ), we prepared four case studies that provide examples of projects that have achieved some success in translating research into practice. This overview summarizes how we identified and developed the cases and describes the cross-cutting findings that emerge from our analysis. The intent was to provide “real-life” examples of the types of accomplishments that have resulted from the IDSRN and how they evolved. It is important to note that the analysis did not involve technical assessment or validation of the tools produced under the IDSRN but rather describes the evolution of the tools and products and how they have been used to date.

METHODS AND CRITERIA FOR SELECTION

We selected cases, by design, that appeared to be among the most successful efforts achieved through the IDSRN. Our aim was to profile such work, understand it better, and learn about the factors that foster (as well as impede) favorable results. We defined successful efforts operationally as those having achieved some level of impact (i.e., adoption) within the system conducting the research and/or having broader relevance as illustrated through adoption or active consideration elsewhere.17 With resources limiting us to four case studies, we selected cases that reflected diverse areas of work carried out by a variety of IDSRN teams. For purposes of selection, we grouped related task orders awarded to a single team on the same body of work; such task orders were typically follow-on task orders whose intent was to move work forward toward application. To be considered, the task orders had to have been completed by July 2004 (exceptions were made for multiple task-order work that already had demonstrated impact even though a follow-on task order was still being pursued).

Of the 58 task orders awarded under the IDSRN between FY2000 and FY2003, eight were

not completed by July 31, 2004, and therefore were not eligible for the case study analysis. Of the remaining 50 task orders, about 30 had some operational impact beyond just a final report or a single paper, and of those, we identified about 24 unique areas of research (where a follow-on task order is considered the same area of research as the task order from which it emerged). (See Chapter IV for more information on how we defined operational impact.) With AHRQ’s approval, we selected four case studies based on projects with the greatest impact, as revealed in the partner/collaborator interviews, discussions with AHRQ staff, final reports, and other deliverables or products, while making sure that selected studies spanned multiple teams/partners, portfolio areas, settings, and funding sources (i.e., core funding versus other sources). Table 1 lists the four selected projects, which address different topic areas, rely on different funding sources, and bring together collaborators/partner teams.

For the case studies, we reviewed existing documentation (such as AHRQ memoranda and

project teams’ proposals and final reports), notes from the partner/collaborator interviews, and notes from our interviews with AHRQ staff. We also conducted interviews with an additional one to two people for each case study, focusing on individuals who adopted the innovation (either internal or external to the delivery system conducting the research).

17 For additional details on these definitions, see Chapter IV.

A-4

Table 1. Projects Selected for Case Study Analysis

Project/Area of Research Task

Orders Portfolio Area and

Funding Source

Impact/Adoption

Bioterrorism models (Weill Medical College)

#1–4 Bioterrorism (AHRQ bioterrorism funding)

Adopted by many groups outside the IDSRN, including New York State Department of Health and state and local entities in Minnesota, North Dakota, Ohio, Texas. Also, used by CDC Stockpile and other federal agencies. BERM software is downloadable from AHRQ and American Hospital Association Web sites.

Culturally and Linguistically Appropriate Service (CLAS) quality assessment and performance measurement projects for Medicare+Choice Organizations (HMO Research Network and Lovelace Clinic Foundation)

#2 and #11

Prevention and training (CMS funding and IDSRN core funding)

Adopted by others in IDSRN as well as by groups outside the IDSRN. UNC disseminated the CLAS guides via workshops across the United States.

Integrated delivery system solutions for transferring medication information across patient care settings (RTI and Providence Health System)

#4 and #6

Patient safety (AHRQ patient safety funding)

Adopted by Providence as well as by groups outside the IDSRN (a Seattle hospital has used the findings).

Race/ethnic differences in quality and outcomes (Center for Health Care Policy and Evaluation (CHCPE) and RAND)

#1 and #7

Health Care for Minority Populations, Women & Children (IDSRN core funding)

Adopted by IDS and groups outside the IDSRN. Related work continues under the health plan disparities collaborative sponsored by AHRQ and RWJF.

CROSS-CUTTING FINDINGS

Role of Environmental Factors

All four cases focused on areas in which important external forces in the environment encouraged (or are encouraging) additional work:

• The race/ethnicity projects by the Center for Health Care Policy and Evaluation (CHCPE)/RAND and by Lovelace Clinic Foundation were influenced by a general increase in interest in disparities across health plans in the last few years, along with a changing perspective on the legality of collecting data on members’ race and ethnicity. In fact, one member of the CHCPE/RAND team suggested that health

A-5

plans now believe that they must begin investigating disparities in order to remain competitive in the marketplace.

• The work on medication information transfer by the RTI/Providence Health System team has been affected by the recent announcement by the Joint Commission on Accreditation of Healthcare Organizations that medication reconciliation across the continuum of care will be a primary safety objective of the organizations it accredits as of 2005. This development has resulted in strong interest in the medication information transfer projects among the hospitals and senior management within Providence Health System. Moreover, the project team expects that its final study results will be released publicly in early 2005, just as hospitals across the country begin work on the new safety objective.

• In the case of Weill Medical College’s bioterrorism work, the events of September 11, 2001, had an enormous effect on the demand for and interest in this type of work; the target audience for the work included a wide variety of organizations, such as federal agencies, state and local departments of health, and integrated delivery systems.

Importance of Generalizability

Producing work that is generalizable is another important aspect of the cases. All four cases developed models or tools that were in some sense generic in order to make the models/tools applicable to a wide audience. In some cases, widespread applicability was the original intent of the project, whereas in other cases, the project evolved over time into a more generalizable tool. For example:

• From the start of the project, Lovelace Clinic Foundation’s work on cultural competency (funded by the Centers for Medicare and Medicaid Services (CMS) but awarded through the IDSRN) was intended to produce practical tools for Medicare plans across the country, and Lovelace developed its tools accordingly. The work ended up much more broadly applicable than originally envisioned, extending beyond just Medicare plans or even health plans for that matter.

• The fact that the Weill team could not gain access to sensitive data meant that it developed a “generic” simulation model for assessing bioterrorism preparedness. Rather fortuitously, the generic model proved much more useful to many more organizations.

• When RTI and Providence Health System began their work on medication information, they had no notion of the similarity of medication transfer issues across integrated delivery systems. Across the three integrated delivery systems studied, the project team found startling similarities in process despite different geographic locations and delivery systems. As a result, the team was able to demonstrate the application of a fairly generalizable process assessment method (failure mode and effect analysis, or FMEA) to help assess points of vulnerability for medication errors and other related issues that occur between admission and discharge.

A-6

• The race/ethnicity display tool developed by RAND and CHCPE evolved organically. The team had developed the tool for its own use when examining potential racial/ethnic disparities in health outcomes for a research project but soon realized that the tool could be useful to plans interested in examining disparities. The team adapted it accordingly for more general use.

Importance of Follow-on Work

Another important theme across the four case studies was that each case encompassed several projects. Although the inclusion of multiple projects was not a criterion for case study selection, it seems that AHRQ rewarded more successful task orders with follow-on work, thereby allowing project teams to take their inquiry to the next level and begin applying their research in more practical, operational ways (such as implementing an intervention or developing a tool). For all cases included here, a single task order spurred a body of work (including, for some investigators, work funded outside the IDSRN). For example, although the most operational product of RAND/CHCPE’s race/ethnicity work was a display tool, its IDSRN task orders spawned other work and recently helped lead to the creation of a disparities collaborative for health plans sponsored jointly by AHRQ and the Robert Wood Johnson Foundation.

Role of Demand-Driven Research

The importance of “demand-driven” research was evident in several of the case studies. Because demand-driven research reflects the needs of management or others in an organization, it ensures an audience that is not only interested in project findings but perhaps also more willing to use those findings for operational purposes (compared to projects that may answer interesting research questions but are not directly driven by operational interests). Lovelace’s work on cultural competency resulted in two guides produced for the express purpose of aiding Medicare plans in meeting their Quality Assessment and Performance Improvement (QAPI) requirements, as established by CMS. As such, the guides had a direct tie to federal requirements, making them operational and relevant to a wide audience. Similarly, the fact that the RTI/Providence project on medication information transfer evolved directly from questions posed by a physician group within Providence Health Plan meant that management was interested in the study results. Finally, Weill’s work related to bioterrorism was demand-driven in that major parts of it (for example, the planning guide) came out of specific requests by the U.S. Department of Health and Human Services. In addition, several types of organizations had a strong need for the models produced; such organizations were a captive audience, given that few other models exist in the area of bioterrorism preparedness.

Challenges Faced

Although the cases presented here represent IDSRN “success stories,” all of the cases faced challenges, most of which seemed to be idiosyncratic to the individual project. For example, the Weill team was unable to obtain sensitive state and local data on bioterrorism responsiveness for its models, while OMB clearance rules for government contracts meant that Lovelace Clinic Foundation could only survey a small number of plans when examining plan response to their

A-7

cultural competency guides. Time was a constraining factor for at least two of the cases, but in different ways. RAND and CHCPE indicated that they needed time for several iterations of refinements to their tool based on feedback gathered from plans. Lovelace encountered more concrete deadlines as it faced the challenge of completing the cultural competency guides in time for plans’ use in meeting the 2003 QAPI requirements of CMS.

SUMMARY

The case studies that follow highlight the important work that occurred under several IDSRN projects. The studies represent success stories and are indicative of the effectiveness of the IDSRN program in creating practical and useful tools. Moreover, all four projects have recently spawned new work and/or have strong potential for additional application in the future. The order in which the following cases are presented is alphabetical by project name and does not reflect any ranking by success or effectiveness.

A-9

CASE STUDY 1: WEILL MEDICAL COLLEGE’S BIOTERRORISM WORK

PROJECT IMPETUS

The events of September 11, 2001, poignantly illustrated the importance of an adequate public health sector response to terrorism and pointed to the challenges associated with current capacities and research that could provide a platform for future response. With support from Congress, the U.S. Department of Health and Human Services (HHS) received funding to support a range of efforts to address existing limitations, and AHRQ, as an agency of HHS, was part of the response. Though the IDSRN was not originally conceived as a vehicle for improving the public health infrastructure, its link with an organized delivery system and its focus on translating research to practice provided a useful resource for AHRQ and HHS in undertaking applied research to develop tools that could be used to support the public health response to bioterrorism. The case study illustrates the serendipitous application within New York City of work initiated before September 11, 2001, to help plan for anthrax or smallpox vaccinations and hospital bed allocation that would be required in response to an anthrax attack.

BACKGROUND

Weill Medical College of Cornell University (Weill) received IDSRN funding totaling $993,000 for four bioterrorism projects between fiscal years 2000 and 2003. While the body of research produced a variety of tools, it culminated in the development of two important new interactive computer models designed to serve the needs of end-users in the public health and emergency response community: the Bioterrorism and Epidemic Outbreak Response Model (BERM), which estimates the minimum staff needed to operate a network of dispensing clinics in the event of an anthrax or smallpox epidemic, and the Regional Hospital Caseload Calculator, which calculates the rate of casualties produced by anthrax or plague releases based on a set of changeable assumptions (see Table 2).

PURPOSE OF WEILL’S BIOTERRORISM WORK AND HOW IT IS USED AND DISSEMINATED

Weill’s BERM allows users to input the characteristics of their community or patient population (e.g., population size), specify the appropriate values of variables such as the type of outbreak (e.g., noncommunicable versus communicable) and level of staff “downtime,” and then estimate the staffing necessary to respond to an attack. (See Table 3 for a description of the capabilities of Weill’s bioterrorism tools.) BERM’s target audience is state and county bioterrorism planners, though hospitals have also used the software. While the BERM software has been Weill’s “most useful product to date” according to principal investigator Nathaniel Hupert, other simulation models (e.g., regional hospital caseload calculator) have also been important and have formed the basis for other work by the Weill team.

In mid-September 2004, AHRQ released its planning guide for mass prophylaxis. The

document, developed by Weill and designed to be used in conjunction with the BERM software,

A-10

Table 2. Relevant Projects for Weill Medical College Case Study

Project Name Dates Award Description

Integrated delivery systems and clinical preparedness for bioterrorist events

9/2000–9/2001

$250,000 Assessed New York Presbyterian Healthcare System’s existing capability of responding to and managing a public health threat, modeling a hypothetical release of anthrax in New York City. Project produced the first scalable computer model of civilian medical response to bioterrorism as well as a model to distinguish anthrax cases from flu cases.

Optimizing clinical preparedness for bioterrorist attacks by using simulation modeling and data from recent anthrax attacks

4/2002–3/2003

$165,000 Refined simulation model under the first task order listed above by creating interactive spreadsheet-based model that incorported national "best practice" data and allowed community-specific tailoring of mass prophylaxis staffing calculations for antibiotic dispensing and vaccination campaigns. (Model is known as BERM.)

National Guidelines for Mass Distribution Centers

9/2002–4/2003

$240,000 Assembled resource materials and developed national guidelines for mass prophylaxis distribution centers to assist national public health authorities. Developed resources such as computer simulation models.

Developing tools for statewide planning and response to bioterrorist attacks

10/2002–9/2003

$338,000 Developed interactive computer model for estimating regional casualty production rates from large-scale anthrax and plague releases, thus improving hospital surge capacity planning. Also completed working group on risk communication around bioterrorism that led to report with specific recommendations for improving incorporation of media into BT planning efforts.

is now available from the AHRQ Web site. It is a “how-to guide” for communities, medical personnel, the media, and government, providing step-by-step information on what organizations are involved, what resources are needed, where and when those resources are needed, and how to organize a clinic for distribution of prophylaxis in the case of an anthrax or similar attack. Given that “no comprehensive guide for Mayor Joe in small town America” has existed up to this point (according to Weill staff), the guide should be a useful planning tool.

Dissemination of Weill’s bioterrorism work has been extensive and has occurred mostly

outside the IDSRN, in large part because of the work’s broader appeal beyond integrated delivery systems (IDSs). Importantly, Weill staff have been able to use some IDSRN funding specifically for dissemination, capitalizing on several dissemination opportunities in a number of forums and at several levels (e.g., both within and outside the state of New York at the national, state, and local levels).

A-11

Table 3. Description of Selected Tools Produced by Weill

Tool What Tool Allows Users to Do

BERM software To calculate the number of staff needed to carry out a prophylaxis campaign for either:

• Antibiotic dispensing

• Vaccination

Users can change assumptions as appropriate to particular situation (e.g., population size, staff limitations, time frame for campaign)

Community-Based Mass Prophylaxis Planning Guide To understand the five components of a mass prophylaxis response to epidemic outbreaks:

• Surveillance

• Stockpiling

• Distribution

• Dispensing

• Follow-up

Computer simulation models for bioterrorism planning To aid in planning for a citywide response to bioterrorist attacks, including optimal distribution of antibiotics to ambulatory patients and improving hospital treatment capacity

Screening protocol to identify potential early inhalation anthrax cases

To aid medical providers in discriminating between inhalational anthrax cases and viral respiratory infections

The BERM software is available through both the American Hospital Association (AHA)

and AHRQ Web sites. To date, the software has been downloaded about 900 times from the AHA Web sites, mostly by state and county bioterrorism planners, but also by some hospitals.18

Other forums for dissemination of Weill’s bioterrorism work have included Denver Health’s

regional bioterrorism workshop (which targets organizations in the western United States), AHRQ’s bioterrorism workshops in San Diego and Atlanta in November and December 2003, AHRQ’s audio conference on bioterrorism tools in April 2002, and the annual meetings of the IDSRN. Information on Weill’s work has also been distributed through the Centers for Disease Control’s (CDC) Strategic National Stockpile electronic listserv, which includes about 1,000 individuals working in bioterrorism response. In addition, in June 2004, Dr. Hupert was featured on a CDC telecast on mass prophylaxis that reached 45,000 people across the country; during the telecast, he described the BERM software.19 The Weill team also trained several staff from the Logistics Branch of the National Pharmaceutical Stockpile at CDC.

18 AHA conducted a Web-based survey to gauge users’ perspectives on the tool and has received 162 responses, which are now undergoing analysis.

19 See http://www.phppo.cdc.gov/phtn/webcast/antibiotic/default.asp.

A-12

Several state and local governmental organizations in New York have used Weill’s bioterrorism simulation models. For example, New York City’s Office of Emergency Management and Department of Health used the simulation models to estimate capacity for the community’s response to an outbreak or attack. Moreover, the New York State Department of Health distributed copies of the BERM software to the state’s 57 counties.

Agencies outside the state of New York, including the Cuyahoga County Board of Health in

Cleveland, the Minnesota Department of Health, and agencies in North Dakota, Florida, New York, California, and Texas, have also made extensive use of Weill’s simulation models. One staff person from the Minnesota Department of Health has been traveling to each county in that state to present the BERM software and teach staff how to adjust the model to reflect local characteristics.

The use of Weill’s tools by New York Presbyterian (NYP), the hospital with which Weill

Medical College is affiliated, has been fairly limited to date. However, the Weill team hopes to build additional relationships with NYP now that it is beginning to focus on the hospital response to bioterrorism. NYP’s chief medical officer and head of the emergency room both plan to be involved in a new bioterrorism project that the IDSRN recently awarded to Weill.

Hupert and colleagues have published papers on this bioterrorism work in Annals of Internal

Medicine, Medical Decision Making, and Archives of Internal Medicine, and have presented the work at meetings of the Society of General Internal Medicine, Society for Medical Decision Making,�and AcademyHealth.

FACTORS FACILITATING AND IMPEDING SUCCESS

Several factors facilitated Weill’s success in developing and disseminating a number of products related to bioterrorism. Most important, Weill’s bioterrorism work was a case of “being in the right place at the right time with the right people,” according to both AHRQ and Weill staff. The immediacy of the need for bioterrorism information after September 11, 2001, combined with the scarcity of existing information, meant that many organizations—both within and outside the state of New York were attuned to developments in the area of bioterrorism response and thus ready to use any available tools. In addition, the Weill bioterrorism work has benefited greatly from an energetic principal investigator in the person of Nathaniel Hupert, who has strived to disseminate the project results to a broad audience. Moreover, continued financial support by the IDSRN for several projects meant that the Weill team could build on previous work and extend applications over time.20

Nonetheless, Weill faced some important impediments in its bioterrorism work. The Weill

team was unable to gain access to the New York City Department of Health’s data related to the fall 2001 anthrax attack.21 As a result, the team had to shift its focus from validating its model

20 Weill’s work on bioterrorism under the IDSRN has led to other work, such as an $840,000 AHRQ grant on

“Modeling U.S. Health Systems’ Epidemic Response Capacity.” 21 As civilians, members of the Weill team have continually faced issues of security clearance.

A-13

with data specific to New York City to developing a more generic simulation model. Consequently, Weill produced a tool that was much more widely applicable and therefore offered greater potential for dissemination than originally planned. Now, the Weill team approaches its bioterrorism work with the goal of producing generic models that can be used by any entity.

At the outset, building relationships with state and local government agencies was also a

challenge, despite the fact that Weill had some existing relationships as a result of preparing for possible computer and other issues at the start of the year 2000 (or “Y2K”). While these relationships have developed and improved over time, maintaining stable relationships with the New York City Department of Health remains difficult given significant staff turnover in that department over the last few years.

KEY LESSONS AND RELEVANCE TO OTHER IDSRN EFFORTS

The timing of the attacks of September 11, 2001, played a large role in the relevance of and demand for the tools, guides, and papers developed in the course of Weill’s bioterrorism work. Moreover, the fact that little information existed on bioterrorism response meant that Weill had a wide audience for its products. As a result, Weill appears to have played a particularly important role in the field of bioterrorism planning and response over the last few years.

Another important facet of Weill’s work is the creation of generic models that can be

applied to any geographic area, not just New York, thereby making the work useful to a broad audience. Though other IDSRN tools share the feature of generalizability to some extent, such a feature is significant for other IDSRN projects, highlighting the importance of work that moves beyond the idiosyncrasies of a single area or delivery system.

A-15

CASE STUDY 2: LOVELACE CLINIC FOUNDATION’S GUIDES ON CULTURAL COMPETENCY

PROJECT IMPETUS

As the nation’s population becomes more diverse and large purchasers continue to focus on care delivery as a way to improve health care outcomes, cultural competency has emerged as an area of growing concern within the Centers for Medicare and Medicaid Services (CMS) and AHRQ. During the late 1990s, CMS sought to enhance quality improvement activities in Medicare managed care plans by building on the provisions of the Balanced Budget Act of 1997. Accordingly, CMS now requires Medicare plans to undertake annual quality assessment and performance improvement (QAPI) projects. After deciding on a racial/ethnic focus for QAPI projects for 2003 (with plans required to choose between addressing culturally and linguistically appropriate services or clinical health care disparities), CMS set out to develop a tool for plans to use in pursuing projects related to cultural competency. With its focus on organized delivery systems and translating research into practice, AHRQ’s IDSRN offered a vehicle that permitted CMS to support Medicare plans by sponsoring development of a tool through the IDSRN. At the same time, AHRQ had an interest in racial and ethnic disparities. When this topic became a departmental priority in the late 1990s, AHRQ saw cooperation with CMS as a means to expand knowledge generally on how health plans might make their systems more sensitive to cultural and linguistic issues as related to the provision of high-quality care.

BACKGROUND

Using both CMS funding and IDSRN core funding, AHRQ awarded two projects related to culturally and linguistically appropriate services (CLAS) to Lovelace Clinic Foundation (Lovelace), a not-for-profit research organization that works closely with Lovelace Health System. Funding for the projects totaled almost $300,000. The first project involved the development of two CLAS guides for Medicare managed care plans while the second focused on tracking the influence of the guides among these plans (see Table 4).

Table 4. Relevant Projects for Lovelace Clinic Foundation Case Study


To model culturally and linguistically appropriate service (CLAS) quality assessment and performance improvement projects (QAPI) for Medicare+Choice organizations

10/2001–9/2002

$250,000 Development of two guides to help managed care organizations plan for and provide culturally and linguistically appropriate services to Medicare+Choice patients.

Knowledge transfer and use of CLAS guides

9/2003– 6/2004

$49,000 Follow-on analysis to track use of CLAS guides by Medicare plans.

A-16

PURPOSE OF THE CLAS GUIDES AND HOW THEY ARE USED AND DISSEMINATED

The purpose of the first CLAS project was to produce two guides that provided detailed instructions on (1) meeting the language needs of members with limited English proficiency and (2) conducting cultural planning and assessment. The guides, each of which exceeds 100 pages, describe processes in a step-by-step fashion; for example, the planning guide outlines items such as what a health plan needs to think about when developing a CLAS assessment team and what type of CLAS data should be presented at a briefing of a plan’s senior leadership.22 Under the second project, Lovelace investigated the perceptions of the CLAS guides among eight plans that had used the guides. The latter project did not evaluate the reach of the guides among Medicare plans generally but rather spoke to their effectiveness among a small group plans.

According to the final report from the second CLAS project, plans found the guides helpful

in carrying out their QAPI projects. Plans with small minority enrollment described the guides as a case of “overkill” or an “embarrassment of riches” in regards to the information provided (Lovelace Clinic Foundation 2004). However, plans with larger minority enrollments and/or that were further along with CLAS thought the guides were more useful. Use of the CLAS planning guide (which is more general) was much more common than use of the language guide (known more formally as the “oral linguistics” guide).23 About half of the respondents interviewed for the second CLAS project reported that the CLAS guides “influenced their decision to select a CLAS project” as opposed to a disparities project for their QAPI requirement. Plans also reported that the guides provided concrete tools and examples to use in developing their QAPI projects and/or confirmed the direction in which they were headed (Lovelace Clinic Foundation 2004).

After the completion of the first project in late 2002, CMS sent one hard copy of each guide

to all Medicare managed care organizations. The guides were available for download from the CMS Web site as of November 2002, with summaries of the guides posted on the AHRQ Web site in February 2003; the guides are still available on these Web sites.24

The guides have also been disseminated extensively via a series of free regional workshops

on CLAS conducted by faculty at the University of North Carolina (UNC) School of Public Health and sponsored by CMS, with supplemental funding from the Office of Minority Health at HHS and the Robert Wood Johnson Foundation. (Note that UNC’s work was completely

22 To aid in the development of the guides, the project staff at Lovelace created an advisory board of CLAS

experts with whom they spoke every month. They also engaged in highly structured tasks, including a literature review, interviews with “model” health plans about their CLAS activities, and interviews with experts in CLAS. These tasks provided important background information for developing the guides.

23 A University of North Carolina faculty member who conducted workshops on CLAS and distributed the CLAS guides noted that some individuals perceived the language guide as repetitive of the more general CLAS guide.

24 AHRQ cannot post the guides in their entirety because the guides are not Americans with Disabilities Act (ADA) compliant. However, AHRQ does offer a link to the CMS Web site, which provides the guides in their entirety.

A-17

independent of the IDSRN.) More specifically, UNC conducted five workshops in Albuquerque, Chicago, Orlando, Philadelphia, and San Francisco. While the workshops were targeted at health plans, participants included representatives of hospitals, government agencies, health departments, community health centers, and advocacy groups. Attendance varied from 22 (Orlando) to almost 70 (San Francisco), with total participation of about 275. The workshops appear to have been an important means for informing plans about the CLAS guides, given that, according to UNC faculty, most attendees did not already know of the guides’ existence. Evaluation forms completed by workshop participants indicated that the guides were “extremely helpful as tools for assistance and support of CLAS efforts” (Havala Hobbs 2003).

Although the guides were originally prepared for Medicare managed care plans, Lovelace

staff reported that other plans also found the guides useful. Some Medicaid plans have used the guides, though such plans are typically already more advanced in the area of cultural competency as compared with Medicare plans. In addition, Cindy Brach at AHRQ sent the guides to Medicaid directors in all states.

The Lovelace Health System also found the CLAS guides useful, according to Lovelace

Clinic Foundation’s Margaret Gunter (the project’s principal investigator). Gunter believes the guides have raised awareness of cultural competency in Lovelace Health System; in fact, the health system recently created a diversity council, which Gunter sees as a response to the work on CLAS.


Several factors facilitated the success of the CLAS projects. The Lovelace team felt that its interactions with CMS and AHRQ were highly positive; one member of the project team described the work as “a real partnership.” Cindy Brach at AHRQ played an important role as the project’s task order officer; not only did she provide substantive experience in the area of cultural competency, but she also took it upon herself to disseminate the work to interested parties, describing herself as “on a personal mission of dissemination and impact.” Although the tool was available on CMS’s Web site, Brach saw to it that summaries of the guides were also available on AHRQ’s Web site.

A few factors made the development of the CLAS guides, and later the study of them,

somewhat challenging. While the accomplishments of the CLAS projects were in keeping with the projects’ original objectives, the Lovelace team members stated that they did not know what they were “getting into” in terms of the amount of work involved in developing a tool that would be both useful and easily accessible to plans. In fact, Margaret Gunter from Lovelace described the development of practical guides as “not our usual thing” and “more difficult than writing up a research paper.” Lovelace hired special editors to ensure that the guides were useful and user-friendly. The time and money spent by Lovelace on the first CLAS project far exceeded the contract award; nonetheless, the Lovelace team members “felt it had to be done” because they wanted the guides to be as useful as possible. In addition, because CMS wanted to produce and release the tools in time for Medicare plans’ 2003 QAPI projects, Lovelace was under tight time constraints and had to deliver on schedule. The fact that Lovelace’s expertise lay in producing research papers rather than practical guides meant an extra challenge in terms of making the guides as useful and practical as possible.

A-18

In the second project, two factors impeded a more complete assessment of plans’ perceptions and use of the CLAS guides: limited funding and the need for OMB clearance to survey a large number of plans. While the report produced under the project provides useful information, it cannot speak to the use and usefulness of the guides among the entire target audience (Medicare managed care plans).


The partnership between AHRQ and CMS was an important component of the project. The fact that the CLAS guides produced by Lovelace were directly tied to plans’ needs based on CMS’s QAPI requirements meant that the guides had a clear operational link. While the impact and extent of use of the guides remains difficult to measure, the CMS link proved extremely important in giving the work an operational focus and instantly creating a target audience. At the same time, the guides allowed CMS to provide plans with needed tools, thereby helping to defuse resistance to federal requirements (Lovelace Clinic Foundation 2004).

The audience for some tools produced under the IDSRN may be broader than originally

expected (Havala Hobbs 2003). UNC faculty involved in the CLAS workshops noted that the guides have proved useful not only to Medicare plans (the original target audience) but also to many health care organizations (e.g., hospitals, community health centers) interested in pursuing CLAS activities. Similarly, Lovelace reported that health plans other than Medicare plans have expressed interest in the guides. Based on this information, additional outreach to other organizations may be warranted. While, to our knowledge, no additional work is currently planned on the CLAS guides, it may make sense to continue to disseminate the existing guides to a broad audience.

A-19

CASE STUDY 3: MEDICATION INFORMATION TRANSFER WORK BY RTI AND PROVIDENCE HEALTH SYSTEM

PROJECT IMPETUS

With growing concerns over the fragmented health care system, questionable quality of information systems, and related medical errors, AHRQ has identified patient safety as one of its priority areas. Transition points in the delivery of care have recently received attention because of the potential for poor outcomes that may result. A prominent example of a transition problem is the failure to document medications fully and/or accurately when patients move from one setting to another, such as from inpatient care to home or a nursing home.

BACKGROUND

RTI International (RTI), along with Providence Health System (Providence) Portland service area, was awarded two projects totaling $500,000 on the transfer of medication information across patient care settings.25 During the first project, RTI and Providence developed a model that estimated that use of an e-medication list could result in as much as a 50 percent reduction in medication errors across six steps in the inpatient care process—with admission and discharge representing the most extreme vulnerability points. The second project, which is still in progress, evaluates the effectiveness of a transition pharmacist at Providence Portland Medical Center hired in accordance with the findings from the first project; the latter project is scheduled for completion by mid-January 2005.26 Table 5 describes the projects.

Table 5. Relevant Projects for RTI/Providence Case Study


Integrated delivery system solutions for transferring medication data across patient care settings

10/2001– 10/2002

$200,000 Produced model showing that a tool such as an e-medication list could prevent up to 50 percent of medication errors.

Estimating risk reduction and cost of enhancing medication information transfer across patient care settings

5/2003– 1/2005

$300,000 Follow-on project to evaluate the effectiveness of a transition pharmacist hired by Providence to pursue the strategy of an e-medication list developed in the previous project.

25 Intermountain Health Care and UNC Health Care were peripherally involved in the first project. The

Portland-area Kaiser Permanente Group was supposed to be involved in the second project, but recent contracting changes meant that Kaiser patients are no longer seen by Providence Portland Medical Center and therefore were no longer a feasible control group.

26 The project was originally scheduled to be completed by October 2004 but was granted an extension through January 2005.

A-20

PURPOSE OF MEDICATION INFORMATION TRANSFER WORK AND HOW IT IS USED AND DISSEMINATED

The purpose of the first project was to develop an in-depth understanding of the medication information transfer process and to identify where mistakes are most likely to occur. The RTI/Providence team conducted focus groups and key informant interviews with health care professionals at three integrated delivery systems (Providence, Intermountain Health Care, and UNC Health Care). The team used the collected data to generate a detailed process model that identifies the points between admission and discharge at which medication errors are most likely to occur. This was supplemented by qualitative case study data that was systematically collected at each of the three sites. The team then developed an evaluation framework to assess medication information transfer practices, essentially paring down what it had learned into a set of fundamental principles. Finally, the team proposed several solutions for transferring accurate medication information between inpatient and ambulatory care settings. The solutions included (1) adding an electronic medication administration record to software already in use—thereby incorporating the patient’s medication list into both the outpatient electronic medical record and the inpatient system—and (2) adding a transitional care clinical pharmacist to the hospital staff, given that the greatest risk for error is associated with admission and discharge.

For the follow-on project, Providence hired a transition pharmacist charged with both

serving as a bridge between admission and discharge and using software to ensure that the same medication list was entered into both the inpatient record at discharge and the ambulatory care record. To be eligible for the intervention, patients had to be a Medicare beneficiary enrolled in Providence Health Plan, receive ambulatory care from Providence Medical Group, and be receiving inpatient care from a hospitalist at Providence Portland Medical Center, an acute care facility with 350 beds. Patients who met the eligibility criteria and provided informed consent were randomized into one of two groups: the treatment group received standard services with the aid of the transition pharmacist, whereas the control group simply received standard services.

As of September 2004, a relatively low number of patients had enrolled in the intervention

(about 80 to 90 each in the treatment and control groups), in large part because of the informed consent process. Moreover, some contamination of the control group likely occurred simply because the transition pharmacist has been well integrated into the hospital’s care management team and has undoubtedly sensitized physicians and nurses to medication issues, perhaps leading to better medication lists and discharge summaries for all patients. Therefore, no change in outcomes (e.g., readmission rates, adverse events) as a result of the treatment has been detected to date. To improve the study’s statistical power, however, the team is now considering other control groups such as patients who received care before the intervention and patients at other Providence hospitals.

To date, the project has been limited to Providence Health System and, in particular, to

Providence Portland Medical Center, where the transition pharmacist is employed. Dr. Bruce Bayley (principal investigator at Providence) has disseminated findings from the first project to other Providence hospitals, including, among others, presentations to the Providence Portland Patient Safety Committee and the Interdisciplinary Planning group for the JCAHO National Patient Safety goal (discussed more later). In addition, RTI has presented the findings to

A-21

Intermountain Health Care, UNC Health Care, UPMC Health System, and Baylor Health System using funding from an AHRQ Partnerships for Quality grant.27

Some dissemination has also occurred outside RTI’s IDSRN team. A Seattle hospital has

begun to support discharge planning and transition by using extensive information sharing; according to Gordian Solutions (a hospital consulting firm), the decision on the part of the Seattle hospital was a response to a presentation delivered by Dr. Lucy Savitz on the work of the first project. Also, the project was presented at the annual IDSRN Program meeting in the 2002; and the FMEA method was demonstrated via presentation of project results at the University of North Carolina’s Program on Health Outcomes Research Day in August 2003 by Dr. Savitz. Ms. Teresa Maddalone, the PHS transition pharmacist, presented the project to the Oregon Society of Health System Pharmacists in May 2004. The project was also highlighted at the AHRQ Translating Research into Practice meeting by Dr. Bayley as part of a panel discussion led by Dr. Savitz on conduits and linking agents for accelerating the adoption of new information. Drs. Bayley and Savitz presented the study design at AHRQ’s Patient Safety Conference in September, 2004. Finally, the team has a publication forthcoming in AHRQ’s Advances in Patient Safety: From Research to Implementation entitled “Barriers Associated with Health Care Handoffs” (by Bayley, Savitz, et al.).


One of the most important aspects of the RTI/Providence project is that it is operationally directed, “demand-driven” research. Providence originally pursued the project because its primary care physician group wanted to reduce hospital readmission rates. As a capitated physician group, the physicians had a strong interest in better managing illness to prevent readmission. According to Bruce Bayley, the physician group knew that hospitalists focused on the hospital stay itself and on stabilizing the patient (and less on longer-term concerns such as management of chronic illness) and suspected that chronic illness medications were sometimes put on hold during the hospital stay. This project then was of keen interest to Providence staff, especially the physician group. Moreover, the physician group was not the only group within Providence focused on reducing readmissions: the Providence Health Plans have exerted pressure to reduce hospitalization rates and financial officers at Providence hospitals have expressed concern with readmissions because of overcrowding in Providence hospitals.

The leveraging of other available resources is another important aspect of the project.

Concurrent with the first project under the IDSRN, pharmacy residents at Providence were conducting two additional pilot projects: one on the effect of pharmacist-conducted inpatient education and counseling and another on the effect of a clinical pharmacist working closely with hospitalists. Although funded by Providence (rather than by AHRQ), the work helped provide evidence that a clinical pharmacist focused on transitions in care at admission and discharge could have a positive impact on inpatient care. Pharmacy residents benefited from the IDSRN project team’s guidance and feedback on the residents’ research while the project team gained knowledge from the pilot projects, leading to a “win-win” situation for both groups.

27 The Partnerships for Quality grant has enabled the RTI team and its integrated delivery system partners to

meet four times per year and present findings to one another.

A-22

A recent development encouraging interest in the project is the Joint Commission on Accreditation of Healthcare Organizations’ announcement that medication reconciliation is a safety objective for hospitals in 2005–2006.28 (Specifically, the goal is to “accurately and completely reconcile medications across the continuum of care.”) While senior management at Providence was already interested in medication issues, the Joint Commission’s announcement has further crystallized interest. One Providence staff member described its patient safety committee as “chomping at the bit for information” from the second project. Moreover, all hospitals in the Providence system have recently approached Bruce Bayley and others on the team to ask for guidance and information on how to proceed in meeting the Joint Commission’s new safety objective.

Interventions in real-life settings always face a number of challenges or barriers. Providence

staff noted that, even within an integrated delivery system, the focus is within settings rather than across them. The computer systems between laboratory and pharmacy are not integrated, and many systems are outmoded and awkward. Moreover, maintaining accurate outpatient records is difficult; one staff member reported that records are “maybe 75 percent accurate.” These factors contribute to medication errors and make it difficult to improve medication information transfer.

Finally, despite senior management’s strong interest in medication information transfer, it is

crucial to make the “business case” for such an intervention; failure to do so may mean limited or no internal funding. Based on findings of the first project, the pharmacy director at Providence Portland Medical Center proposed to senior management that the Center budget for and hire a transition pharmacist. However, senior management remained unconvinced of the business case for the pharmacist and did not fund the position. (Providence and RTI subsequently obtained IDSRN funding for the second project, with the transition pharmacist’s salary paid 50 percent by IDSRN funding and 50 percent by Providence for one year.) The project team hopes to make the business case for a transition pharmacist with the second project.

KEY LESSONS AND RELEVANCE FOR OTHER IDSRN EFFORTS

The fact that AHRQ provided follow-on funding for the second project is important; that funding has allowed the team to extend its work and make it operational. According to one project team member, “When AHRQ sees what is done and the results are interesting, they are amenable to seeing how the results can be diffused. Medication information transfer is a nice example of that.” Moreover, several members of the project team described the second project as the most operational project on which they had worked under the IDSRN. It is important to note that AHRQ has funded similar work by another IDSRN team; specifically, Weill Medical College collaborated with the Visiting Nurse Service of New York City to improve transitions between hospital and home for heart failure patients. The findings presented here may be relevant to that work.

The fact that the medication information transfer projects were motivated by a need within

Providence to reduce readmissions and more fully understand their root cause has meant that senior management has been attuned to the project and interested in the results. Yet, even though

28 JCAHO has asked hospitals to begin developing potential solutions in 2005 for implementation in 2006.

A-23

the research was “demand-driven,” it still needs to develop a strong business case to support continued work (e.g., permanent staffing of a transition pharmacist).

One of the most interesting (and serendipitous) developments during the project was the

Joint Commission’s announcement of medication reconciliation as a safety objective for 2005–2006, which increased Providence Health System’s senior management’s interest in the project. The project is likely to continue to garner attention as the Providence hospitals strive to meet the Joint Commission’s new objective in the next two years. The study results are also likely to have widespread utility beyond the IDSRN as hospitals across the U.S. struggle with the Joint Commission’s medication reconciliation recommendation.

A-25

CASE STUDY 4: STUDY OF RACIAL AND ETHNIC DIFFERENCES IN QUALITY AND OUTCOMES BY RAND AND CHCPE

PROJECT IMPETUS

Disparities in health care quality and outcomes by race and ethnicity have long been believed to be large, but it was often difficult to measure those disparities because health plans and other entities generally lacked racial and ethnic data on their members or populations served. Concerns surrounding disparities have grown over time, and recent research has indeed confirmed large and significant gaps between population subgroups. After years of reluctance on the part of plans to collect data on member’s racial and ethnic background (in large part because of legal concerns), several plans started to collect such information within the last three to five years, marking an important change.29 To enhance the capability of integrated delivery systems to collect the information and track utilization and quality by race and ethnicity, AHRQ has funded several projects under the IDSRN.

BACKGROUND

RAND and the Center for Health Care Policy and Evaluation (CHCPE) pursued two projects in the area of racial and ethnic disparities, with total funding of $263,000 (Table 6).30 The first project investigated racial and ethnic differences in cardiovascular disease and diabetes measures by using claims and enrollment data from commercial and Medicare plans. The second project focused on developing a tool with various graphics and displays for use by plans to assess whether racial and ethnic disparities exist.

PURPOSE OF WORK ON RACIAL/ETHNIC DIFFERENCES AND HOW IT IS USED AND DISSEMINATED

Under the first project, RAND and CHCPE examined Health Plan Employer Data and Information Set (HEDIS) measures on cardiovascular disease management and diabetes management by race and ethnicity (as well as by socioeconomic status and gender) in order to identify whether and where disparities exist. Using claims and enrollment data for over 2 million privately insured persons in 10 commercial plans and almost 200,000 Medicare beneficiaries in nine Medicare plans, the analysis revealed numerous disparities by race/ethnicity for both cardiovascular disease and diabetes HEDIS measures. Moreover, the disparities were widespread across the plans studied.

29 America’s Health Insurance Plans conducted a survey of health plans in 2003–2004 and found that over two-

thirds of those plans collecting data initiated their programs in 1998 or later (and over one-third started as recently as 2002 or 2003), suggesting that such data collection is a relatively new endeavor (America’s Health Insurance Plans 2004).

30 The team was also awarded a project to investigate whether differential diffusion of medical technologies exists across racial/ethnic groups; the project is not included as part of the case study.

A-26

Table 6. Relevant Projects for the RAND/CHCPE Case Study


Socioeconomic, racial/ethnic, and gender differences in quality and outcomes of care as related to cardiovascular disease

9/2000–4/2002

$200,000 Examined claims and enrollment data from commercial and Medicare plans to determine whether disease management varied within and across settings and care models for patients of different socioeconomic status, race/ethnicity, and gender. Found many significant disparities.

Refining a tool to assess racial/ethnic and socioeconomic status disparities in health plan performance on HEDIS measures

9/2003–03/2004

$63,000 Refined the race/ethnicity tool (in the form of an Excel spreadsheet) for use by plans. Gathered feedback from several plans to make the tool as user-friendly as possible.

The idea of developing a display tool to aid in looking at and understanding racial/ethnic data evolved during the first project. As RAND and CHCPE “worked through the data, there were reams and reams, and it was hard to look at.” Therefore, the team created a tool to examine data visually (in the form of bar charts and graphs) and thereafter realized that the tool could be useful for others examining racial/ethnic disparities. Thus, the purpose of second project was to create a user-friendly tool for use by plans to enable quality improvement (QI) staff and others to explore the data interactively.

To gather feedback on the tool, the project team interviewed QI staff at several UnitedHealth

Medicare plans and at Molina, a Medicaid-only health plan. While the tool’s display components were useful (providing easy-to-read graphics), plans found the tool “clunky” and experienced difficulty with data input. In addition, Molina, whose membership includes many racial/ethnic groups, found the tool to be limited (e.g., allowing for examination of only white versus black, not of Asians) and not sufficiently user-friendly for application by nurses or other nonresearch staff.31 One member of the project team admitted that “the tool looks like a beautiful car but under the hood it’s not so pretty.” Nonetheless, plans found the tool helpful, if not completely refined. According to RAND staff, the medical director of one plan was particularly pleased with the fact that the tool could help teach QI staff to look at data more rigorously and think about topics such as confidence intervals.

The work has begun to have substantial influence outside the IDSRN.32 In the last year,

several plans told AHRQ of their interest in tackling health care disparities. As a result, AHRQ and the Robert Wood Johnson Foundation are cosponsoring a health plan disparities collaborative. The collaborative, which now includes 10 plans, met for the first time in mid-September 2004. (Both UnitedHealth and Molina are participating in the collaborative.) Under the collaborative, AHRQ will provide technical assistance to participating plans on an as-needed basis to help the plans start measuring disparities among patients with diabetes. The

31 Although the tool was developed to examine outcomes related to cardiovascular disease and diabetes,

Molina used it to look at prenatal and postpartum outcomes.

32 AHRQ plans to include a link to the tool on its Web site in the near future but cannot post the tool directly because it is not ADA-compliant.

A-27

collaborative—which largely focuses on commercial and Medicare populations but includes one Medicaid-only plan will explore several ways to track disparities, including use of the tool developed by RAND and CHCPE.33 Participating plans will gather baseline data on racial/ethnic disparities, develop an intervention targeted to the disparities observed at baseline, and measure both process and clinical outcomes over time to determine whether improvements result from plan interventions. One AHRQ staff member expects the tool to be “fairly widely used among the 10 [participating plans].”

RAND has received inquiries about the tool from plans outside the IDSRN or the disparities

collaborative, especially California Medicaid plans. In fact, RAND has conducted a “demo program” for the interested parties. The state Medi-Cal agency has also expressed interest in the tool.

The two projects resulted in a paper on gender disparities in cardiovascular and diabetes care

published in Women’s Health Issues and several presentations at the annual meetings of Academy Health and the Society for General Internal Medicine. An upcoming special issue of Health Affairs will highlight the tool. Moreover, a recent publication on disparities by the National Academy of Sciences cited the final report from the first project (as well as other work by Allen Fremont at RAND) (National Research Council 2004).


Numerous factors have created a “consistent drumbeat moving [disparities research] forward,” according to one member of the project team. RAND and CHCPE staff (Fremont, Shah, and Horstman 2003) have identified several reasons for the growing momentum on race/ethnicity issues:

• Several reports focusing on and increasing the body of evidence on disparities, such as the Institute of Medicine’s Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care

• CMS’s provision of members’ race/ethnicity data to plans

• CMS’s quality assessment and performance improvement initiative, which required all Medicare plans to pursue either a disparities or cultural competency project in 2003

• Increased recognition by plans that race/ethnicity can be approximated by using several methods such as geocoding and surname analysis

• Emerging consensus that collecting racial/ethnic data from plan members is legal

33 In fact, Allen Fremont from RAND is closely involved with the collaborative. RAND has received another

contract from AHRQ (independent of IDSRN awards) to pursue this work.

A-28

In addition, large plans such as Aetna have begun to collect racial/ethnic data on their members. According to one member of the project team, plans are starting to realize that they will have to collect these data sooner or later and that their competitors are already doing so.

Some purchasers mainly large employers are increasingly interested in the question of

disparities and see value in pursuing study of this area. This development also has contributed to increased interest on the part of CHCPE staff.

One important challenge facing the project is that sound tools take several iterations to develop and refine. The project team sees the race/ethnicity tool as requiring one more iteration to make it more user-friendly and hopes that the work will be funded under the IDSRN in the near future. The team views the collaborative as an opportunity for several additional plans to test the tool and provide feedback.


The topic of racial and ethnic disparities has evolved from one that plans were afraid to address because of possible legal ramifications to one that they feel they must begin to address because of increased national attention to disparities, growing comfort with the legality of collecting and using racial/ethnic information, and a growing sense that the competition is beginning to track the information and use it to improve quality.

RAND and CHCPE’s realization that the tool could be useful to other plans coincided with

the IDSRN’s evolution toward more practical, applied work. AHRQ’s willingness to fund the follow-on work was crucial in ensuring that the work moved beyond something used as an aid in a research project to a practical tool applicable to a broad audience.

AHRQ’s use of the IDSRN tools in other non-IDSRN venues is an important lesson growing out of the project. The newly formed disparities collaborative is an appropriate forum through which to disseminate the display tool developed by RAND and CHCPE. It will be important for AHRQ and the project team to track use of and collect feedback on the tool as the collaborative moves forward. AHRQ, RAND, and CHCPE might also work together to consider additional venues for the tool in order to disseminate it to a larger number of plans.

A-29

CITATIONS

America’s Health Insurance Plans. 2004. http://www.ahip.net/content/pressrelease.aspx?docid= 112, accessed September 22, 2004.

Fremont, Allen, Mona Shah, and Thomas Horstman. 2003. Refining a Tool to Assess Racial/Ethnic and SES Disparities in Health Plans’ Performance on HEDIS Measures. Proposal in Response to AHRQ Request for Task Order.

Fremont, Allen, Steve Wickstrom, Chloe Bird, Jose Escarce, Thomas Horstman, Elizabeth McGlynn, Thomas Rector, and Mona Shah. 2002. Socioeconomic, Racial/Ethnic, and Gender Differences in Quality and Outcomes of Care as it Related to Cardiovascular Disease. Final Report for Contract #290-000-0012. Prepared for the Agency for Health Care Research and Quality.

Havala Hobbs, Suzanne. 2003. A Practical Guide to Culturally and Linguistically Appropriate Services in Health Services Delivery: Summary and Recommendations. Final Report to CMS.

Lovelace Clinic Foundation. 2004. Evaluation of the Usefulness of CLAS Guides to M+CO Plans: Facilitating Knowledge Transfer and Utilization of IDSRN Products and Tools Task Order. Draft report to AHRQ.

National Research Council of the National Academies. 2004. Eliminating Health Disparities: Measurement and Data Needs. Michele Van Ploeg and Edward Perrin, eds. Washington, DC: National Academies Press.

APPENDIX B

TASK ORDER CHARACTERISTICS

Table B-1. Task Order Summaries By Team# Title Dates Amount Project Description Findings/Outcomes Dissemination Actual changes in practice as a

result of task orderOther comments

ABT ASSOCIATES (as provided in memo from AHRQ) (based on information from AHRQ newsletters; AHRQ impact memo;

interview notes.

(based on information from interviews)

1 Impact of EpiCare on the management of diabetes in the Geisinger Health System

10/1/2001 - 7/31/2003

$214K A randomized controlled experiment was conducted among patients of the Geisinger Health System in Pennsylvania to determine the effect of electronic medical system automated reminders on compliance with standards of care for diabetes.

Study found no change in performance of physicians (e.g., procedures, preventive activities) given the electronic alerts, relative to those who did not receive them. Physicians did not like the automated alert system.

Presentation at the first annual meeting of the IDSRN in 2002. Presentation to Geisinger management. Proposal indicated that project would include a peer-review journal article, but we were unable to find in the literature.

Stopped using physician electronic alerts except for those at very high risk; they are now moving towards providing non-intrusive reminders. Also, has been applied broadly, well beyond just diabetes care.

2 Use of electronic communication among patients and providers to improve quality of care (MyChart project)

6/19/2002 - 8/18/2003

$200K Abt Associates and Geisinger Health System assessed how Geisinger’s electronic medical record for ambulatory care and other electronic communication, such as patient access to portions of his/her medical record, can improve quality and other aspects of health care.

Compared to non-users, MyChart users were concentrated in the 46-64 age range and had more clinic visits. However, users with many clinic visits did not use MyChart more. About 43% of patients who registered for MyChart never activated their accounts.

A paper is forthcoming in the Journal of the American Informatics Association (November 2004). Also seem to have submitted a manuscript to the British Medical Journal . Presentation to Geisinger management.

Retrained physicians and office staff in using MyChart and made MyChart more user friendly based on patient feedback. Currently in the process of advertising MyChart heavily and hope to get 80,000 new patients to use MyChart in the next year.

Geisinger will publish a book entitled Implementing an Electronic Health Record in November 2004.

3 Modeling techniques for transitions of care in integrated delivery systems

7/22/2002 -11/21/2003

$250K Abt Associates and Geisinger Health System examined transition of care factors that contribute to risk and hazards associated with iatrogenic injury to patients as they pass through a series of complex subsystems of medical care.

Outcomes not known. [This TO was not a focus in interviews.]

One manuscript was supposed to be produced; unknown whether it has been and whether submitted somewhere for publication. It is likely that Abt also presented this research to Geisinger; Geisinger management has been fairly involved with the TOs. TOO said PI is presenting this project in Berlin this summer.

Actual changes not known. TOO thought the project would be valuable since it is a first effort to use risk modeling for health care and would add to what is already known about transitions of care. He thought it demonstrates potential usefulness of technology, but its usefulness would be as an instrument that predicts points of vulnerability and it’s not there yet. [This TO was not a focus in interviews.]

Page 1



4 Access to health care services for the rural elderly

9/1/2002 -8/31/2004

$279K Geisinger Health System is identifying elderly persons at risk for falls and comparing three different approaches to reducing falls in those living in the community: 1) a medical record intervention that notifies the clinician of the patient’s increased fall risk and directs him/her to an evidence-based fall prevention practice guideline located on the EMR; 2) both the medical record intervention and a prompt to refer the patient to a Geriatric Assessment Clinic and receive feedback; and 3) a community-based care management intervention that includes assessment by a case manager and referral for visual screening, physical therapy evaluations, or geriatric assessment clinic as deemed appropriate. A control group of patients receives usual care from their physicians.

Outcomes not known. Unknown. Proposal indicates that dissemination should include a presentation to Geisinger management. TOO said the audience is anyone who deals with care for the elderly; anyone who has the ability of an EMR to flag when someone is at risk.

Unknown.

5 A systematic regional inventory of critical resources to respond to bioterrorism

10/1/2002 -12/31/2003

$478K A web-based data repository, accompanied by a "process" manual, is being developed to serve as a supply inventory to improve preparedness for a bioterrorist threat. The tool will be available for others who wish to adapt it for their own needs. The geographic area of interest from which data are drawn to populate the supply inventory is the Geisinger catchment area in rural Pennsylvania.

See TO#6 below. (For TO #5, they built a tool with tabular reports. TO #6 refined the tool, adding a mapping feature (making it more visual instead of tabular).

See TO#6 below. See TO#6 below. See TO #6 below.

6 Adding mapping capabilities to the bioterrorism inventory tool

10/16/2003 -2/5/2004

$49K Note that this TO builds on Abt’s TO #5. Abt and Geisinger Health System are adding mapping functions to a web-based bioterrorism preparedness inventory tool previously developed through the IDSRN. The mapping capabilities will allow users to overlay geographic features and demographic data for large parts of the state of Pennsylvania; print maps; zoom in and out of mapped areas; determine travel distances; access additional information and find a particular map location (address / town / county / zip code) and re-position the map to the center of that location.

Main product is a web-based software tool for others to use.

Results were presented at third annual IDSRN meeting, presentation to "interested parties in surrounding New England states such as Vermont and Connecticut. Also presented to the Pennsylvania Office of Homeland Defense, the Regional Emergency Management and Bioterrorism Planning Committee, the Office of Information Technology in the Governor’s Office of Administration, and the PEMA Regional Counter-Terrorism Task Force Symposium. Also presented at an AHRQ-sponsored conference on biotterorism in May 2004; Seventh Annual Internat’l Crime Mapping conference in April 2004. TOO said they are working on putting the tool on the Web. The tool would need to be adapted to local areas since it uses resources in the Pennsylvania area.

Tool is available for download from Abt’s web site and may be soon from AHRQ’s web site.

CHCPE/UNITED

Page 2



7 Socioeconomic, racial/ethnic and gender differences in quality and outcomes of care as related to cardiovascular disease

09/18/00 - 4/15/2002

$200K CHCPE and RAND assessed disparities in health care quality for cardiovascular disease and diabetes, a cardiac risk factor, using claims and enrollment data from commercial and Medicare plans affiliated with UnitedHealthcare and HEDIS and non-HEDIS quality measures. Variations were determined for socioeconomic status, race/ethnicity and patient gender in the management of cardiovascular disease and its risk factors within and across different settings and models of care.

Examined claims and enrollment data from commercial and Medicare plans to determine whether management of cardiovascular disease varied within and across settings and models of care, for patients of different SES, race/ethnicity, and gender. Found many significant disparities, even after controlling for other factors. Moreover, disparities were not limited to a few plansTo inform policy makers and health services researchers about the extent and nature of disparities in managed care plans.

Results presented at the annual meeting of the Society of General Internal Medicine in May 2002 and at the Academy Health meeting in June 2002.

United has started to look at race/ethnicity data more; large employers like that they are looking at it.

8 Private sector data and measures for the National Quality Report

09/18/00 - 9/30/2002

$400K CHCPE evaluated the feasibility of using data from private health care organizations in developing AHRQ’s annual National Healthcare Quality Report. This report, the first of which is to be published in 2003, will provide a comprehensive picture of the quality of the Nation’s health care systems.

Conceptualized quality measures and ran the data. Additional outcomes of project are not known.

Information was not disseminated. Little to nothing.

9 Value-based contracting: understanding what is behind the hospital volume-outcome link

9/25/2000 - 10/5/2002

$250K CHCPE conducted exploratory research to identify specific processes of care, related or unrelated to volume, which may explain variations in outcomes of certain complex procedures. (Included lit review and large number of qualitative interviews.)

Lit review indicated that enthusiasm for the volume-outcome link is disproportionate to evidence; moreover, NIS data analysis indicated few statistically significant differences between volume/mortality between high and low volume hospitals. [Note: this TO was not a focus of interviews.]

Presentation at the HMO Research Network conference in April 2002 (Frytak); presentation at the Minnesota Health Services Research conference in 2002. TOO said CHCPE has no manuscript yet because while the aim had been a short brief and longer AHRQ publication, AHRQ changed its mind and wanted CHCPE to publish independently.

Not clear that any have occurred. [Not a focus of interviews though.]

10 Does differential diffusion of new innovations contribute to disparities in health care?

10/1/2001 - 1/15/2003

$200K CHCPE and RAND assessed whether differential diffusion of cardiac technologies over time contributed to observed racial, ethnic and socioeconomic disparities in health care. The study included 7 years of claims data from commercial managed care plans affiliated with UnitedHealthcare. Eight cardiac technolgies in varying stages of diffusion were assessed.

Disparities are much larger for newer technologies; little disparity exist for older technologies and the gaps are narrowing over time.

Indicated in April 2004 that the paper was not yet done (not that TO has been over for some time). Presented at IDSRN annual meeting. TOO also thinks the project guided AHRQ’s development of The Learning Collaborative to look at disparity issues. TOO also said the target audience for the project was researchers so it would be of interest to an academic audience.

Not clear that any have occurred. TOO said the final product had general recommendations, but no section discussing implications for other IDSs.

Impetus for project came from Carolyn Clancy. She had a long-standing interest in new technology and the idea that differential diffusion is a major contributor to health disparities.

11 Incidence of reduced use of prescribed medications in response to out-of-pocket costs among Medicare beneficiaries

10/1/2001 -8/31/2003

$190K This Minneapolis-based division of UnitedHealth Group examined data for a sample of Medicare+Choice enrollees to examine how different out-of-pocket requirements affected their prescription medicine-taking behavior.

Surveying a random sample of elderly persons with chronic conditions in managed care plans, the authors found that lower drug benefits, higher OOP costs, lower income and poorer health were associated with underuse of medications. One-third of respondents did not fill a prescription or reduced dosage because of OOP costs.

Health Affairs article in July/August 2004 (Rector and Venus)

Not clear that any changes at United have occurred as a result of this study.

Page 3



12 Care for suspected pneumonia: the Evercare model

11/1/2002-4/30/2004

$210K Investigators are identifying factors that affect the quality of care received by nursing home residents suspected of having pneumonia. Analyses will determine the facility, provider, and resident characteristics that influence the adherence of nurse practitioners to treatment guidelines, patient mortality and hospitalization rates, and time to recovery.

There was large variation in hospitalizations for pneumonia across sites; also varied depending on weekday/weekend, RN staffing, advance directives. (Rector said that Mach already knew that there was substantial variation; may not have know the extent of various factors.)

Presentation at the third annual IDSRN meeting (Rector and Finch); in April, Rector indicated that they had submitted a manuscript to Medical Care .

Have increased leadership in Florida as a result of the study. Harder to make changes based on the weekend findings (more hospitalizations during weekend days) because the necessary changes in staffing would be quite expensive.

SOLE SOURCE

13 Refining a tool to assess racial/ethic and SES disparities in health plan performance on HEDIS measures

9/29/2003 -3/31/2004

$63K Builds on TO#1. A tool developed by RAND and CHCPE as part of previous IDSRN work will be refined and adapted to facilitate analyses of the nature and extent of racial/ethnic and SES disparities among different subgroups and across a variety of measures using health plan data. Algorithms and macros import raw data into a simple but flexible Excel model that allows users to instantly view or print key information in tabular and graphical form for any measures and subgroups. This tool will be useful to health plans interested in examining racial/ethnic disparities in performance on HEDIS measures.

Refining the race/ethnic tool so plans can use; also gathering plan feedback on the tools. Plans liked the tool but found it clunky. Currently in the process of trying to make tool more user friendly.

Was supposed to be disseminated through the Learning Collaborative. Not sure about the extent to which this has been done. TOO said AHRQ will provide technical assistance through the Learning Collaborative for other plans to learn to use the tool. (They were just beginning this process when we interviewed them in April.)

United has started to look at race/ethnicity data more; large employer like that they are looking at it. (Two different plans within United and Molina are using the tool to assess disparities across a set of measures.)

14 Developing and testing health plan tools to identify at-risk enrollees

10/6/2003 - 4/5/2005

$161K CMS Health of Seniors survey data are being linked to UnitedHealth Group Medicare+Choice plans’ administrative (enrollment and claims) data to compare members whose functional status decreased, remained stable or increased on a variety of key health care variables. The task is to try to develop a predictive model that plans can apply to their administrative data to identify members at risk of declining functional status for care management programs. Analyses of the survey data will be used to guide development of claims-based algorithms.

TO is still in progress. TO is still in progress. TO is still in progress. SOLE SOURCE

DENVER HEALTH15 Capacity to conduct

studies on the impact of race/ethnicity on access, use and outcomes of care

01/29/01 -10/28/2001

$163K Denver Health collaborated with the Research Triangle Institute and Weill Medical College of Cornell University to assess the extent to which data from integrated delivery systems could be used to determine the impact of race/ethnicity on access to, and use of, health care and associated clinical outcomes. The study also yielded information on strategies that might be adopted to improve capacity to examine the health consequences of minority disparities.

There is no uniform policy across IDSs for obtaining information on race/ethnicity, and validation of and quality control standards for race/ethnicity measures have not been established. Found that information on provider's race/ethnicity and language spoken, as well as availability of translation services, would be useful.

Presentations at 2002 and 2003 AcademyHealth research conferences and at University of Colorado Health Science Diversity Week Research Exchange. Also presented at Weill Medical College seminar.

Denver Health has implemented a permanent process into its system to collect race/ethnicity data, with patients self-identifying their race/ethnic background at initial self-registration.

Page 4



16 Effect of using rules technology with provider order entry in medical error reduction

1/1/2002 - 7/31/2003

$245K Denver Health studied the effectiveness of using computerized order entry systems to reduce medication errors. The project focused on adverse events related to medication use that can lead to hyper- and hypokalemia, nephrotoxicity, thrombocytopenia, and hepatic inflammation. In addition, the investigators assessed provider satisfaction with the computerized decision support system within CPOE.

As a result of the alert system, there was an increase in the percent of time the provider did not complete the medication order; overall a modest effect on the medication ordering process. Providers felt the alerts were not helpful on a practical level, mainly because of other components of the computerized provider order entry. Wanted less alerts overall, and wanted them to be more tailored to patient characeristics.

Presentation at the 2003 annual meeting of the Society of General Internal Medicine presentation at a 2003 conference of the American Association of Health Plans; poster at the American Medical Informatics Association 2002 meeting. Presented at 2002 Academy for Health Services Research and Health Policy Annual Research Meeting and presented at 2002 IDSRN annual meeting.

Continuing this work with another demonstration.

17 Use of encoded guidelines in an electronic medical record system for targeted tuberculin testing and treatment of latent TB

1/1/2002 - 7/31/2003

$154K This Denver Health study analyzed the effects of using computerized decision support systems on appropriate screening for latent tuberculosis, by encoding the CDC’s guideline for this disease in the electronic medical record. The rules technology that was developed to encode the guideline is transferable to other information systems that have clinical decision support technology. This technology can also be used to encode the numerous other clinical guidelines that have been developed by various agencies and organizations.

The CDC guidline for TB risk factors was too broad for the Denver Health population; 70% of patients triggered the rule. Changed to a less stringent rule as a result, continued with alerts. Physicians compiled with guideline 17% of the time. Of the PPDs read, 45% were positive. Resulted in improved screening and identification of patients with TB.

Presentation at a 2003 conference of the American Association of Health Plans. Also presented at a variety of other summits and conferences.

Denver Health has continued the intervention in 2 clinics and plans to expand to other settings.

18 A study of the impact of race/ethnicity on access, use and outcomes of care

11/1/2001- 10/30/2002

$225K In this follow-up to their first task order, Denver Health collaborated with Weill Medical College of Cornell University to compare diabetic outcomes for patients from two major healthcare systems: the Denver Health System and the New York Presbyterian Healthcare System. The major objectives of this study were: 1) to assess whether racial or ethnic disparities in access and use of care related to diabetes are present within and between the two hospital systems; 2) to describe efforts in both health systems to address disparities; 3) to propose approaches to mitigating disparities in these health systems; and 4) where one or more of these approaches is being implemented, to evaluate the impact of each approach in reducing disparities.

TOO said Denver Health confirmed their findings from Task Order #1 to themselves. There may not be significant within system differences between racial/ethnic disparities. Note that there were limitations to data, however.

Presentation at the 2003 AcademyHealth research conference; poster at 2002 AcademyHealth research conference. Also presented at first annual IDSRN meeting and a seminar at Weill in December 2002.

Not clear what changes, if any, this TO resulted in.

Page 5



19 Rocky mountain regional care model for bioterrorism preparedness

10/15/2002-10/24/2003

$452K Denver Health assessed surge capacity and short- and long-term health system needs for regional bioterrorism preparedness. The study examined effects of regional care models and their impact on resource allocation and capacity in the event of a potential bioterrorist event; the effect of a potential bioterrorist event on hospital and health system’s costs, outcomes, and staffing; facility characteristics needed to establish isolation or quarantine units for a region; and characteristics of the model that are exportable to regional, state and local policymakers.

Project provides updates twice a month to the Rocky Mountain Center for Medical Response to Terrorism, Mass Casualties, and Epidemics. In January 2003, over 50 people attended the first regional working group meeting, which included bioterrorism preparedness representatives from 6 states (Colorado, North Dakota, South Dakota, Wyoming and Utah), U.S. Public health Service Region VIII, U.S. Air Force (Office of the Surgeon General, Homeland Security Office, and US Northern Command, Army National Guard, county and local government representatives and regional hospital representatives.

A U.S. Northern Command participant in Denver Health’s Rocky Mountain Regional Care Model was particularly interested in how to identify "best" alternative care sites in the event of a bioterrorist threat (e.g., advantages vs. disadvantages of churches, schools, hotels, auditoriums, etc.) depending on the type of attack. In response to this need, Denver Health developed an “Alternative Care Site Selection Matrix Planning Tool” consisting of a slide show, a matrix for users in an Excel spreadsheet format and a document describing the tool. U.S. Northern Command shared it with the 2004 Olympics Planning Committee in Athens, Greece. Also, the Denver Center for Public Preparedness (funded by the CDC) will utilize the Rocky Mountain Regional Care Model tools during training for about 30 public health workers from around the country. Finally, the hospital medical cache concept was adopted for use by Denver Health's Denver Center for Public Health Preparedness; it can be used regionally to supplement hospital supplies during a surge event. Also publishing in Biosecurity and Bioterrorism

Many organizations in region appear to be using tools. Benchmarks have helped people think about how to prioritize resources. Increased communication between organizations in Region 8 (spanning several states). TOO said when Denver Health presented the model at a regional meeting, some present immediately saw its applicability.

As an outgrowth of this TO, team received a HRSA Bioterrorism, Nuclear, Incendiary, and Chemical Explosives (BNICE) $2.3 million grant.

20 Rocky mountain regional health emergency assistance line and triage (HEALTH) hub model

10/15/2002-10/14/2003

$267K Denver Health assessed requirements, specifications and resources to develop a public health emergency contact center in Colorado and Hawaii, with exportability to seven other states. The model addresses integration with public health agencies and efficient management of patient demand surges during and after a public health emergency such as a bioterrorist event.

Created a HEALTH model toolset for use by national, state and local policy makers. Identified best practices, challenges and shortfalls of public health agency responses to information, risk communication, risk interpretation and triage in public health emergencies. "Created a call center that was well-integrated with public health, and that could adapt and expand to provide the surge capacity and alleviate impact."

See outcomes. Developed tools for AHRQ regional workshops on bioterrorism, held regional working group meeting, etc. Several presentations at the AHRQ-sponsored bioterrorism conference in May 2004.

TOs #5 and 6 are closely related.

Page 6



21 Improving efficiency through hospital redesign

9/22/2003-10/22/2004

$350K The overarching focus of this project is to remove the silos of care between and across hospital disciplines. Denver Health will serve as a pilot site to plan redesign of one or more of its hospital units or departments based on input from operational, organizational and regulatory experts, providers and administrators, and patients and families. Three primary areas of hospital redesign are being addressed: internal and external process redesign and infrastructure redesign. Conducting a lit review, talking with non-health and health organizations that have done re-design, and will also do a flow study. We have an external committee as well as in internal steering committee.

The project is expected to provide an environmental scan on good examples of design, etc. The end result will be a how-to manual.

Gabow has made numerous presentations already on the project to groups like the Denver Health Leadership Forum, the Denver Metro Chamber of Commerce, etc. She also presented on the project at a national meeting of public hospitals and has presented at Denver Health’s "Day of Celebration" to which legislators, city councilors, and the press are invited.

TO only recently completed. SOLE SOURCE. Steering committee meets twice a month about redesign.

EMORY22 Capacity to conduct

studies on the impact of race/ethnicity on access, use and outcomes of care

09/25/00-6/24/2001

$171K Emory examined Aetna’s capacity to study racial/ethnic disparities using Aetna databases. Efforts were made to identify potential strategies for improving such capacity, such as seeking information on race/ethnicity from large employers or directly from patients by survey.

Barriers to collecting race/ethnicity on enrollment forms include diverse state regulations, privacy issues and changing internal systems. Found that a mail survey of current enrollees was not a good way to collect race/ethnicity data (extremely low response rate). A telephone survey of employers revealed that the largest barriers to collecting these data related to legal and confidentiality issues. Emory concluded that collecting these data on a majority of MCO members may take years.

Presented results to management at Aetna. Also presented externally at the Academic Medicine and Managed Care Forum meeting in May 2001. Staff apparently worked on a paper for a peer-reviewed journal, but we could not find in the literature.

Aetna started collecting data on race/ethnicity from its new enrollees beginning in 2003.

23 Predicting health care use resulting from terrorism: tools to aid state planning

10/1/2002-9/30/2003

$270K Terrorism-induced psychological and clinical sequelae have a great potential to overburden health care systems. Emory examined the large-scale impact on the health care system of terrorism and anthrax events in the fall of 2001 for 5 million Aetna members. Stress-related medical resource use patterns in inpatient, outpatient and emergency room settings before, during and after these events were analyzed for persons insured by Aetna across the US. The study accounted for seasonality, geographic distance from the events, short- and long-term impact of the events, and impact on special populations. The goal of the study was to permit improvement in future bioterrorism resource allocation planning activities.

Provided information on cost trends after an event like 9/11. TOO said they tapped into a "worried well" around the country, looking at whether demand for care increase after 9/11. Found that little had changed. One of few projects that has looked at cost and utilization, but was limited by Aetna data.

Currently working on a final manuscript (presumably a peer-reviewed paper, as indicated in the proposal); note that TO has been over for some time.

Aetna seemed to participate in this TO as a "good corporate citizen" but did not seem to glean much from it or change practice as a result. "Aetna is not as interested in things like bioterrorism because it doesn’t affect their bottom line." But Aetna’s Director of Quality Improvement indicated that this TO provided important information to their underwriters about cost trends that occur after an event like 9/11.

Page 7



24 Improving the management of falls in long-term care facilities

1/5/2004-1/4/2006

$376K This quality improvement project will modify key components of an existing Falls Management Program for adaptation in community settings. The feasibility and impact of this modified program will be evaluated in 26 Georgia nursing homes. For nursing homes interested in improving falls management, this project will produce educational materials, training manuals, a computer web-based user-friendly interface for entering fall reporting form data, a web-based, centralized quality improvement data base and related quality improvement reports, and qualitative data on adherence.

TO is still in progress. (Primary deliverable is expected to be a falls monitoring tool for use in nursing homes; includes an incident reporting form.)

TO is still in progress. TOO thinks the work will be translatable to nursing homes outside the Ethica system.

TO is still in progress. SOLE SOURCE. Partnered with Ethica (a company that owns nursing homes) for this project.

HMO RESEARCH NETWORK/KAISER25 Researching the

implementation of otitis media practice guidelines developed on the basis of AHRQ-supported evidence reports in a pediatrics population

09/18/00-10/31/2002

$196K Lovelace Clinic Foundation in Albuquerque, NM, evaluated the implementation of a clinical practice guideline for otitis media with childrentreated within an integrated delivery system. The study compared health care provided to Hispanic and non-Hispanic enrollees and Medicaid and non-Medicaid enrollees.

Intervention clinics showed significant reduction in acute otitis media cases pre-post compared to control clinics; intervention clinic physicians exposed to full intervention showed significant decrease in % of acute otitis media cases relative to those intervention physicians who did not attend education sessions

Presentation at the American Association of Health Plans’ Building Bridges IX conference.

Intervention was effective in changing diagnosis and treatment patterns in compliance with guidelines.

TO #11 is a follow-on to this TO. TOO said this was one of the first IDSRN projects to have implementation and evaluation.

26 Model culturally and linguistically appropriate service (CLAS) quality assessment and performance improvement projects (QAPI) for Medicare+Choice organizations

10/10/2001- 9/30/2002

$250K Two guides were developed at Lovelace Clinic Foundation to help managed care organizations plan for and provide culturally and linguistically appropriate services to Medicare+Choice patients. The study was funded by the Centers for Medicare and Medicaid Services (CMS) through an interagency agreement with AHRQ.

Conducted literature review to identify organizations involved with CLAS, conducted interviews with M+C enrollees with limited English proficiency. Produced two guides related to CLAS.

Presentation at the HMO Research Network 2002 conference (Paez), 2003 AcademyHealth conference (Gunter and Paez). UNC-Chapel Hill does a workshop series on CLAS, in which it uses two guides developed under this task order (this is funded jointly by CMS, HHS Office of Minority Health and RWJF). Workshop has been held in Philadelphia, Chicago, San Francisco, Orlando, and Albuquerque. Guides are available for download from CMS and AHRQ web sites and were mailed to all M+C plans.

Need more assessment of this, but presumably the guides have made many plans more sensitive to CLAS issues among M+C enrollees and perhaps more broadly.

27 Use of linked data bases to assess selected quality indicators for maternal health

10/10/2001 -12/23/2002

$193K Kaiser Permanente Northern California assessed the usefulness of selected quality indicators for predicting health outcomes for pregnant women and children. The study also examined whether variations exist in patterns of care among subpopulations of pregnant women and children.

Found significant variations across maternal demographic groups in rates of 4 quality indicators. Some limitations to study, such as lack of data to perform adequate risk adjustment.

Unknown. Not clear that any occurred.

Page 8



28 Relationship of provider group characteristics to quality of care and medication errors in ambulatory care settings

10/1/2001 - 6/20/2003

$300K HealthPartners Health Plan and Albuquerque-based Lovelace Health Systems studied the effect of selected medical group practice characteristics on quality of care and patient safety. Two medication safety issues were targeted: drug-drug interactions and recommended laboratory monitoring of drug therapy. Rates of screening for breast cancer, cervical cancer, and cholesterol were calculated as measures of quality of care.

Using claims data from Lovelace and HealthPartners, found that 6-7% of adults taking selected chronic medications were at risk for a drug-drug interaction each year and among all HMO members, the risk was 2%. In the range of 45-50% of users of chronic medications receive inadequate lab monitoring. No medical group characteristics or quality of care indicators were associated with increased risk. Patient characteristics were the best predictors of drug-drug interactions.

Presented at the HMO Research Network 2002 conference. Forthcoming in the American Journal of Managed Care in November 2004 (Solberg et al.).

This TO generated knowledge and awareness of failures and led to improvements in lab monitoring.

Note that PI changed during course of project.

29 Association of working conditions with prescribing errors in primary care settings

10/10/2001 -4/5/2003

$198K Kaiser Permanente Georgia and HealthPartners Health Plan examined whether primary care practice workload and organizational culture were related to rates of risky prescribing events, such as potentially dangerous drug combinations and the failure to monitor with laboratory tests, when warranted.

Final report not available. Unknown.

30 Developing data to monitor and reduce emergency department overcrowding

4/15/2002 - 4/4/2003

$249K HealthPartners and Kaiser Permenante Colorado used data from emergency department (ED), ambulatory care, and inpatient settings to develop measures of ED overcrowding, its causes, and its consequences. The study was intended to provide health care systems and their policymakers with the means to accurately measure the risk and extent of ED overcrowding.

Developed 113 potential measures of ER overcrowding and reduced the number to 38. Used Delphi method to prioritize the 38 measures.

Two papers from this TO were published in Annals of Emergency Medicine (Solberg et al. 2003; Asplin et al. 2003). Presented at the HMO Research Network conferences in 2003 and the 2nd IDSRN annual meeting.

The long-term monitoring system has begun to be implemented within the HealthPartners IDS and Overlook Hospital in JN monitors most of the measures in both systems. There is potential to implement the measures in different types of hospitals.

Similar work by this TO’s PI was funded by KFF.

31 Improving processes of care and coordination: identifying and sharing what works

5/1/2002 -9/30/2003

$250K The MacColl Institute for Healthcare Innovation (IHI) provided a conceptual framework, research context, and menu of potential best practices in one or more areas of care process redesign and/or coordination of care. These data, along with existing evaluation data, were used to assess "best practices" in care and coordination.

Developed a "Care Model" framework for coming up with best practices workshops and then interviewed leaders and experts from 31 organizations to develop a menu of best practices. Also developed a Guide to Practice Redesign.

Project team made recommendations to AHRQ about dissemination, but it is unclear whether they themselves disseminated anything. TOO said they haven’t published anything.

32 Use of electronic communication among patients and providers to improve quality of care (#2)

8/1/2002 -12/31/2003

$199K Investigators at Kaiser San Francisco are assessing how electronic communication, which includes e-mail and other Internet-based methods of communication, can improve communication, knowledge, access, effectiveness, timeliness, patient safety and other aspects of health care.

Qualitatively described e-communications across 12 IDSs based on interviews, and quantitatively described e-communications in Kaiser Northern CA at the patient level. Most IDSs have only nascent functionalities/capabilities for e-communication. Found there has been a substantial increase in e-communications, but there is a digital divide because it growing more rapidly among white, educated groups.

According to interview, this TO is putting out two manuscripts for publication. (have not yet been published yet, in Annals of Internal Medicine ). Also, 2 presentations at 2003 AcademyHealth research meeting. Also presented internally.

Not clear that any substantive changes occurred as a result of this study. However, according to interview, this TO started the process of getting information on this and started them working with vendor on these issues, which is a first step.

PI noted: "We are one of the few places that actually have quantitative data on this [electronic communication]."

Page 9



33 Patient characteristics, adverse drug reactions, best practices and patient safety: a study of HIV care in San Francisco

12/1/2002- 5/31/2004

$544K This project is evaluating practices undertaken to reduce adverse drug reactions and other untoward consequences of treatment for persons who are HIV positive.

Helped Kaiser Northern CA define categories of medication errors, which hadn’t really been done for HIV care. Developed a different chart abstraction form for each error category. Found that there was a lot of incorrect dosing. Also, medical record review revealed discrepancies between online system and medical record.

Interview indicated that they are preparing a manuscript for Medical Care .

Interviews seem to indicate that they are more aware of medication errors as a result of study and have incorporated in pharmacy and provider feedback, but the extent to which this has been done is not clear.

SOLE SOURCE.

34 Prevalence and strategies for appropriate prescription medication dosing in children

11/1/2002 - 11/30/2003

$195K Kaiser Permanente Northwest is assessing ambulatory prescription medication errors for children to broaden knowledge on the types of errors that occur and how to most effectively reduce errors. The study examines the impact of a computerized physician order entry system and other interventions that address medication errors.

Interview with TOO indicated the project was moving into the final stage. [Final report not available.]

Manuscript submitted to a peer-reviewed journal was supposed to substitute for the final report; we could not locate article in existing literature. Presentation at the Pediatric Academic Societies meeting in 2004.

TOO thought Group Health Cooperative and Kaiser Northwest would discover a list of drugs critical to examine and would begin to solve the problem within their own networks.

Kaiser Northwest, Group Health Cooperative and Fallon Community Health Plan are involved with this TO.

35 Knowledge transfer and utilization of CLAS guides and otitis media treatment guidelines

9/15/2003 -6/14/2004

$49K Follow-on analyses are being conducted for data gathered for a previous Otitis Media Guideline Implementation task order through the IDSRN. A plan will be proposed for dissemination of these new results. In addition, the project is tracking the use of CLAS guidelines, produced for CMS in another task order, for use in Medicare+Choice plans.

Unknown. Presentation at a 2003 conference of the American Association of Health Plans (Gunter). Planned to submit manuscript to peer-reviewed journal.

This TO actually tries to measure how and to what extent CLAS information from TO #2 has been used, and what changes it has resulted in.

Follow-on to TO #2.

36 Mesuring the clincal benefit and value of diabetes care

1/4/2004 -4/1/2005

$250K A natural history model of diabetes mellitus is being refined and adapted by Kaiser to estimate the net clinical benefit provided by participating plans or systems to their diabetic patients. Two years of data from various participating health plans are being collected and entered into the model for a specified set of performance and cost measures. The study will examine variation among plans or systems in apparent clinical benefit and costs of care. The investigators will recommend a natural history model, data collection methods, and a reporting format to best express the relative value of programs in the eyes of various stakeholders.

TO still in progress. TO still in progress. TO still in progress. SOLE SOURCE. Kaiser Northern CA and Henry Ford Health System are involved with this TO. TOO said this project fits with the business case priority with which health plans are concerned.

MARSHFIELD37 Quality-related

provisions in health plan-hospital contracts

11/9/2000 - 6/10/2001

$99K Marshfield and Project HOPE conducted a systematic review of 100 health plan contracts with providers to identify how different kinds of health plans (e.g., HMOs, PPOs) were operationalizing quality provisions in their provider contracts.

Based on a review of 116 provider agreements in 58 MCOs spanning 9 states, quality of care plays a minor role in most provider contracts; most only involved compliance with loosely defined quality management programs.

Sutton submitted paper for publication but it was not accepted; currently reorganizing and redrafting.

N/a (TO did not involve Marshfield).

Note that Marshfield staff or data were not involved in this TO.

Page 10



38 Assessing the impact of organizational interventions

09/25/00 -2/28/2002

$200K Marshfield evaluated the benefits of several organizational interventions, including the redesign of a charity care program and centralization of a clinic’s medical managementof anticoagulant therapy (coumadin clinic).

A variety of results, including the fact that the control group in the coumadin clinic intervention faced higher adverse event relative risks than the treatment group; enrollees in community care program had similar ED use to the intervention group and both these groups had more ED use than HMO population.

Presented at the AcademyHealth research meeting in 2003. Results from the coumadin clinic study were incorporated into Congressional testimony in April 2001. Also, presentation to Marshfield management. Revised paper on coumadin clinic for JAMA currently.

Marshfield is expanding their charity care intervention, but not really as a result of the TO. Still doing the coumadin clinics.

39 Increasing patient safety by improving compliance to clinical practice guidelines for diabetes management through electronically-generated reminders on patient interval reports and day sheets in a multispeciality group practices setting

10/1/2001 -4/30/2003

$138K This study was conducted to determine if low-cost electronic alerts or prompts could increase compliance among physicians to conform with evidence-based practice guidelines for physicians who treat patients with diabetes.

Improvements in compliance observed for both treatment and control groups over the study period.

Presentations at the National Rural Health Association meeting, the International Conference of the International Society for Quality in Health Care meeting, and the Wisconsin Public Health and Health Policy Institute, 2003.

Not clear what, if any, changes were made.

RTI40 Validating the

HCUP quality indicators

10/1/2000 - 5/30/2003

$449K RTI examined data and measurement issues and applications of how the prevention and inpatient quality indicators can be used by hospitals and community health care systems in quality measurement and self-assessment efforts.

By examining inpatient data from Utah, Oregon and Pennsylvania, RTI identified measurement issues that influence the use of the quality indicators by a variety of stakeholders. RTI concluded the indicators are useful tools overall for monitoring and tracking health care quality.

Task Order Officer said RTI did produce a publication and targeted providers, economists and public health professionals. They presented at AcademyHealth in 2003, the National Association of Health Data Organizations Annual Meeting in 2002, the International Health Economics Association 4th World Congress, and APHA.

No internal changes.

41 Assessing the information technology infrastructure in integrated delivery systems

09/25/00 - 3/15/2002

$200K RTI designed this project to increase understanding of factors that facilitate or undermine the diffusion of information technology within integrated delivery systems. The study identified strategies that can be tailored to overcome barriers to diffusion.

RTI used collaborator data, surveys and interviews to make cross-system comparisons about how different systems acquire, use and evaluate information technology.

Presentation at the National Meeting of the Academy of Management in August 2002 (Weiner and Savitz); 3 posters at the 2002 AcademyHealth meeting; poster at 2003 AcademyHealth meeting; presentation at the International Health Economics Association meeting in 2003. (Savitz et al.)


42 Validating the HCUP patient safety indicators

10/1/2001 - 6/30/2004

$300K RTI is assessing the latest generation of AHRQ Quality Indicators, the Patient Safety Indicators, using clinical data from its partners, Intermountain Health Care, UPMC Health Systems and Providence Health System. These indicators are screening tools for flagging potential quality-of-care problems.

Completed a comparative assessment of quality indicators associated with nursing care in hospital settings. Conducted modified Delphi process to match IDS interests and initiatives to available indicators (project still underway at time of interviews)

Participated in 2 AHRQ-sponsored meetings: panel discussion sponsored by AHRQ User Liaison Program and panel discussion of AHRQ quality indicators at AcademyHealth meeting. Submitted a proposal under the Knowledge Transfer & Utilization of IDSRN Products and was awarded TO #7 for dissemination.


Page 11



43 Integrated delivery system solutions for transferring medication data across patient care settings

10/1/2001 - 10/30/2002

$200K RTI examined how integrated delivery systems manage information transfer, a potential cause of medical errors. One study goal was to develop tools to evaluate the usefulness of technological solutions for this problem.

Produced a model that demonstrated if you developed a tool such as an e-medication list to be used at admission and discharge, up to 50% of medication errors could be prevented. Providence decided to hire a transitional pharmacist to use the tool and AHRQ provided them with an add-on to this task order to evaluate the use of the pharmacist (TO #6)

RTI convened a meeting with members of Providence and Kaiser Northwest to discuss their findings; presented at AcademyHealth Meeting in 2003 and submitted a manuscript to Healthcare Management Review .

Providence hired a transitional pharmacist as a result of this task order. Part of the pharmacists’s salary was paid from Task Order #6.

A Seattle hospital is using these findings.

44 AHRQ-sponsored workbook for sharing regional bioterrorist preparedness tools

10/1/2002 -1/30/2004

$486K Using Intermountain Health Care and the University of Pittsburgh Medical Center as examples of integrated delivery systems, Research Triangle Institute (RTI) is assessing the health system requirement and interorganizational relationships necessary for regional bioterrorism preparedness. The assessment examines the effects of regional care models and their impact on resource allocation and capacity in the event of a potential bioterrorist event, and the full spectrum of expected health care needs following a potential bioterrorist event, including hospital, home care, and long term care needs.

After developing an inventory of resources from the bioterrorism task orders and looking at the costs to the systems of planning, AHRQ wanted the results in a usable workbook and funded this task order to do so. RTI also received an add-on to work with "super-users" to integrate the workbook for sharing regional bioterorism tools.

Presentations to the IHC expert panel in June 2003; audio web cast on bioterrorism sponsored by AHRQ in June 2003. (Allswede) Savitz worked with AcademyHealth to plan regional bioterrorism preparedness seminars and showcase products of this research. Also participated in ULP conferences in Salt Lake City and Pittsburgh. Presentations at an AHRQ-sponsored bioterrorism conference in May 2004.

Task order just finished 7/04.

45 Estimating risk reduction and cost of enhancing medication information transfer across patient care settings

5/1/2003 - 10/31/2004

$300K RTI and Providence Health System are using advanced methodologies to assess risk associated with patient hand-offs and transitions at admission to and discharge from hospital. The study will examine the cost of such care process enhancements, report the business case for this intervention in an integrated delivery system (IDS) setting and assess the impact of enhanced risk and cost information to IDS decision makers.

This project built off an earlier task order which developed a model for improving medication information transfer at admission and discharge from hospitals. This task order allows RTI to evaluate the effectiveness of the transition pharmacist Providence hired after the ealier task order.

Task order recently completed. Poster at National Patient Safety Foundation Conference in March 2003; poster at AHRQ patient safety meeting in March 2003.

Task order recently completed. SOLE SOURCE.

46 Knowledge transfer and utilization of PS on-line query tool

9/15/2003 - 4/14/2004

$61K RTI is developing a secured, password-protected, Web query application for hospitals to directly generate their own Patient Safety Indicators (PSIs) from a statewide hospital discharge database. Utah will serve as the example in extending the work of an ongoing PSI project initiative led by RTI in conjunction with its IDSRN partner, Intermountain Health Care in Salt Lake City, Utah. The intent of the web application is to transform the current paper-and-postage practice into a paperless process that is much more cost efficient and timely.

RTI had intended to disseminate findings from earlier PSI task order to patient safety leaders at Intermountain when opportunity arose to work with Utah DOH to develop an interactive, web-based query tool. The enhanced dissemination strategy will extend the impact of the project to asll hospitals in Utah. They will also work to insure the tool is generalizable enough to be modeled by other states.

None noted as task order finished after interviews.

None noted as task order finished after interviews.

This task order was funded after 2nd annual IDSRN meeting when AHRQ decided to fund projects that looked promising to do knowledge transfer and utilization.

Page 12



47 Development of curricula to train users in the application of quality indicators (QIs) and the interpretation of QI output

10/6/2003 -4/5/2005

$124K A curriculum is being developed to train users in the application and interpretation of the AHRQ Quality Indicators (QIs). An initial core curriculum is being developed for a single stakeholder group (e.g., purchasers), and will be pilot-tested in a live training session held in conjunction with an agreed upon meeting. After initial testing and any necessary refinements to the core curriculum, it will be modified to respond to some of the unique needs of users representing other major stakeholder groups. These modified curricula will then be pilot-tested in additional live training sessions held in conjunction with other meetings.

TO still in progress. TO still in progress. TOO said the audience is anyone interested in quality measurement and the goal is to have the curricula available on the Web. RTI wants to develop a self-training mechanis, on the tool so anyone can walk through the training themselves.

SOLE SOURCE.

UNIVERSITY OF MINNESOTA48 Relationship of

nursing home policies and quality

10/1/2001 -9/30/2003

$295K The goal of this task order was to assess the frequency of negative events such as patient falls, hip fractures and pressure ulcers, and to determine the extent to which the variation in occurrence of these could be explained by the characteristics of the nursing home and its clinical and administrative practices.

Resulted in 1-2 papers (see next column); but "nothing disseminatable" because they didn’t find any relationship between nursing policies and care provided.

Paper forthcoming in Nursing and Health Policy Review (Mueller et al. 2004). Note that during our April 2004 visit, Kane indicated that two "weak" papers had been sent out to journals.

None known.

49 Effects of financial incentives in medical group practices and the work environment on quality of care

10/1/2001 -10/31/2003

$242K This study included three separate analysis to assess: 1) the influence of the structure and culture of the group practices and practice and physician level financial incentives on per member per year (PMPY) costs, 2) the effects of these same variables (but in a restricted model) on prescription drug errors, and 3) the influence of the variables in the restricted model on mammography and Pap smear rates in the practices. The study included data on over 70,000 enrollees who received care in 136 practices during the study year (2001) and 250,490 prescriptions that were filled for those enrollees.

Helped MGMA understand how to better disseminate information to its members; now they are setting up a research network of medical group practices under TO #4 (which builds directly on this TO).

Papers in the Journal of the American Board of Family Practice (Curoe, Kralewski, and Kaissi, 2003); Minnesota Physician (Kaissi, Kralewski, and Silversmith, 2002); Journal of Ambulatory Care Management (Kaissi, Kralewski, and Dowd 2003); Minnesota Medicine (Kralewski et al. 2004); Health Care Management Review (Kaissi et al. 2004); posters presented at annual MGMA meetings; poster at 2003 AcademyHealth research conference (Kaissi). In his interview, Kralewski noted that he presented at the MGMA conference to a group with standing room only.

University of Minnesota and MGMA have applied for an RO1 grant from AHRQ--an outgrowth of this TO.

50 Impact of payment policies on the cost, content and quality of care

5/1/2002 - 10/31/2003

$250K Economic incentives inherent in the relationship between health plans and health care providers (physicians and hospitals) were examined to identify the influence of such incentives on the cost, quality, and type of services received by patients.

Unknown. [This TO was not a focus in interviews.]

Unknown. [This TO was not a focus in interviews.]

University of Minnesota and MGMA have applied for an RO1 grant from AHRQ--an outgrowth of this TO.

Page 13



51 Establishing a National Network of Medical Group Practices

6/1/2003 -1/31/2005

$300K MGMA and the University of Minnesota are developing the Group Practice Research Network (GPRN), a collaboration of leading medical group practices across the nation interested in research initiatives to improve medical group practices. GPRN’s goal is to facilitate an aggressive, flexible and statistically valid program of research that can respond quickly to emerging issues in health care delivery in medical practices. Network members will generate research ideas, participate in research projects and test the feasibility of practice improvement initiatives.

TO still in progress, but they plan to use network to communicate research findings in order to change practice.

TO is still in progress. Not yet available. SOLE SOURCE.

WEILL52 Integrated delivery

systems and clinical preparedness for bioterrorist events

09/25/00 - 9/24/2001

$250K This project assessed a health system's existing capability of responding to and managing a public health threat. The investigators modeled a hypothetical release of anthrax in New York City, producing the first scalable computer models of civilian medical response to bioterrorism. The model includes an outpatient prophylaxis component and a hospital surge capacity component. An important modeling goal was to pinpoint the limits of system capacity in response to a hypothetical bioterrorist mass casualty event.

Developed simulation model to evaluate the relationship between mass prophylaxis and health system capacity to treat victims after an attack.

Information was used as a template by the NYC Office of Emergency Management and the Department of Health. See TO #2 below.

[Note that impact of bioterrorism TOs was felt externally--stage 3 impact.]

Note that Weill was the only team that applied for this TO.

53 Optimizing clinical preparedness for bioterrorist attacks using simulation modeling and data from recent anthrax attacks

4/1/2002 - 3/31/2003

$165K A simulation model previously developed by Weill for the IDSRN (TO#1) was refined for quick dissemination to the New York health system and others seeking help in addressing community concerns with terrorist attacks.

Developed computerized staffing software to help hospitals and health systems plan antibiotic dispensing and vaccination campaigns to respond to bioterrorism or large-scale natural disease outbreaks; downloadable from web.

Presentation at the 25th Annual Meeting of the Society of General Internal Medicine in May 2002 (Hupert); paper in the Annals of Internal Medicine (Hupert et al. 2003); paper in Medical Decision Making (Hupert et al. 2002); presentation at AcademyHealth annual research meeting in June 2003; presentation at IOM conference on SARS (Hupert); presentation at the 2002 annual meeting of the Society for Medical Decision Making (Hupert). Smallpox version of model is available nationwide on AHA and AHRQ web sites. New York Department of Health distributed copies of the model to each NY county. Also used by the Minnesota Department of Health.

Increased preparedness; screening tool for physicians to differentiate anthrax infections from flu cases.

This project led to the PI (Nathaniel Hupert) receiving another AHRQ grant to use the anthrax modeling data to look at bed and surge capacity issues. TOO said this project looks most successful because Weill had the right time, the right opportunity, and the right people.

Page 14



54 National Guidelines for Mass Distribution Centers

9/1/2002 -4/30/2003

$240K Weill assembled resource materials and developed national guidelines for mass prophylaxis distribution centers to assist national public health authorities. The resource materials included generation of computer simulation models, establishment of a "best practices" databank, and development of a clearing/warehouse Web site.

Building on work from TO #1 and 2, developed spreadsheet-based planning tools to assist public health and emergency management personnel in resource and staffing planning.

Presented at the second annual IDSRN meeting. Also see TO #2 above. The Cuyahoga County Board of Health in Cleveland adapted Weill’s model to prepare a mass prophylaxis planning template. Also used by North Dakota; Florida; Monroe County NY (Rochester); Amarillo, TX. Presentation for a web-assisted audio conference on bioterrorism tools in April/May 2002 (Hupert).

Increased preparedness Information from this TO was used to train personnel in strategic national stockpile workgroups. (SEE PREVIOUS TASK ORDER AS WELL.) Unfortunately, Weill was not able to get access to NY Department of Health data for this TO, as they had hoped. As a result, switched to a model for hospital administrators.

55 Developing tools for statewide planning and response to bioterrorist attacks

10/1/2002 - 9/30/2003

$338K The goal of the project was to expand and implement New York Presbyterian Healthcare System’s existing surge capacity information system (developed under TO#1) as a regional model for New York State. The information system estimates staffing requirements, cost and outcomes of a bioterrorist event affecting NYPHS and establishes appropriate data systems and readiness measures. Weill developed risk communication messages and dissemination plans for NYPHS staff, New York metro area health care providers, and the public.

Bioterrorism Emergency Response Model (BERM)

A public health planner in the Southwestern U.S. used Weill’s Bioterrorism Emergency Response Model (BERM) to assist his team, whose members had been reluctant to accept the large staffing numbers that they had been forecasting in their own plan. The BERM provided the team with a means to look at specific data sets and conduct “what if” analyses in an efficient manner. The BERM gave the team adequate confidence in its numbers to move forward with recruitment and training. Presentation at the 2004 AcademyHealth reseearch conference. BERM is downloadable through the AHA web site; in an 8 month period, the model was downloaded 830 times. http://www.aha.org/aha/key_issues/disaster_readiness/resources/vaccination.html

Downloadable software. NY state department of health distributed the smallpox model to all 57 counties in the state.

56 An examination of the role of leadership in enhancing patient safety

12/16/2002 -12/15/2003

$294K TO involved a detailed literature review on workplace safety among non-healthcare organizations and methods to promote patient safety and sustainable changes within healthcare organizations, as well as a national conference on the role of organizational leadership in improving patient safety.

Literature review identified major challenges instilling a safety culture in health care; rest of manuscript uses a learning organization approach as a foundation for conceptualizing and catalyzing efforts around safety culture.

Weill then sponsored a conference on this topic (see 2003 annual meeting binder notes). One paper under review at the Journal of Health Care Management .

NY Presbyterian Healthcare System implemented GE's Six Sigma program to train a group of its physicians to improve patient safety in several different hospital units.

AHRQ rates this TO as having impact within its own IDS.

Page 15



57 Improving transitions and outcomes for heart failure patients through a hospital-home health care information exchange

9/1/2003 - 5/31/2004

$212K An information-based hospital-home care transition intervention is being designed and pilot-tested to improve care and outcomes for heart failure patients. The study goal is to improve communication among members of the transition team at discharge and at readmission to: increase “on-time,” safe, appropriate hospital discharge; reduce medical errors associated with hand-offs; and reduce unnecessary re-hospitalizations, and improve selected patient outcomes.

Development of an electronic discharge order set, which includes reminders about beta blockers and ace inhibitors. User friendly, improves process (better than handwritten orders) and quality of care.

Hope to meet with VNS soon to discuss dissemination plans.

Intervention has been adopted in both geriatrics and internal medicine at Cornell--moving from a paper system to the use of an electronic system.

SOLE SOURCE. Involves collaboration with the Visiting Nurse Service of NYC. AHRQ plans to provide a follow-on task order to this one; in this follow-on, Weill wants to develop a web-based interface with VNS and also to disseminate it. May eventually turn into an RO1 application.

58 eICUs: Impact on quality and cost performance

9/15/2003 - 12/31/2004

$361K The study goals are to evaluate the clinical outcomes and impact of an eICU system in a medical ICU at a large academic medical center, perform a budgetary perspective cost-impact economic analysis of the eICU system across a variety of ICU settings, and assess provider attitudes regarding the use of the eICU system.

TO still in progress. TO still in progress. TO still in progress. SOLE SOURCE. TOO said the idea for this generated from Leapfrog.

Page 16

Table B-2. Summary Statistics

TOPIC # of TOs

# of Partners with TO in This Area

Total ($) # of TOs Starting

in FY 2000

# of TOs Starting

in FY 2001

# of TOs Starting

in FY 2002

# of TOs Starting

in FY 2003

Total Funding FY 2000

Total Funding FY

2001

Total Funding FY

2002

Total Funding FY

2003

PRIORITY AREASInformation Technology in IDS 3 2 $612,857 0 3 0 0 0 $612,857 0 0Organization & Coordination of Care 6 4 $1,598,615 0 0 4 2 0 0 $948,709 $649,906Improving Patient Safety 11 6 $2,994,165 0 4 5 2 0 $933,000 $1,466,848 $594,317Bioterrorism 8 5 $2,701,338 0 0 8 0 0 0 $2,701,338 0Knowledge Transfer & Utilization 5 4 $345,948 0 0 0 5 0 0 0 $345,948Developing Data & Measurement Capacity 3 3 $1,020,131 3 0 0 0 $1,020,131 0 0 0Health Care for Minority Populations, Women & Children 7 3 $1,427,130 2 5 0 0 $395,703 $1,031,427 0 0Structuring Health Care Delivery 5 4 $998,778 5 0 0 0 $998,778 0 0 0Long-term Care 3 3 $798,209 0 0 1 2 0 0 $209,968 $588,241Quality Improvement 3 3 $770,710 0 0 0 3 0 0 0 $770,710Working Conditions & Quality of Care 2 1 $498,000 0 2 0 0 0 $498,000 0 0Financial Incentives & Cost of Care 2 2 $432,000 0 2 0 0 0 $432,000 0 0

Total 58 $14,197,881 10 16 18 14 $2,414,612 $3,507,284 $5,326,863 $2,949,122

Care Management 2 1 $499,833 1 1Cost, Organization and Socioeconomics 9 4 $2,090,638 3 2 2 2Data Development 7 4 $1,706,243 3 2 2Health Information Technology 8 6 $1,710,232 1 4 2 1Long-term Care 3 3 $798,209 1 2Pharmaceutical Outcomes 1 1 $190,000 1Prevention 0 0 $0

SUMMARY STATISTICS ACROSS ALL TASK ORDERS (TOs) AND PARTNERS

PORTFOLIO AREAS

Page 1


Quality Improvement and Patient Safety 17 8 $3,952,962 2 6 5 4System Capacity and Emergency Preparedness 11 6 $3,249,764 1 9 1Training 0 0

Total 58 $14,197,881 10 15 20 13

TOPIC # of TOs


Total ($) # of TOs Starting

in FY 2000

# of TOs Starting

in FY 2001

# of TOs Starting

in FY 2002

# of TOs Starting

in FY 2003

Total Funding FY 2000

Total Funding FY

2001

Total Funding FY

2002

Total Funding FY

2003

FUNDING SOURCECore Funds 30 9 $6,419,046 9 5 4 12 $2,139,639 $1,103,427 $948,709 $2,227,271Patient Safety 16 8 $3,990,022 0 9 5 2 0 $1,928,857 $1,466,848 $594,317Bioterrorism 9 5 $2,951,311 1 0 8 0 $249,973 0 $2,701,338 0Long-term Care 2 2 $287,502 0 0 1 1 0 0 $209,968 $77,534HSAG 1 1 $50,000 0 0 0 1 0 0 0 $50,000ASPE 1 1 $25,000 1 0 0 0 $25,000 0 0 0CMS 1 1 $250,000 0 1 0 0 0 $250,000 0 0OMH 1 1 $225,000 0 1 0 0 0 $225,000 0 0

Total* 61 $14,197,881 11 16 18 16 $2,414,612 $3,507,284 $5,326,863 $2,949,122

TASK ORDER LOCATION (AHRQ DIVISION)CDOM 32 9 $7,705,443 5 8 6 13CP3 18 7 $4,698,971 2 6 10 0CQuIPS 4 3 $1,044,626 1 1 2 0COE 3 2 $552,843 1 1 0 1OEREP 1 1 $195,998 1 0 0 0

Total 58 $14,197,881 10 16 18 14

*Total is more than 58 because some TOs had more than one funding source

Page 2


TOPIC # of TOs


# of TOs Starting

in FY 2000

# of TOs Starting

in FY 2001

# of TOs Starting

in FY 2002

# of TOs Starting

in FY 2003

TASK ORDER OFFICER (location)Jim Battles (CQuIPS) 1 1 0 0 1 0Arlene Bierman (COE) 1 1 1 0 0 0Cindy Brach (CDOM) 2 1 0 1 0 1Carolyn Clancy & Ernest Moy (AHRQ Director; COE (CQuIPS) 1 1 0 1 0 0Carole Dillard (CP3) 3 3 1 0 2 0Denise Dougherty (OEREP) 1 1 1 0 0 0Steve Fox (COE) 1 1 0 1 0 0Fred Hellinger (CDOM) 4 4 1 1 2 0Kerm Henriksen (CQuIPS) 1 1 0 0 1 0Rhonda Hughes (CP3) 1 1 0 1 0 0Joanna Jiang (CDOM) 1 1 1 0 0 0Ed Kelley (CQuIPS) 1 1 1 0 0 0Bill Lawrence (COE) 1 1 0 0 0 1Peggy McNamara (CDOM) 2 2 2 0 0 0Kelly Morgan & Helen Burstin (CP3) 2 1 1 1 0 0Eduardo Ortiz (CP3 ) 6 4 0 4 2 0Cynthia Palmer (2 w/Remus) (CDOM) 18 7 1 6 2 9Sally Phillips (CP3) 6 3 0 0 6 0Denise Remus (2 w/Palmer) (CDOM) 3 1 1 1 0 1Tom Shaffer (CDOM) 3 3 0 0 1 2Robin Weinick (CDOM) 1 1 0 0 1 0

Total: 21

Page 3


TOPIC # of TOs


# of TOs Starting

in FY 2000

# of TOs Starting

in FY 2001

# of TOs Starting

in FY 2002

# of TOs Starting

in FY 2003

TYPE of DATA USEDUsed partner/collaborator data 50 9 8 15 16 11Used other data sources 8 6 2 1 2 3

TOPIC # of TOs


# of TOs Starting

in FY 2000

# of TOs Starting

in FY 2001

# of TOs Starting

in FY 2002

# of TOs Starting

in FY 2003

TASK ORDER PRODUCTSFinal report only 10 7 3 3 4 0Final report plus presentation & manuscript 11 8 1 8 1 1Final report plus manuscript 10 6 1 2 4 3Final report plus tools 8 5 1 0 3 4Final report plus other presentation 7 7 3 2 1 1Final report plus AHRQ presentation 3 3 1 0 2 0No Final Report (other deliverables) 9 6 0 1 3 5

Total 58

Page 4

Analysis of Dissemination and Impact of the Integrated Delivery System Research Network. Washington,...

Documents

Transcript of Analysis of Dissemination and Impact of the Integrated Delivery System Research Network. Washington,...