Rachel Richesson, Duke UniversityAllen Flynn, University of MichiganChris Dymek, AHRQGerald Perry, University of Arizona

Data Analysis, New Knowledge, and then What? Perspectives on MobilizingComputable Biomedical Knowledge

#MobilizeCBK

Panelists

Rachel Richesson, PhD, MPH, Duke University @rrichesson MCBK Steering Committee

Allen Flynn, PharmD, PhD, University of Michigan MCBK Standards Workgroup

Chris Dymek, EdD, Agency for Healthcare Research and Quality MCBK Sustainability for Mobilization and Inclusion Workgroup

Gerald (Jerry) Perry, MLS, University of Arizona MCBK Sustainability for Mobilization and Inclusion Workgroup

#MobilizeCBK

Better Health

Better Care

…Lower Cost

The Triple Aim…

Knowledge

Information

Data

Better Health Requires This

Not Just This

Journals

Knowledge to Practice

Knowledge should be FAIR*

• Findable• Accessible• Interoperable• Reusable

*FAIR: https://www.force11.org/group/fairgroup/fairprinciples

MetadataLibraries

Data standards

•Descriptive attributes•Provenance•Licensing info.•Implementation guidance

•Performance Data•Monitoring

Vol. 366(6464):447-453October 25, 2019

Impact of Bias and Missing Data

• Incomplete data• Under-representation of

populations• Misclassification• Measurement errors• Missing data for certain

groups

Knowledge is inappropriate or harmful for some patients

Interventions propagate disparity…

Some groups denied access to procedures; or receive inappropriate care

These patients do worse

Data not captured on some groups

“FAIRness” Enables Innovation

Data

Research, discovery, generation of evidence

Applications/Action-- targeted, personalized, useful,

usable, …

Knowledge

Mobilizing Computable Biomedical Knowledge (MCBK): A multi-stakeholder movement

Mission and Vision: A Manifesto Interim Home: The University of MichiganGovernance: Steering Committee

Activities to Date: Public meetings (2018, 2019) Four workgroups Web resources Webinars National and global collaborations (connections to Learning

Health Systems initiatives and communities)

MCBK Manifesto

www.MobilizeCBK.org

It is no longer sufficient to represent knowledge only in words and picturesDecisions should be informed by the best available knowledge MCBK is committed to making use of knowledge to improve healthMCBK is committed to upholding the integrity, reliability, and validity

of computable knowledgeMCBK is committed to open, transparent, equitable, and inclusive

approaches to making computable knowledge FAIR

Highlights of MCBK Mission and Vision

MCBK Workgroups and Co-Chairs Standards for MCBK Robert Greenes and Bruce Bray

Technical Infrastructure for MCBK Leslie McIntosh and Chris Shaffer

Policy and Coordination to Ensure Quality and Trust Jodyn Platt and Blackford Middleton

Sustainability for Mobilization and Inclusion Chris Dymek and Gerald (Jerry) Perry

Steering Committee: Julia Adler-Milstein, Bruce Bray, Milton Corn, Chris Dymek, Peter Embi, Charles Friedman, Bob Greenes, Stan Huff, Dipak Kalra, Nancy Lorenzi, Leslie McIntosh, Blackford Middleton, Mark Musen, Jodyn Platt, Jerry Perry, Rachel Richesson, Chris Shaffer, Umberto Tachinardi, John Wilbanks

2nd Public Meeting – July 2019July 18-19, 2019Natcher Conference CenterNational Institutes of Health

17

190+ registrants 160+ participants24 posters 17 technical demonstrations4 workgroup sessions

#MobilizeCBK

www.MobilizeCBK.org

Panelists and Topics

• Allen Flynn, PharmD, PhD, Univ. of Michigan• MCBK Standards Workgroup

• Chris Dymek, EdD, Agency for Healthcare Research and Quality

• MCBK Sustainability for Mobilization and Inclusion Workgroup

• Jerry Perry, MLS, University of Arizona • MCBK Sustainability for Mobilization and Inclusion

Workgroup

• Questions and Discussion

CBK Artifact Lifecycle

AHRQ experience with CDS & CBK

CBK as Scholarly Communication

Allen Flynn, University of Michigan

CBK ARTIFACTLIFECYCLE

PART of PANEL:Data Analysis, New Knowledge, and then What? Perspectives on Mobilizing Computable Biomedical Knowledge

A single instance of machine-executable knowledge packaged for use

GENERAL EXAMPLES An implementation of a machine-learning algorithm with documentation in a ZIP file A computable guideline with user instructions available in an online repository A software container with a set of production rules and a rules engine to execute them A risk model implemented in a text file using a high-level programming language

What is a CBK artifact?

In the sense of 1s and 0s, these things can be seen as “data”, but in the MCBK movement we consider their meaning as computable knowledge

A series of changes in form that unfold over time, returning to a starting state

An array of ordered steps spanning the life of some thing

A repeating cycle of birth, life, and death

What is a lifecycle?

CBK Artifact Lifecycle in 9 SegmentsObjective:

Mobilize CBK artifacts by turning them into shareable, safe, and effective computable knowledge products

1. Create a CBK Artifact

DATA Computable Knowledge

Evidence Recommendations Rules

TECHNICAL WORKSCIENTIFIC WORK

2. Validate a CBK Artifact To what degrees do the logic and related outputs of a CBK artifact have …

face validity?

content validity?

criterion validity?

construct validity?

statistical validity?

external validity?

SCIENTIFIC WORK

3. Harden & Optimize a CBK Artifact

Original Fail-safe + Robust

HARDEN

OPTIMIZE

Original Highly performant

TECHNICAL WORK

4. Test & Certify a CBK Artifact

CBK Artifact Test/Certify (Review) Certified& Badged

TECHNICAL WORKSCIENTIFIC WORK

5. Localize & Calibrate a CBK Artifact

Certified Artifact Localize & Calibrate Useful

TECHNICAL WORKSCIENTIFIC WORKOPERATIONAL WORK

6. Deploy a CBK Artifact

Useful Artifact Made to Run Locally

TECHNICAL WORK

7. Integrate a CBK Artifact

Useful Artifact that Runs Connected to a source of input DATA

and atarget for output DATA

TECHNICAL WORK

8. Use & Evaluate a CBK Artifact

Operational CBK

SCIENTIFIC WORKOPERATIONAL WORK

Use ImpactDATA

Implement in Practice

9. Withdraw a CBK Artifact

Use Not in Use Anymore

SCIENTIFIC WORKOPERATIONAL WORK

Examples of Real CBK Artifacts• Computable guidelines

> Pharmacogenomic guidelines> Preventive medical service guidelines> Vaccination schedule guidelines

• Computable risk scores> Surgery risk score> Lung cancer diagnosis risk score> Cardiovascular disease risk score

• Computable complexity scores> Medication-regimen complexity score

THESE LIFE CYCLES CAN UNFOLD IN BROAD CONTEXTS THE GO BEYOND THE SCOPE OF A SINGLE ORGANIZATION.

Various Uses for CBK Artifacts• Clinical decision support

• Biomedical research and discovery

• Population and public health analyses and science

• Education of health stakeholders, including all providers

• Engineering work to make better CBK artifactsCBK ARTIFACTS GET USED FOR A WIDE VARITEY OF PURPOSES.

Bringing Computable Knowledge to the Point of Care

Chris Dymek, EdDDirector, Digital Healthcare Research Division

Health DatapaloozaFebruary 10, 2020

Agenda

• Background► AHRQ► Digital Healthcare Research► Vision for the Future

• A Focus on Clinical Decision Support (CDS)• Learnings from AHRQ’s CDS Efforts to date• Related Efforts• MCBK and Sustainability

AHRQ Mission

To produce evidence to make health care safer, higher quality, more accessible, equitable, and

affordable, and to work within HHS and with other partners to make sure that the evidence is

understood and used

37

Digital Healthcare Research

How can the various components of the ever evolving digital healthcare ecosystem best come together to positively affect healthcare quality, safety and effectiveness?

https://digital.ahrq.gov/

Vision for the Future

• Clinical & Contextual

• Patient-generated

My Data

• Guidelines• Relevant

research findings

Current Biomedical Knowledge

39

Vision for the Future

• Clinical & Contextual

• Patient-generated

My Data

• Guidelines• Relevant

research findings

Current Biomedical Knowledge

40

Needs to be computable and FAIR!

AHRQ Clinical Decision Support

• Long history of AHRQ investment in CDS• Recent initiative based on patient-centered outcomes research and

ACA legislative requirements (Sec 6301)► (b) INCORPORATION OF RESEARCH FINDINGS.—The Office [AHRQ/OCKT], in consultation

with relevant medical and clinical associations, shall assist users of health information technology focused on clinical decision support to promote the timely incorporation of research findings disseminated under subsection (a) into clinical practices and to promote the ease of use of such incorporation.

► (c) FEEDBACK - The Office shall establish a process to receive feedback from physicians, health care providers, patients, and vendors of health information technology focused on clinical decision support, appropriate professional associations, and Federal and private health plans about the value of the information disseminated and the assistance provided under this section.

41

AHRQ CDS Initiative (2016- )

42

Advancing evidence into practice through CDS and making CDS more shareable, standards-based and publicly- available

CDS Connect is…

A website► A place to discover shared CDS► https://cds.ahrq.gov/cdsconnect

A platform► To share CDS “artifacts”

A set of tools► Including a CDS Authoring Tool

and other open-source softwareA community

► Of users and work group members from a diverse set of perspectives

43

CDS Connect – Summary (2019)

44

Repository

57 Artifacts

10 Contributing organizations

70Registered Accounts

Federal Partners

VHA

CDC

ONC

CMS

NIH

Authoring Tool

205 Registered Accounts

Website

17,000 Unique Page

Views

75,000 Total Page

Views

5,000+Downloads

Work Group

140 Volunteers

80Distinct

Organizations

Presentations

15 Peer-Reviewed

at National Conferences

30+Invited at National

Conferences, Webinars,

other venues

Open Source Software

5 Software Packages

| 45 |Clinical Decision Support: Knowledge Translation

Knowledge Level Description Example

Level 1 (L1) Narrative Guideline for a specific disease that is written in the format of a peer-reviewed journal article

Level 2 (L2) Semi-structured

Flow diagram, decision tree, or other similar format that describes recommendations for implementation

Level 3 (L3) Structured Standards-compliant specification encoding logic with data model(s), terminology/code sets, value sets that is ready to be implemented

Level 4 (L4) Executable CDS implemented in a local execution environment (e.g., CDS that is live in an electronic health record (EHR)) or available via web services

Adapted from: Boxwala, A. A., et al. (2011). "A multi-layered framework for disseminating knowledge for computer-based decision support." Journal of the American Medical Informatics Association : JAMIA 18 Suppl 1: i132-139.

Pilots

• Three pilots of standards-based CDS using Health Level Seven (HL7) Clinical Quality Language (CQL) and HL7 Fast Healthcare Interoperability Resources (FHIR) Draft Standard for Trial Use 2 (DSTU2).► Created CDS for cholesterol management and pilot tested in a community

health center setting (Alliance Chicago)► Created and piloted a pain management summary dashboard presented

via a SMART on FHIR app (OCHIN)► Piloted certain United States Preventive Services Task Force (USPSTF)

recommendations on a consumer health platform (b.well) to demonstrate that such platforms can leverage CDS from CDS Connect

46

Select Pilot Learnings

• Mapping local codes to standardized codes is necessary and hard. Since every site may use different local codes, there is no global solution to mapping. In addition, mappings must be kept up to date as codes are added.

Interoperability

Local usefulness

47

Select Pilot Learnings

• Cannot assume that preferred terminologies are in use► Pilot organizations reported that many laboratories did not provide data with LOINC codes (or

any other standardized terminology). They also reported that many pharmacies provided medication data using National Drug Codes (NDC) rather than RxNorm. As a result, pilot partners were required to map these data and codes to the standardized systems expected by the CDS logic

• Some concepts still do not have standardized codes► As a result, placeholder codes may be needed until standardized codes are

available. • Concepts often have multiple possible representations.

► The best representation to use may vary from site to site or vendor to vendor. The following are some examples of different representations encountered in CDS Connect pilots: − pregnant: Condition (“pregnancy”) or Observation (“pregnancy status = is pregnant”)− on dialysis: Procedure (“dialysis”) or Condition (“dependence on dialysis”)

48

• Better dissemination/use of AHRQ resources to support evidence-based care transformation

• Make AHRQ evidence/knowledge FAIR in the context of a larger ecosystem

49

AHRQ evidence-based Care Transformation Support (ACTS)

Evidence Discovery and Implementation

50

EHRCDS/Knowledge

Service

Knowledge and Learning NetworkSupported by Standards, Processes, Tools, Trust Framework, Governance

Non-AHRQ Resources

Distribution

Source Repositories(examples)

Systematic Review Data Repository

Clinical Practice

Guidelines

Support for discovery,

interoperability, best practices

CDS Connect

Quality Improvement &

Shared Decision

Making Tools

Adapted from Middleton, Blumenfeld, et al, AHRQ evidence-based Care Transformation (ACTS) initiative, 2019

MCBK: Sustainability for Mobilization and Inclusion

• Seek to mobilize diverse stakeholders in an ongoing and active engagement around the value proposition of computable biomedical knowledge (CBK)

• Focus on communications and engagement with stakeholders as a necessary prerequisite in order to establish an equitable and FAIR CBK ecosystem

• Seek engagement through diverse and active communication channels with stakeholders from CBK: ► “creator” communities, including professional societies, accrediting bodies,

entrepreneurs and businesses; ► “hosting and dissemination” communities, including publishers, libraries and

commercial brokerages; ► “consumer” communities, including healthcare providers and clinical care

delivery systems, and healthcare provider and consumer advocacy organizations; and,

► funding communities, including Federal, charitable, philanthropic, association-based, and for-benefit entities that support innovation and equity in healthcare

51

Gerald (Jerry) Perry, University of Arizona

CBK AS SCHOLARLY COMMUNICATION: LIBRARIANS’ PERSPECTIVE

PART of PANEL:Data Analysis, New Knowledge, and then What? Perspectives on Mobilizing Computable Biomedical Knowledge

Emerging form of scholarly communication:Requires description w/

standardized schema (metadata);Requires registration

(credit to creator)

How do librarians view CBK?

Intended for “open” use? Needs to be discoverable/accessible…via registries; repositories…how to do with mutable,

potentially iterative objects?how to support associated

incumbencies (software, code, etc.)?

How do librarians view CBK?, cont.

Libraries preserve scholarly communications. Only the starting state iteration (Allen’s presentation)? How is CBK communicated; where/when

in the artifact lifecycle?Learning Health Sciences (journal)

manuscript format for CBKLooking for a repository home!

Challenge – aligning with CBK artifact lifecycle; issues for libraries

Role for peer review and post-”publication” comment?How to control for predatory CBK?Libraries teach info and digital

literacy, increasingly data literacy…

Challenge – aligning with CBK artifact lifecycle; issues for libraries (cont.)

Libraries/librarians champion right to read and consumer privacyLast un-surveilled spaces leftStrong commitment to social

justice, equitable access and FAIR representation

Insuring equity in creation/use of CBK…

Algorithms can/do oppress!Weapons of Math Destruction

(math modeling leading to negative consequences, C. O’Neil, formerly Columbia U)Algorithms of Oppression

(search engines reinforcing racism, S. Noble, UCS)

Insuring equity in creation/use of CBK, cont.

Patient advocacy groupsDon’t under-estimate!

Policy makersAlgorithmic Accountability Act – US House

legislation (4/19)JournalistsOpen Access/Open Science advocatesLibraries!

Insuring involvement of critical stakeholders

Informatics-speak:Standards Infrastructure Policy (FAIRness)Sustainability

MCBK community of practice, from library POVLibrary-speak:Metadata schemaDiscoverabilityEquity Inclusion

Working thru standards, infrastructure, policy and sustainability challenges: Is there a diversity component? If NO, why?

Is there a social justice component?What does this mean for ultimate

consumers (patients, communities)Across the artifact lifecycle, who decides?

Who’s POV is centered?

Applying an ethical lens…

The “Quintuple Aim” – National Academy of Medicine report, Artificial intelligence in Health Care: Hope not Hype, Promise not Peril “Ensure equity and inclusion are

stated and measured goalswhen designing and deploying health care interventions.”

Applying an ethical lens, cont.

Engage critical stakeholdersMobilize communications and incorporate

feedbackMCBK is a learning health system!Find opportunities to lead, educate, and

engage inclusively

Insuring equity and FAIRness…

Interdisciplinary nature of academic libraries makes them natural hub for supportNeeds outpacing supply of capacityOpportunities:How to balance high quality support with in-depth consultation in

era of scarcity? What strategies are successful in creating inclusive programming? How to recruit qualified talent/develop current staff? Credit and thanks to Dr. Jeff Oliver, UAL Data Science Specialist

Challenges in supporting data science thru libraries

Data Science Resources & Training

UA Libraries response to challenges:

https://datascience.arizona.edu/dsrt

Mobilize diverse stakeholders in active engagement around CBK value propositionEnvision a robust ecosystem:public-private partnershipssupporting open standardsgenerating value for usersengendering equity

Primary focus to-date on communications

Sustainability for Mobilization and Inclusion

#MobilizeCBK

www.MobilizeCBK.org