Post on 12-May-2023
Rachel Richesson, Duke UniversityAllen Flynn, University of MichiganChris Dymek, AHRQGerald Perry, University of Arizona
Data Analysis, New Knowledge, and then What? Perspectives on MobilizingComputable Biomedical Knowledge
#MobilizeCBK
Panelists
Rachel Richesson, PhD, MPH, Duke University @rrichesson MCBK Steering Committee
Allen Flynn, PharmD, PhD, University of Michigan MCBK Standards Workgroup
Chris Dymek, EdD, Agency for Healthcare Research and Quality MCBK Sustainability for Mobilization and Inclusion Workgroup
Gerald (Jerry) Perry, MLS, University of Arizona MCBK Sustainability for Mobilization and Inclusion Workgroup
#MobilizeCBK
Knowledge should be FAIR*
• Findable• Accessible• Interoperable• Reusable
*FAIR: https://www.force11.org/group/fairgroup/fairprinciples
MetadataLibraries
Data standards
•Descriptive attributes•Provenance•Licensing info.•Implementation guidance
•Performance Data•Monitoring
Impact of Bias and Missing Data
• Incomplete data• Under-representation of
populations• Misclassification• Measurement errors• Missing data for certain
groups
Knowledge is inappropriate or harmful for some patients
Interventions propagate disparity…
Some groups denied access to procedures; or receive inappropriate care
These patients do worse
Data not captured on some groups
“FAIRness” Enables Innovation
Data
Research, discovery, generation of evidence
Applications/Action-- targeted, personalized, useful,
usable, …
Knowledge
Mobilizing Computable Biomedical Knowledge (MCBK): A multi-stakeholder movement
Mission and Vision: A Manifesto Interim Home: The University of MichiganGovernance: Steering Committee
Activities to Date: Public meetings (2018, 2019) Four workgroups Web resources Webinars National and global collaborations (connections to Learning
Health Systems initiatives and communities)
It is no longer sufficient to represent knowledge only in words and picturesDecisions should be informed by the best available knowledge MCBK is committed to making use of knowledge to improve healthMCBK is committed to upholding the integrity, reliability, and validity
of computable knowledgeMCBK is committed to open, transparent, equitable, and inclusive
approaches to making computable knowledge FAIR
Highlights of MCBK Mission and Vision
MCBK Workgroups and Co-Chairs Standards for MCBK Robert Greenes and Bruce Bray
Technical Infrastructure for MCBK Leslie McIntosh and Chris Shaffer
Policy and Coordination to Ensure Quality and Trust Jodyn Platt and Blackford Middleton
Sustainability for Mobilization and Inclusion Chris Dymek and Gerald (Jerry) Perry
Steering Committee: Julia Adler-Milstein, Bruce Bray, Milton Corn, Chris Dymek, Peter Embi, Charles Friedman, Bob Greenes, Stan Huff, Dipak Kalra, Nancy Lorenzi, Leslie McIntosh, Blackford Middleton, Mark Musen, Jodyn Platt, Jerry Perry, Rachel Richesson, Chris Shaffer, Umberto Tachinardi, John Wilbanks
2nd Public Meeting – July 2019July 18-19, 2019Natcher Conference CenterNational Institutes of Health
17
190+ registrants 160+ participants24 posters 17 technical demonstrations4 workgroup sessions
Panelists and Topics
• Allen Flynn, PharmD, PhD, Univ. of Michigan• MCBK Standards Workgroup
• Chris Dymek, EdD, Agency for Healthcare Research and Quality
• MCBK Sustainability for Mobilization and Inclusion Workgroup
• Jerry Perry, MLS, University of Arizona • MCBK Sustainability for Mobilization and Inclusion
Workgroup
• Questions and Discussion
CBK Artifact Lifecycle
AHRQ experience with CDS & CBK
CBK as Scholarly Communication
Allen Flynn, University of Michigan
CBK ARTIFACTLIFECYCLE
PART of PANEL:Data Analysis, New Knowledge, and then What? Perspectives on Mobilizing Computable Biomedical Knowledge
A single instance of machine-executable knowledge packaged for use
GENERAL EXAMPLES An implementation of a machine-learning algorithm with documentation in a ZIP file A computable guideline with user instructions available in an online repository A software container with a set of production rules and a rules engine to execute them A risk model implemented in a text file using a high-level programming language
What is a CBK artifact?
In the sense of 1s and 0s, these things can be seen as “data”, but in the MCBK movement we consider their meaning as computable knowledge
A series of changes in form that unfold over time, returning to a starting state
An array of ordered steps spanning the life of some thing
A repeating cycle of birth, life, and death
What is a lifecycle?
CBK Artifact Lifecycle in 9 SegmentsObjective:
Mobilize CBK artifacts by turning them into shareable, safe, and effective computable knowledge products
1. Create a CBK Artifact
DATA Computable Knowledge
Evidence Recommendations Rules
TECHNICAL WORKSCIENTIFIC WORK
2. Validate a CBK Artifact To what degrees do the logic and related outputs of a CBK artifact have …
face validity?
content validity?
criterion validity?
construct validity?
statistical validity?
external validity?
SCIENTIFIC WORK
3. Harden & Optimize a CBK Artifact
Original Fail-safe + Robust
HARDEN
OPTIMIZE
Original Highly performant
TECHNICAL WORK
4. Test & Certify a CBK Artifact
CBK Artifact Test/Certify (Review) Certified& Badged
TECHNICAL WORKSCIENTIFIC WORK
5. Localize & Calibrate a CBK Artifact
Certified Artifact Localize & Calibrate Useful
TECHNICAL WORKSCIENTIFIC WORKOPERATIONAL WORK
7. Integrate a CBK Artifact
Useful Artifact that Runs Connected to a source of input DATA
and atarget for output DATA
TECHNICAL WORK
8. Use & Evaluate a CBK Artifact
Operational CBK
SCIENTIFIC WORKOPERATIONAL WORK
Use ImpactDATA
Implement in Practice
Examples of Real CBK Artifacts• Computable guidelines
> Pharmacogenomic guidelines> Preventive medical service guidelines> Vaccination schedule guidelines
• Computable risk scores> Surgery risk score> Lung cancer diagnosis risk score> Cardiovascular disease risk score
• Computable complexity scores> Medication-regimen complexity score
THESE LIFE CYCLES CAN UNFOLD IN BROAD CONTEXTS THE GO BEYOND THE SCOPE OF A SINGLE ORGANIZATION.
Various Uses for CBK Artifacts• Clinical decision support
• Biomedical research and discovery
• Population and public health analyses and science
• Education of health stakeholders, including all providers
• Engineering work to make better CBK artifactsCBK ARTIFACTS GET USED FOR A WIDE VARITEY OF PURPOSES.
Bringing Computable Knowledge to the Point of Care
Chris Dymek, EdDDirector, Digital Healthcare Research Division
Health DatapaloozaFebruary 10, 2020
Agenda
• Background► AHRQ► Digital Healthcare Research► Vision for the Future
• A Focus on Clinical Decision Support (CDS)• Learnings from AHRQ’s CDS Efforts to date• Related Efforts• MCBK and Sustainability
AHRQ Mission
To produce evidence to make health care safer, higher quality, more accessible, equitable, and
affordable, and to work within HHS and with other partners to make sure that the evidence is
understood and used
37
Digital Healthcare Research
How can the various components of the ever evolving digital healthcare ecosystem best come together to positively affect healthcare quality, safety and effectiveness?
https://digital.ahrq.gov/
Vision for the Future
• Clinical & Contextual
• Patient-generated
My Data
• Guidelines• Relevant
research findings
Current Biomedical Knowledge
39
Vision for the Future
• Clinical & Contextual
• Patient-generated
My Data
• Guidelines• Relevant
research findings
Current Biomedical Knowledge
40
Needs to be computable and FAIR!
AHRQ Clinical Decision Support
• Long history of AHRQ investment in CDS• Recent initiative based on patient-centered outcomes research and
ACA legislative requirements (Sec 6301)► (b) INCORPORATION OF RESEARCH FINDINGS.—The Office [AHRQ/OCKT], in consultation
with relevant medical and clinical associations, shall assist users of health information technology focused on clinical decision support to promote the timely incorporation of research findings disseminated under subsection (a) into clinical practices and to promote the ease of use of such incorporation.
► (c) FEEDBACK - The Office shall establish a process to receive feedback from physicians, health care providers, patients, and vendors of health information technology focused on clinical decision support, appropriate professional associations, and Federal and private health plans about the value of the information disseminated and the assistance provided under this section.
41
AHRQ CDS Initiative (2016- )
42
Advancing evidence into practice through CDS and making CDS more shareable, standards-based and publicly- available
CDS Connect is…
A website► A place to discover shared CDS► https://cds.ahrq.gov/cdsconnect
A platform► To share CDS “artifacts”
A set of tools► Including a CDS Authoring Tool
and other open-source softwareA community
► Of users and work group members from a diverse set of perspectives
43
CDS Connect – Summary (2019)
44
Repository
57 Artifacts
10 Contributing organizations
70Registered Accounts
Federal Partners
VHA
CDC
ONC
CMS
NIH
Authoring Tool
205 Registered Accounts
Website
17,000 Unique Page
Views
75,000 Total Page
Views
5,000+Downloads
Work Group
140 Volunteers
80Distinct
Organizations
Presentations
15 Peer-Reviewed
at National Conferences
30+Invited at National
Conferences, Webinars,
other venues
Open Source Software
5 Software Packages
| 45 |Clinical Decision Support: Knowledge Translation
Knowledge Level Description Example
Level 1 (L1) Narrative Guideline for a specific disease that is written in the format of a peer-reviewed journal article
Level 2 (L2) Semi-structured
Flow diagram, decision tree, or other similar format that describes recommendations for implementation
Level 3 (L3) Structured Standards-compliant specification encoding logic with data model(s), terminology/code sets, value sets that is ready to be implemented
Level 4 (L4) Executable CDS implemented in a local execution environment (e.g., CDS that is live in an electronic health record (EHR)) or available via web services
Adapted from: Boxwala, A. A., et al. (2011). "A multi-layered framework for disseminating knowledge for computer-based decision support." Journal of the American Medical Informatics Association : JAMIA 18 Suppl 1: i132-139.
Pilots
• Three pilots of standards-based CDS using Health Level Seven (HL7) Clinical Quality Language (CQL) and HL7 Fast Healthcare Interoperability Resources (FHIR) Draft Standard for Trial Use 2 (DSTU2).► Created CDS for cholesterol management and pilot tested in a community
health center setting (Alliance Chicago)► Created and piloted a pain management summary dashboard presented
via a SMART on FHIR app (OCHIN)► Piloted certain United States Preventive Services Task Force (USPSTF)
recommendations on a consumer health platform (b.well) to demonstrate that such platforms can leverage CDS from CDS Connect
46
Select Pilot Learnings
• Mapping local codes to standardized codes is necessary and hard. Since every site may use different local codes, there is no global solution to mapping. In addition, mappings must be kept up to date as codes are added.
Interoperability
Local usefulness
47
Select Pilot Learnings
• Cannot assume that preferred terminologies are in use► Pilot organizations reported that many laboratories did not provide data with LOINC codes (or
any other standardized terminology). They also reported that many pharmacies provided medication data using National Drug Codes (NDC) rather than RxNorm. As a result, pilot partners were required to map these data and codes to the standardized systems expected by the CDS logic
• Some concepts still do not have standardized codes► As a result, placeholder codes may be needed until standardized codes are
available. • Concepts often have multiple possible representations.
► The best representation to use may vary from site to site or vendor to vendor. The following are some examples of different representations encountered in CDS Connect pilots: − pregnant: Condition (“pregnancy”) or Observation (“pregnancy status = is pregnant”)− on dialysis: Procedure (“dialysis”) or Condition (“dependence on dialysis”)
48
• Better dissemination/use of AHRQ resources to support evidence-based care transformation
• Make AHRQ evidence/knowledge FAIR in the context of a larger ecosystem
49
AHRQ evidence-based Care Transformation Support (ACTS)
Evidence Discovery and Implementation
50
EHRCDS/Knowledge
Service
Knowledge and Learning NetworkSupported by Standards, Processes, Tools, Trust Framework, Governance
Non-AHRQ Resources
Distribution
Source Repositories(examples)
Systematic Review Data Repository
Clinical Practice
Guidelines
Support for discovery,
interoperability, best practices
CDS Connect
Quality Improvement &
Shared Decision
Making Tools
Adapted from Middleton, Blumenfeld, et al, AHRQ evidence-based Care Transformation (ACTS) initiative, 2019
MCBK: Sustainability for Mobilization and Inclusion
• Seek to mobilize diverse stakeholders in an ongoing and active engagement around the value proposition of computable biomedical knowledge (CBK)
• Focus on communications and engagement with stakeholders as a necessary prerequisite in order to establish an equitable and FAIR CBK ecosystem
• Seek engagement through diverse and active communication channels with stakeholders from CBK: ► “creator” communities, including professional societies, accrediting bodies,
entrepreneurs and businesses; ► “hosting and dissemination” communities, including publishers, libraries and
commercial brokerages; ► “consumer” communities, including healthcare providers and clinical care
delivery systems, and healthcare provider and consumer advocacy organizations; and,
► funding communities, including Federal, charitable, philanthropic, association-based, and for-benefit entities that support innovation and equity in healthcare
51
Gerald (Jerry) Perry, University of Arizona
CBK AS SCHOLARLY COMMUNICATION: LIBRARIANS’ PERSPECTIVE
PART of PANEL:Data Analysis, New Knowledge, and then What? Perspectives on Mobilizing Computable Biomedical Knowledge
Emerging form of scholarly communication:Requires description w/
standardized schema (metadata);Requires registration
(credit to creator)
How do librarians view CBK?
Intended for “open” use? Needs to be discoverable/accessible…via registries; repositories…how to do with mutable,
potentially iterative objects?how to support associated
incumbencies (software, code, etc.)?
How do librarians view CBK?, cont.
Libraries preserve scholarly communications. Only the starting state iteration (Allen’s presentation)? How is CBK communicated; where/when
in the artifact lifecycle?Learning Health Sciences (journal)
manuscript format for CBKLooking for a repository home!
Challenge – aligning with CBK artifact lifecycle; issues for libraries
Role for peer review and post-”publication” comment?How to control for predatory CBK?Libraries teach info and digital
literacy, increasingly data literacy…
Challenge – aligning with CBK artifact lifecycle; issues for libraries (cont.)
Libraries/librarians champion right to read and consumer privacyLast un-surveilled spaces leftStrong commitment to social
justice, equitable access and FAIR representation
Insuring equity in creation/use of CBK…
Algorithms can/do oppress!Weapons of Math Destruction
(math modeling leading to negative consequences, C. O’Neil, formerly Columbia U)Algorithms of Oppression
(search engines reinforcing racism, S. Noble, UCS)
Insuring equity in creation/use of CBK, cont.
Patient advocacy groupsDon’t under-estimate!
Policy makersAlgorithmic Accountability Act – US House
legislation (4/19)JournalistsOpen Access/Open Science advocatesLibraries!
Insuring involvement of critical stakeholders
Informatics-speak:Standards Infrastructure Policy (FAIRness)Sustainability
MCBK community of practice, from library POVLibrary-speak:Metadata schemaDiscoverabilityEquity Inclusion
Working thru standards, infrastructure, policy and sustainability challenges: Is there a diversity component? If NO, why?
Is there a social justice component?What does this mean for ultimate
consumers (patients, communities)Across the artifact lifecycle, who decides?
Who’s POV is centered?
Applying an ethical lens…
The “Quintuple Aim” – National Academy of Medicine report, Artificial intelligence in Health Care: Hope not Hype, Promise not Peril “Ensure equity and inclusion are
stated and measured goalswhen designing and deploying health care interventions.”
Applying an ethical lens, cont.
Engage critical stakeholdersMobilize communications and incorporate
feedbackMCBK is a learning health system!Find opportunities to lead, educate, and
engage inclusively
Insuring equity and FAIRness…
Interdisciplinary nature of academic libraries makes them natural hub for supportNeeds outpacing supply of capacityOpportunities:How to balance high quality support with in-depth consultation in
era of scarcity? What strategies are successful in creating inclusive programming? How to recruit qualified talent/develop current staff? Credit and thanks to Dr. Jeff Oliver, UAL Data Science Specialist
Challenges in supporting data science thru libraries
Data Science Resources & Training
UA Libraries response to challenges:
https://datascience.arizona.edu/dsrt
Mobilize diverse stakeholders in active engagement around CBK value propositionEnvision a robust ecosystem:public-private partnershipssupporting open standardsgenerating value for usersengendering equity
Primary focus to-date on communications
Sustainability for Mobilization and Inclusion