Overdiagnosis of Urinary Tract Infection Linked to Overdiagnosis of ...

Overdiagnosis of Urinary Tract Infection Linked to Overdiagnosis

of Pneumonia: A Multi-Hospital Cohort Study

SUPPLEMENTARY APPENDIX

BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any relianceSupplemental material placed on this supplemental material which has been supplied by the author(s) BMJ Qual Saf

doi: 10.1136/bmjqs-2021-013565–4.:10 2022;BMJ Qual Saf, et al. Gupta A

Table of Contents Page

Pneumonia Inclusion/Exclusion Criteria 3

Urinary Tract Infection Inclusion/Exclusion Criteria 7

Abstractor Training Process 11

Data Collection Process 12

Quality Assurance Process 13

Overdiagnosis Data Collection Project 17

a. Project Description 17

b. Pneumonia Case Facilitator Form 24

c. Pneumonia Case Reviewer Form 30

d. UTI Case Facilitator Form 31

e. UTI Case Reviewer Form 36

f. Number of Reviewers and Interrater Reliability 37

Strength of Correlation 38

*All tables and figures included were specifically created by co-authors for this publication or for the Michigan Hospital Medicine Safety Consortium and have not been presented elsewhere, unless otherwise specified. The Michigan Hospital Medicine Safety Consortium has approved inclusion of all materials in the manuscript.

2



Pneumonia Inclusion and Exclusion Criteria

Pneumonia Inclusion:

Eligible cases will include patients at the time of abstraction who are:

• Admitted to a medicine service (day #1/day #2 of inpatient hospitalization) • Discharged from the participating hospital with a diagnosis of pneumonia (via discharge

diagnosis ICD 10 codes) • Received an antibiotic (any) on day #1 or day #2 of the hospital encounter (excludes

antibiotics that intended for prophylaxis, surgery, Clostridioides difficile [C.Diff], etc.) • No other pneumonia exclusion criteria apply

Exclusion:

If a patient meets any of the following criteria, they will be ineligible

• Patient pregnant and/or breastfeeding • Patient under the age of 18 • Patient admitted (day #1 or day #2 of inpatient hospitalization) to an intensive care unit (ICU).

o Note: Patients that are admitted to a medical service then are transferred to ICU after

Day 1/ Day 2 are eligible o Patient has an order for palliative care/comfort care during the hospital encounter:Note:

The patient is eligible for abstraction if they have a palliative care consult or hospice

consult only and no orders for palliative care or comfort care are written during the

encounter. The patient is also eligible for abstraction if the patient is on palliative care

for pain management only or if the patient has an order for palliative care/comfort

care after transfer to the ICU. • Patient left against medical advice (AMA) or refused recommended medical care during the

hospital encounter

o Note 1: Patient will be ineligible if he/she refuses any dose of antibiotic intended for

treatment of pneumonia, and/or has an MI-POST (Michigan Physician Orders for

Scope of Treatment) in place. o Note 2: A patient is eligible for abstraction if they left against medical advice or refused

recommended medical care after transfer to an intensive care unit (ICU).

o Note 3: A patient is eligible for abstraction if they refuse medications not related to the

infectious disease state such as venous-thromboembolism prophylaxis.

• Patient on mechanical ventilation during the hospital encounter o Note 1: A patient is ineligible if they are on a Ventilator, Mechanical Ventilator, or have a

documented intubation during a surgical procedure.

o Note 2: A patient is eligible if they use BiPAP or CPAP. A patient is eligible for abstraction

if they have mechanical ventilation after transfer to an intensive care unit (ICU).

• Patient with Cystic Fibrosis. o Note: Patients with cystic lung disease are eligible

3



• Patient on antibiotic treatment (except topical antibiotics) for a concomitant bacterial infection

anytime during the hospital encounter (eg., Osteomyelitis) except those infections that are

secondary to the infectious disease state being abstracted (ex: sepsis, empyema, bacteremia,

COPD, upper respiratory infections, acute chest syndrome, clostridioides difficile infection

[C.Diff], etc.) during the hospital encounter (ER, Obs, Inpatient). For example, for a pneumonia case, if the patient is admitted with and are receiving antibiotic treatment for pneumonia and

also receive treatment for a UTI they would be excluded. Patients with bronchitis, sinusitis,

COPD or upper respiratory infections receiving treatment are eligible.

o Note 1: Patients undergoing treatment with antibiotics ordered for prophylaxis of a

potential concomitant infection are excluded.

o Note 2: Topical antibiotics include ointments (e.g. erythromycin ointment), salves,

powders and ophthalmic (eye) drops or ear drops (e.g. Ciprofloxacin ophthalmic

solution). Patients who receive topical antibiotics are eligible for inclusion. o Note 3: Patients who receive antibiotics for surgery pre- or peri-operative are eligible.

o Note 4: Patients who receive antibiotics given for conditions which are not concomitant

infections such as Common Variable Immune Deficiency (CVID), rosacea, hepatic

encephalopathy, Non-Alcoholic SteatoHepatitis (NASH), Rheumatoid Arthritis (RA), and

Syndrome of Inappropriate Antidiuretic Hormone Secretion (SIADH) are eligible.

o Note 5: Patients who are admitted on treatment for C. Diff and continue that treatment

throughout the hospitalization are eligible.

o Note 6: Patients who are not treated with an antibiotic for a concomitant infection

during the hospital encounter, but are discharged from the index hospital encounter

with a prescription for an antibiotic for a concomitant infection are excluded.

• Patient admitted to a non-medicine service on day 1 or day 2 of the hospital encounter

(example: gynecology, Subacute rehab floor). • Patient discharge falls within the 30-day window of a previously recorded/abstracted admission

(Pneumonia or UTI) • Patient transferred from another hospital (including inpatient short stay unit, inpatient psych

unit, inpatient rehab) o Note 1: Patients transferred from emergency department to emergency department (ED

to ED) are eligible.

o Note 2: If the patient is on an inpatient psych or inpatient rehab unit within your hospital

and is transferred (even if this is technically considered a discharge and readmission for

the purposes of insurance/billing) to an inpatient medicine unit in your hospital, this

case is ineligible as they are being admitted from another inpatient setting.

• Patient has any history of a heart, liver, lung, pancreas, bone marrow, stem cell, or

hematopoietic stem cell transplant • Patient has a history of a kidney (renal) transplant within the year prior to the hospital

encounter • Patient received treatment for the rejection of a kidney (renal) transplant in the 6 months prior

to the hospital encounter

4



• Patient has a diagnosis of HIV with CD4 count <200 on admission or within the 90 days prior

to the hospital encounter

o Note 1: CD4 count can be found in the laboratory section. It can be part of an

immunodeficiency panel and is labeled as absolute CD4 count.

o Note 2: If the patient has a diagnosis of HIV without a CD4 count < 200 or a CD4 count <

200 without HIV, they are eligible.

• Patient has neutropenia (ANC <0.5) on Day #1 or Day #2 of the hospital encounter o Note 1: If the Absolute Neutrophil Count (ANC) value is listed as “Suppressed” and

the White Blood Cell (WBC) Count is 0, this patient should be excluded.

o Note 2: ANC stands for Absolute Neutrophil count. This can be found in the

laboratory section as a part of a CBC with differential. Include labs titled Abs Polynuc

Neutrophil Count, Neutrophil number

• Patient with documentation of fungal pneumonia, mycobacterium/mycobacterial infection,

tuberculosis and/or aspergillus/Aspergillosis during the hospital encounter

o Note 1: Includes patients with Aspergillus fumigatus. o Note 2: Patients with a positive AFB culture collected during the encounter should be

excluded. If the positive AFB culture was collected during the hospital encounter but

the results came after the hospital encounter, the case should still be excluded. o Note 3: Patients diagnosed with Mycoplasma Pneumoniae are eligible. o Note 4: Patients with a positive fungal sputum culture and positive fungal BAL cultures

collected during the hospital encounter are excluded.

o Note 5: If only the lower respiratory gram stain is positive for yeast, this ALONE

would not make the patient ineligible.

• The following are the scenarios that would make a patient INELIGIBLE (i.e. a COVID exclusion): o Patients with documentation of a positive COVID test in the 30 days prior to the hospital

encounter (OR)

o Patients with a positive COVID test during the hospital encounter

(OR) o Patients with a discharge ICD-10 code for COVID-19 (OR)

o Patients with a patient-reported positive COVID test prior to the hospital encounter

(even if not confirmed by visible lab results) (OR)

o Patient receives COVID treatment even after a negative COVID test results (treatment

can include Decadron, Remdesivir, other steroids, etc.)

▪ Note 1: Patients with a discharge ICD-10 code of Z20.828- Contact with and

(suspected) exposure to other viral communicable diseases are eligible. ▪ Note 2: Patients with a discharge ICD-10 code of Z20.822- Contact with and

(suspected) exposure to COVID-19 are eligible if they do not meet the above

criteria. ▪ Note 3: Patients who receive COVID treatment that is STOPPED after a negative

COVID test result (treatment can include Decadron, Remdesivir, other steroids,

etc.) are eligible as long as they do not meet the above criteria. ▪ Note 4: Patients that have persistently positive COVID-19 PCRs during the

hospital encounter are excluded, even if they did not have a discharge ICD-10

code for COVID or the patient did not receive any treatment for COVID.

5



▪ Note 5: Patients with a discharge ICD-10 code of Z86.16 – Personal History

of COVID are eligible if they do not meet the above criteria.

6



Urinary Tract Infecton Inclusion and Exclusion

Criteria Urinary Tract Infection Inclusion:

Eligible cases will include patients at the time of abstraction who are:

• Admitted to a medicine service (day #1/day #2 of inpatient hospitalization) • Discharged from the participating hospital • Had a positive urine culture during the hospital encounter (ER, Obs, Inpatient). INCLUDE:

Patients treated for positive urine culture that was later determined to be contaminated

(and no other exclusion criteria apply) are eligible. o Note 1: A positive urine culture is defined as those flagged as positive by the participating

site’s lab/institution. o Note 2: If the first positive urine culture was collected during the hospital encounter and

results were not finalized until after the patient was discharged, this patient is eligible.

Exclusion:

If a patient meets any of the following criteria, they will be ineligible:

• Patient was pregnant and/or breastfeeding • Patient was under the age of 18 • Patient was in an intensive care unit (ICU) on the day the first positive urine culture was

collected, the 3 calendar days before the first positive urine culture was collected, or the 3

days after the first positive urine culture was collected

• Patient has an order for palliative care/comfort care during the hospital encounter o Note: The patient is eligible for abstraction if they have a palliative care consult or hospice

consult only and no orders for palliative care or comfort care are written during the

encounter. The patient is also eligible for abstraction if the patient is on palliative care for

pain management only or if the patient has an order for palliative care/comfort care after

transfer to the ICU. • Patient left against medical advice (AMA) or refused recommended medical care during

the hospital encounter

o Note 1: A patient will be ineligible if he/she refused any dose of antibiotic intended for

treatment (recommended care) of the urinary tract infection.

o Note 2: A patient is eligible for abstraction if they left against medical advice or refused

recommended medical care after transfer to an intensive care unit (ICU).

o Note 3: A patient is eligible for abstraction if they refuse medications not related to the

infectious disease state such as venous-thromboembolism prophylaxis.

o Note 4: A patient is eligible for abstraction if they refuse a medical intervention such as

vital signs, intermittent straight catheter, foley catheter placement, etc.

• Presence of urinary stents anytime during the hospital encounter o Note 1: If there is documentation of urinary stent placement in the patient’s medical

history without a known placement date and removal date, the patient is ineligible.

o Note 2: If there is documentation of urinary stent placement within the 3 months prior to

the index hospitalization without documentation of removal, the patient is ineligible.

7



o Note 3: If there is documentation of urinary stent placement within the 3 months prior

to the index hospitalization with documentation of removal, the patient is eligible.

o Note 4: If there is documentation of urinary stent placement that occurred more than

3 months prior to the index hospitalization, the patient is eligible (regardless of

documentation of removal).

• Presence of a nephrostomy tube anytime during the hospital encounter • Urologic surgery or procedure during the hospital encounter.

o Examples include but are not limited to cystoscopy, lithotripsy, urteroscopy, suprapubic

catheter, ureteral stent, cystogram etc. Procedures or surgeries do not have to be billed OR

time. o Note: If a suprapubic catheter was inserted prior to the encounter and is changed during

the encounter this is not considered a procedure. Renal biopsy is not considered a

urologic surgery or procedure.

• Patient has a history of urinary diversion surgery or urinary anatomical issues o Note 1: Patients are INELIGIBLE if they have had Urinary Diversion surgery or anatomical

issues including, but not limited to, the following: urostomy, urethra reconstruction,

nephrostomy, neobladder, ileal conduit, pelvic kidney, horseshoe kidney, medullary

sponge kidney, augmentation cystoplasty (ileum), vesico-vaginal fistula repair or non-

repair, Mitrofanoff construction of continent urinary reservoir, urethrovaginal fistula

closure, direct vision internal urethrotomy, repair for traumatic rupture of bladder, ADPKD

(autosomal dominant polycystic kidney disease), cystic dysplasia of one kidney o Note 2: Patients are ELIGIBLE if they have any of the following which are not considered

urinary diversion surgery or anatomical issues: Transurethral Resection of the Prostate

(TURP), artificial urinary sphincter, cystocele repair, nephrectomy, urethroplasty, renal

artery stent, electronic stimulator implanted in the bladder, bladder suspension, bladder

sling, urethral sling, suprapubic catheter, pyeloplasty, anastomotic uretroplasty,

congenital absence of one kidney, prostatectomy, rectovaginal fistula, a patient with a

neurogenic bladder who self-catheterizes, and chronic radiation cystitis, Transurethral

Resection of Bladder Tumor (TURBT), bladder diverticulum, surgically corrected

hypospadias

• Patient is receiving active treatment and/or prophylaxis for a UTI at time of encounter or

transfer to the hospital (includes Macrobid, other antibiotics, and antifungals prescribed

for chronic UTI symptoms) o Note: Exclude gentamicin rinses as active treatment and/or prophylaxis

• Patient received antibiotic prophylaxis for surgery (pre- or peri-operative antibiotics within

the 7 days after collection of the first positive urine culture during the encounter o Note 1: Patient is ELIGIBLE if they received pre-operative antibiotics for prophylaxis for

surgery prior to the first positive urine culture (but during the hospital encounter) or/and

8 days or more after collection of the positive urine culture. o Note 2: Patients are INELIGIBLE if they receive operative antibiotics within the 7 days

after the collection of the first positive urine culture. Patient is eligible if they receive

Hiprex at time of admission or transfer to the hospital.

• Patient on treatment (except topical antibiotics) for a concomitant bacterial infection anytime

during the hospital encounter (ex: Epididymo-orchitis, Osteomyelitis, prostatitis, perinephric

8



abscess, renal abscess, diverticulitis) except those infections that are secondary to the UTI

(i.e. sepsis, bacteremia or Clostridioides difficile infection [C.Diff])

o Note 1: Patients undergoing treatment with antibiotics ordered for prophylaxis of a

potential concomitant infection are excluded. Example: A patient undergoing prophylaxis for spontaneous bacterial peritonitis with Ciprofloxacin upon admission is ineligible

for abstraction. o Note 2: Topical antibiotics include ointments (e.g. erythromycin ointment), salves,

Powders, ophthalmic drops (eye drops) and ear drops. Patients who receive

topical antibiotics are eligible for inclusion. o Note 3: A concomitant bacterial infection also includes cases where a pathogen

identified on a positive blood culture collected during the hospital encounter is not listed

as a pathogen on the first positive urine culture collected during the hospital encounter.

Patients are ineligible if they receive treatment for the blood culture pathogen that is different than the urine culture pathogen.

o Note 4: Patients who receive antibiotics given for conditions which are non-concomitant

infections such as Common Variable Immune Deficiency (CVID), rosacea, hepatic

encephalopathy, Non-Alcoholic SteatoHepatitis (NASH), and Syndrome of inappropriate antidiuretic hormone secretion (SIADH) are eligible.

o Note 5: Patients who are admitted on treatment for C.Diff and continue that

treatment throughout the hospitalization are eligible. o Note 6: Patients who are not treated with an antibiotic for a concomitant infection during the

hospital encounter but are discharged from the index hospital encounter with a prescription for an antibiotic for a concomitant infection are excluded.

o Note 7: If a blood pathogen PCR shows a pathogen that is not reported on the first positive

urine culture and this blood pathogen is treated with antibiotics, this patient is excluded. o Note 8: Patients that are treated for a concomitant infection outside of the UTI

Window should be excluded.

• Patient receiving antibiotics for treatment of COPD during the hospital encounter • Patient admitted to a non-medicine service (example: gynecology, Subacute Rehab floor) • Patient discharge falls within the 30-day window of a previously recorded/abstracted

admission” (Pneumonia or UTI) • Patient has any history of a kidney, heart, liver, lung, pancreas, bone marrow, stem cell,

or hematopoietic stem cell transplant • Patient has a diagnosis of HIV with CD4 count less than 200 on admission or within the 90

days prior to the hospital encounter o Note 1: CD4 count can be found in the laboratory section. It can be part of an

immunodeficiency panel and is labeled as absolute CD4 count. o Note 2: If the patient has a diagnosis of HIV without a CD4 count < 200 or a CD4 count

< 200 without HIV, they are eligible.

• Patient has neutropenia (ANC <0.5) on Day 1 or Day 2 of the hospital encounter o Note 1: If the ANC value is listed as “Suppressed” and the White Blood Cell (WBC) Count

is 0, this patient should be excluded.

o Note 2: ANC stands for Absolute Neutrophil count. This can be found in the

laboratory section as a part of a CBC with differential.

9



• The following are the scenarios that would make a patient INELIGIBLE (i.e. a COVID exclusion): o

Patients with documentation of a positive COVID test in the 30 days prior to the hospital encounter (OR)

o Patients with a positive COVID test during the hospital encounter (OR)

o Patients with a discharge ICD-10 code for COVID-19 (OR) o Patients with a patient-reported positive COVID test prior to the hospital encounter (even if

not confirmed by visible lab results) (OR)

o Patient receives COVID treatment even after a negative COVID test result (treatment can

include Decadron, Remdesivir, other steroids, etc.)

▪ Note 1: Patients with a discharge ICD-10 code of Z20.828- Contact with and

(suspected) exposure to other viral communicable diseases are eligible. ▪ Note 2: Patients with a discharge ICD-10 code of Z20.822- Contact with and

(suspected) exposure to COVID-19 are eligible if they do not meet the above criteria. ▪ Note 3: Patients who receive COVID treatment that is STOPPED after a negative COVID

test result (treatment can include Decadron, Remdesivir, other steroids, etc.) are

eligible as long as they do not meet the above criteria. ▪ Note 4: Patients that have persistently positive COVID-19 PCRs during the hospital

encounter are excluded, even if they did not have a discharge ICD-10 code for COVID

or the patient did not receive any treatment for COVID. ▪ Note 5: Patients with a discharge ICD-10 code of Z86.16 – Personal History of

COVID are eligible if they do not meet the above criteria.

10



Abstractor Training

All data included in the Michigan Hospital Medicine Safety (HMS) Consortium is collected by a

Clinical Data Abstractor. Each participating hospital is required to have a Clinical Data Abstractor. There

are three individual pieces to the HMS Antimicrobial (ABX) Abstractor training: pre-training, full-day

training, and post-training.

Pre-Training

The pre-training period is a self-learning period utilizing an online Learning Management System

(LMS), which is comprised of written modules, videos, and quizzes. During this period, each abstractor

reviews an introductory module which explains the HMS initiative in detail, as well as online modules for

ABX. These modules review each section of each form in the HMS database. After each section is

completed, the abstractor completes a quiz which mirrors the questions that are asked in the live HMS

database. They are then given a fake training case to utilize when answering the quiz questions in the

LMS. If the abstractor gets an answer incorrect, the LMS software provides rationale and additional

education in live time. After the abstractor has completed their online training modules and quizzes in

the LMS, they are instructed to enter a second fake training case in the training site of the HMS ABX

database. This simulates the data collection the abstractor will complete when their training has

concluded and familiarizes them with the HMS database. The pre-training can take approximately 8

hours to complete, and all elements are required to be completed prior to attending full-day training.

Full Day Training

The full-day training is conducted by one of the HMS Quality Assurance Coordinators, either on-

site or virtually with the abstractor. The full- day training begins with a more detailed review of the HMS

initiatives, performance index, coordinating center structure, and abstraction workflow. The ABX portion

of the full-day training involves reviewing ABX eligibility, follow-up review periods, difficult concepts of

abstraction, and required abstraction elements. Additionally, the Quality Assurance Coordinator reviews

the responses to the fake training case that the abstractor entered into the database in detail,

answering any questions the abstractor may have. The overview and ABX session typically take up 4-5

hours of the full -day training session, and the rest of the session is utilized to review the PICC/Midline

training module (if applicable to that abstractor).

Post-Training

After the full-day training, the abstractor is sent four additional fake training cases to enter into

the HMS live database. The abstractor enters each case and submits their responses to the Coordinating

Center for review. The abstractor must pass each training case with a score of at least 95% accuracy to

complete their training. Additionally, during the post-training period, the abstractor must also review an

additional LMS module related to the creation of their eligibility list, which is utilized for case sampling,

and complete an attestation of understanding. This portion of the training can take up to a week or

more to complete, depending on the abstractor. After the conclusions of all portions of their training,

abstractors are then assigned to their home site in the HMS ABX Database and allowed to begin real

case abstraction.

11



Data Collection Process

Chart Abstraction

Data collection for the HMS Antimicrobial Use Initiative will involve the abstraction of patient data for

16 eligible cases every 2 weeks. A specific sampling process is employed for the project to prevent case

selection bias. The process uses a 14-Day cycle that works as follows: The clinical data abstractor

abstracts data on 16 eligible cases from discharge dates covered by a 2 -week period. To minimize

sampling bias, abstractors are expected to select cases from every day of the cycle being abstracted, if

possible.

The HMS Antimicrobial Use Initiative focuses on two (2) infectious disease states: pneumonia & urinary

tract infections. Eight (8) cases of each infectious disease state should be collected each cycle when

possible. There are standard data entry forms that apply to both infectious disease states, in addition to

infectious disease state-specific forms.

Sampling Procedure

The data collection for each eligible case will capture data from 1 year prior to the hospital

encounter through 30 days following the hospital discharge.

To accomplish the selection of cases, the clinical data abstractor completes the following:

1. For each abstraction cycle, obtain a list of patients admitted to all medical services during the

timeframe covered by the cycle. The HMS Coordinating Center creates a sample list to

provide to the site’s informatics/IT department. The list may or may not already have exclusion filters applied, depending on the capabilities of the hospital.

2. Organize the list chronologically by date and time of discharge. Cases to be abstracted will be

from patient discharges 30 days prior. 3. Begin with the first patient on the chronological list and review the patient record to confirm

that the patient should not be excluded. If excluded, document which exclusion criteria

applied. If not excluded, the patient is eligible and data abstraction should occur. 4. Complete a 30-Day Follow-up for patients who are still alive at the end of the hospital stay and

were not transferred to the intensive care unit (ICU) during the hospitalization of interest. (Data

collection stops after a transfer to ICU occurs.) 5. Complete a 30-Day Phone Call for patients who are eligible for a phone call. 6. Repeat this process by continuing to review patients from the chronological list making sure that

cases are distributed across all the days of the 14-day cycle to the greatest extent possible

(meaning discharge dates across all days of the week Sunday-Saturday) until 16 eligible cases are

identified and abstracted for each 2-week period.

12



Michigan Hospital Medicine Safety (HMS) Collaborative Quality Assurance Process

Details on the HMS Databases

HMS has three (3) databases, or registries, currently in production: PICC/Midline, Antimicrobial Use

(ABX), and Sepsis. The databases are used as the primary hub of case abstraction, data reporting, case

volume analysis, resource gathering and abstraction queries. All of the HMS databases use both Drupal

and LimeSurvey software for data collection/abstraction, which are maintained by the HMS

Coordinating Center and our database administrative team. For data reporting, HMS uses Business

Objects software to allow hospitals participating in HMS to access their updated data on a daily basis.

For resource gathering and abstraction queries, HMS utilizes a link to a Zendesk guide, which allows

abstractors to submit questions regarding data abstraction/reporting, obtain updated data definitions

and access resources and quality improvement tools.

Upgrades

Upgrades to the HMS databases occur at least once per year. Upgrades may occur more frequently,

depending on updates made to the project throughout the year and changing data needs for quality

improvement projects. During an upgrade, the HMS database undergoing updates is taken offline and is

unavailable for data abstraction. Upgrades may occur if we have spelling or grammatical errors to fix,

selections to add or remove, questions to add or remove, branching questions to add or remove, and/or

functionalities to improve or update. After completion of an upgrade, data entered prior to the upgrade

is archived and restored in the database. If new questions are added, the abstractors are not expected

to return to previously entered cases to enter new data fields as case entry can span several years. The

HMS Coordinating Center keeps track of all updates using a ticketing system and the data analytics and

statistician team are made aware of all updates.

Data Validation: Audits

Audits are conducted to ensure that the data is being collected consistently across all participating

hospitals. The goal is to identify issues with the abstraction process so that they can be appropriately

addressed via education and/or changes to the data entry system. Each HMS-participating hospital is

audited by a trained member of the Coordinating Center at least once per year. To date, 437

pneumonia cases and 385 UTI cases have been audited. It is the expectation that each audited site will

attain a 95% or greater rate of accuracy to receive full points on the HMS Performance Index. To

determine the audit score, the auditor calculates a score for each individual case based on the average

number of audit fields as noted below (see Medical Record Review). Then using the individual scores for

each case, an overall audit score is calculated by averaging all of the audit cases combined. If a site

receives a score of less than 95% on an audit, every attempt will be made to re-audit that site in the

same year.

The audit consists of four parts: medical record review, review of eligibility lists, review of

inclusion/exclusion criteria and practices, and post-audit follow up.

13



Medical Record Review:

The primary focus of the audit is a medical record review of pre-selected cases by one to three HMS

auditors. For each initiative, key complication cases are required to be audited to ensure accurate

outcome measures for reporting purposes. Clostridioides difficle is the primary complication that is

required for audit for the antimicrobial initiative. Prior to the audit, the primary auditor queries the data

analytics team to obtain the list of required complication cases that are due for audit and a random

sample of additional non-complication cases. On average 7 to 10 cases are audited if one auditor is

present. If a site has a large number of unaudited complication cases, a second or third auditor will join

to complete additional cases. The list of cases is distributed to the abstractor 1 to 2 weeks in advance of

the audit. Prior to sending the list of audited cases, the abstractor is locked from making updates to

previously completed cases. Upon the on-site audit, the auditor(s) independently reviews the medical

documentation for each case from the hospital’s Electronic Medical Record (EMR) and compares it to what was entered into the HMS database. At the end of the audit day, the auditor’s case findings and

discrepancies between the EMR and the information entered into the HMS databases (if applicable) will

be reviewed in detail with the abstractor. At the resolution of the audit, these discrepancies (if validated

as incorrect by both the abstractor and auditor), are corrected in the database by the abstractor to

ensure case accuracy. The auditor will also provide additional education, as needed, as issues are

identified. If during the medical record review a completed eligible case is determined by the auditor to

be ineligible, a score of 90% is assigned to the case and added to the overall average score. The average

number of fields that are audited for the pneumonia initiative is 2,072. The average number of fields

audited for the positive urine culture is 2,420.

Eligibility List Review:

The second item reviewed during an audit is the eligibility/discharge lists and coding at the site being

audited. Prior to the audit, the abstractor connects with their hospital’s information technology (IT)

group for the coding used to generate their eligibility/discharge lists for each project PICC/Midline (one

list for both) and Antimicrobial (separate lists for Pneumonia and Positive Urine Culture). This coding is

reviewed by the auditor and feedback is provided regarding updates that need to be made to the

coding, if applicable.

Inclusion/Exclusion Criteria Review:

The final item reviewed during an audit is inclusion/exclusion criteria. The purpose of this review is to

ensure that the abstractor understands the inclusion/exclusion criteria for each project and is applying

those criteria appropriately when reviewing cases. At least one case for each project deemed ineligible

by the abstractor is randomly selected and reviewed with the auditor(s). Once a case is identified, the

abstractor shows the auditor(s), in the medical record, the reason the case was excluded from

abstraction. If a case was deemed ineligible by the abstractor, but was determined through this review

that it was actually eligible for abstraction, another case from the same project will be reviewed until a

legitimate ineligible case is found. If the abstractor has incorrectly identified a case as ineligible, the

auditor(s) will provide additional on-site education about eligibility criteria. If more than 2 randomly-

selected cases were deemed ineligible by the abstractor, but are determined to be eligible for

abstraction after review, a score of 90% will be added to the final audit summary for each additional

case that is found to be eligible.

14



Post-Audit Follow Up:

After the audit has concluded, the primary auditor composes a summary of the findings, including

specific areas to update in the HMS databases, education provided to the site during the audit, and a

summary of any findings from the eligibility/discharge list review. The final audit summary is provided to

the site within two to three weeks of completion of the audit. This summary will be sent to the site’s abstractor(s), quality administrator, and physician champion. The summary will include a percentage

score for the audit, which is calculated based on the average of the scores for all cases reviewed. Upon

receiving the final audit summary, the abstractor(s) has three months from the date of receipt to make

all updates in the HMS database noted in the final report. The final audit score is then factored into the

sites performance index scorecard for the given year. During a typical year, 5% of the performance index

is associated with the audit score(s) completed during the performance year.

Data Validation: Data Checker

Each HMS database (PICC/Midline and ABX) has a robust data checker that can identify in real time

errors in abstraction that have occurred on a case-by-case basis. Abstractors are trained to run a data

checker on each case before submitting it to the database so that any data errors are identified at the

time of the initial abstraction and can be corrected prior to submission. Additionally, a live daily report is

available, which provides a culmination of all data errors on all cases entered into the database that an

abstractor is able to see in order to correct potential discrepancies in data abstraction.

The PICC/Midline database has a total of 92 individual checks for PICC cases and 92 individual checks for

Midline cases. The PICC/Midline data checks range from notifications of potential date errors (i.e., date

of discharge entered is prior to the date of inpatient admission) to validation of patient status at the

time of line insertion (i.e., the patient status at the time of line insertion is identified as intensive care

unit (ICU), but there is no ICU information captured in the database for that case).

The ABX database has a total of 226 individual checks. These data checks range from potential length of

stay issues (i.e., a case where the length of stay = 1) to verification that cultures are appropriately

entered (i.e., it was noted in a form that a respiratory culture was collected, but no respiratory culture

form was completed for that case). The data checker is also utilized to highlight which days in our Daily

Entry tab need to be completed for that case. As a note, the “Daily Entry tab” is the section of forms that are utilized to enter daily antibiotic and vital sign information for ABX cases. After the entry of all

data forms, the abstractor runs the data checker for that case and upon completion of the data checker

validation, the Daily Entry tab days that need to be completed are highlighted in red and all days that do

not need to be completed remain gray. This allows the abstractors to know exactly which days need to

be entered and verify that they have fully completed data abstraction for that case.

Data Validation: Global Data Checks

The HMS Coordinating Center conducts global data checks on an ad hoc basis during the data analysis

process to identify any issues that might occur across the entire database that may not be included in

the data checker. These global data checks are typically run when we identify an error as part of another

process, such as coding a report and realizing something does not pull into the report as expected. The

HMS Coordinating Center will do data queries to sites throughout the year with prompts to analyze their

15



data in accordance with the medical record if we notice discrepancies outside of the data checker

elements.

Data Validation: Site Specific

Each hospital receives site specific data reports via a printed version quarterly and daily within the

database/registry. Included in these reports are the sites overall score for each measure by quarter and

a detailed list of cases that have been identified as opportunities for improvement (i.e. also titled fall-

outs). Each hospital is encouraged to review these fall-outs with their local team to perform audit and

feedback, identify trends, and assist with overall quality improvement. Occasionally, during this review

the local team will identify a potential issue with how the fall-out was determined based on the clinical

scenario. In some instances, the case is reviewed and justification for the coding/calculation is

reinforced to the local site. In other instances, modifications to the code and/or additional

modifications to the data registry questions are required. Typically, the latter is more common at the

initial launch of a new measure. For more longstanding measures, modifications are rare.

16



Case review project summary, definitions, and guidelines

HMS has been collecting data on patients discharge from the hospital with a pneumonia diagnosis

(pneumonia cohort) or receiving a positive urine culture (urinary tract infection [UTI] cohort) since

2017. Our processes for data collection and validation are explained elsewhere in this appendix. In each

cohort, patients are classified as either having the disease state (pneumonia or UTI) or not. Patients not

meeting criteria are considered overdiagnosed. The definitions of how these are classified are below.

Definition of over-diagnosis of urinary tract infection

Urine Culture

Definition of over-diagnosis of pneumonia

Pneumonia

17



To assess the accuracy of the algorithms developed by the Michigan Hospital Medicine Safety

Consortium (HMS), we conducted a series of case review sessions. The topics for these sessions were

developed in conjunction with our technical experts and based on feedback from hospitalists,

infectious diseases physicians, pharmacists, and abstractors at participating hospitals. We evaluated

each topic raised to identify the number of patients in the cohort that fit the established criterion. From

within each topic, cases that met the criterion were randomly selected from participating hospitals and

reviewed during the case review sessions. During the project, we conducted 15 different case study

sessions. Sessions identified in bold represent overdiagnosis of infection. The number of cases reviewed

is identified in ( ).

Pneumonia

• Pneumonia treated with <3 days of antibiotics (6) • Overdiagnosis of pneumonia with atelectasis noted on the Chest X-Ray/Chest CT (6) • Overdiagnosis of pneumonia with COPD exacerbation at discharge (5) • Pneumonia with COPD exacerbation at discharge with ≤3 days of antibiotics (4) • Pneumonia with COPD exacerbation at discharge >3 days of antibiotics (5) • Pneumonia with negative procalcitonin and >3 days of antibiotics (4) • Pneumonia with CHF exacerbation at discharge (5) • Pneumonia when patients did not have radiographic findings of pneumonia in days 1 to 3 of

hospitalization (4) • Pneumonia patients with radiographic components and only signs or symptoms of cough,

sputum, dyspnea or hypoxemia (5)

Urinary Tract Infection

• Urinary Tract Infection treated with no Antibiotics (4) • ASB with Altered Mental Status (4) • ASB with SIRS (two or more during encounter) (3) • ASB with ≥2 SIRS criteria on a single day (4) • ASB with symptom of urinary retention, received inpatient antibiotics and history of

neurogenic bladder/urinary retention (5) • ASB patients admitted from home and discharged to skilled nursing facility (5)

Each case review session included two to four physician volunteers to review the cases. These physicians

worked for hospitals participating in the HMS consortium. Infectious diseases (ID), pulmonary/critical

care, and hospital medicine physicians participated in this research. When possible, we tried to have

physicians with varying backgrounds in a single session. Each case review session lasted for two hours

and all sessions were hosted virtually.

Prior to review, the session facilitator – a hospitalist physician – provided an overview the case review

session structure. This introduction included a review of the definitions used to define over-diagnosis,

the case review processes – including the documents they would review, the documentation they would

use to record their assessments, and the scales and definitions that were used to evaluate the data they

were given.

18



1. Review of the forms used to facilitate the conversation.

These forms are included in this appendix in their entirety.

2. Description of data presented for review.

All individuals participating in the case review process were provided with several different types of

data.

First, they were provided with a summary of the data collected by HMS as an Excel spreadsheet.

Screen shots of these spreadsheets are included below. The data included is “dummy” data in order to protect patient confidentiality.

Pneumonia

19



20




Additionally, individuals were provided with documentation from the patient’s hospitalization. All information was de-identified using HIPAA guidelines to ensure that patient’s confidentiality was

preserved. Data provided to individuals participating in each case review included:

21



Pneumonia

• Emergency department note • Reports (not images) from X-Ray/CT scans from day 1 or day 2 of hospital admissions • Patient’s history and physical • Discharge summary • Discharge summary from 1st readmission after initial discharge (within 90 days), if applicable


• Emergency department note • Patient’s history and physical • Discharge summary • Discharge summary from 1st readmission after initial discharge (within 90 days), if applicable

3. Overview of the process for each case review

• Each reviewer reviewed all documentation independently and completed a case review

form • The facilitator asked reviewers to report case review scores prior to discussion • After everyone reports their scores, if there was disagreement, the facilitator asked

the group to discuss areas of disagreement • All reviewers were asked to identify how HMS could improve our measures of infection by

collection or interpreting data differently.

4. Review of the data scales and definitions

a. Likelihood of bacterial infection:

• Absent: patient’s presentation is explained by another disease or is not consistent with UTI/bacterial pneumonia

• Possible: patient’s presentation is consistent with another disease or diagnosis, but could

be consistent with UTI/bacterial PNA • Probable: patient’s presentation is consistent with UTI/bacterial PNA, but could be

explained by another disease or diagnosis • Definite: Patient’s presentation is consistent with UTI/bacterial PNA and cannot

be explained by another disease or diagnosis

b. Appropriateness

Uses a nine-point scale, where 1 = highly inappropriate; 5 = neutral or uncertain; 9 = highly

appropriate

22



5. Criteria used to assess whether an outcome present in a readmission was the result of a side

effect from the antibiotics received during the index hospitalization.

Adapted from the World Health Organization-the Uppsala Monitoring Center (WHO-UMC) Causality Categories available at: https://www.who.int/medicines/areas/quality_safety/safety_efficacy/WHOcausality_assessment.pdf

23



https://www.who.int/medicines/areas/quality_safety/safety_efficacy/WHOcausality_assessment.pdf

Case Reviewer Information – Number of Reviewers

Our project included 15 case reviews, that included a total of 20 physicians from 14 different hospitals.

Physician reviewers could participate in more than one session. A summary of the number of reviewers

in our sessions is noted below.

Kappa was calculated when there was agreement for all reviewers in a session.

92% of cases were agreed to be over-diagnosed with UTI

Kappa for reviewer agreement was 0.72, indicating substantial agreement

94% of cases were agreed to be over-diagnosed with pneumonia

Kappa for reviewer agreement was 0.72, indicating substantial agreement

Cohen suggested the Kappa result be interpreted as follows: values ≤ 0 as indicating no agreement and 0.01–0.20 as none to slight, 0.21–0.40 as fair, 0.41– 0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.00 as almost perfect agreement.

37



Interpreting the Size of a Correlation Coefficient

Size of Correlation Interpretation

.90 to 1.00 (-.90 to -1.00) Very high positive (negative) correlation

.70 to .90 (-.70 to -.90) High positive (negative) correlation

.50 to .70 (-.50 to -.70) Moderate positive (negative) correlation

.30 to .50 (-.30 to -.70) Low positive (negative) correlation

.00 to .30 (.00 to -.30) Negligible correlation

Table adapted from:

Mukaka MM. Statistics corner: A guide to appropriate use of correlation coefficient in medical

research. Malawi Med J. 2012;24(3):69-71

38



Overdiagnosis of Urinary Tract Infection Linked to Overdiagnosis of ...

Documents

Transcript of Overdiagnosis of Urinary Tract Infection Linked to Overdiagnosis of ...