FORECASTING HOSPITAL EMERGENCY DEPARTMENT ...

FORECASTING HOSPITAL EMERGENCY DEPARTMENT VISITS FOR

RESPIRATORY ILLNESS USING ONTARIO’S TELEHEALTH SYSTEM

An Application of Real-Time Syndromic Surveillance

to Forecasting Health Services Demand

by

ALEXANDER GORDON PERRY

A thesis submitted to the Department of Community Health and Epidemiology

in conformity with the requirements for

the degree of Master of Science

Queen’s University

Kingston, Ontario, Canada

August 2009

Copyright © Alexander Gordon Perry, 2009

i

Abstract

Background: Respiratory illnesses can have a substantial impact on population health

and burden hospitals in terms of patient load. Advance warnings of the spread of such

illness could inform public health interventions and help hospitals manage patient

services. Previous research showed that calls for respiratory complaints to Telehealth

Ontario are correlated up to two weeks in advance with emergency department visits for

respiratory illness at the provincial level.

Objectives: This thesis examined whether Telehealth Ontario calls for respiratory

complaints could be used to accurately forecast the daily and weekly number of

emergency department visits for respiratory illness at the health unit level for each of the

36 health units in Ontario up to 14 days in advance in the context of a real-time

syndromic surveillance system. The forecasting abilities of three different time series

modeling techniques were compared.

Methods: The thesis used hospital emergency department visit data from the National

Ambulatory Care Reporting System database and Telehealth Ontario call data and from

June 1, 2004 to March 31, 2006. Parallel Cascade Identification (PCI), Fast Orthogonal

Search (FOS), and Numerical Methods for Subspace State Space System Identification

(N4SID) algorithms were used to create prediction models for the daily number of

emergency department visits using Telehealth call counts and holiday/weekends as

predictors. Prediction models were constructed using the first year of the study data and

ii

their accuracy was measured over the second year of data. Factors associated with

prediction accuracy were examined.

Results: Forecast error varied widely across health units. Prediction error increased with

lead time and lower call-to-visits ratio. Compared with N4SID, PCI and FOS had

significantly lower forecast error. Forecasts of the weekly aggregate number of visits

showed little evidence of ability to accurately flag corresponding actual increases.

However, when visits were aggregated over a four day period, increases could be flagged

more accurately than chance in six of the 36 health units accounting for approximately

half of the Ontario population.

Conclusions: This thesis suggests that Telehealth Ontario data collected by a real-time

syndromic surveillance system could play a role in forecasting health services demand for

respiratory illness.

iii

Acknowledgements

This project was unique and challenging because it combined elements of Epidemiology

and Engineering. The following individuals and organizations deserve recognition for

their roles in this project:

Dr. Kieran Moore, Adam van Dijk, and the other members of the Queen’s Public Health

Informatics (QPHI) team for their advice and for providing the resources necessary to

carry out the project

Dr. Will Pickett whose open-mindedness and willingness to supervise this cross-

disciplinary project made it possible

Dr. Michael Korenberg of the Department of Electrical and Computer Engineering for his

insightful suggestions and for agreeing to supervise a project outside his home department

in addition to the many other projects with which he is involved

Dr. Miu Lam for his advice on statistical aspects of the project

The Kingston General Hospital for its financial support through the KGH Scholarship

Don McGuinness for his advice and help with ICD code translation

Dr. Linda Levesque for her advice and support

Finally, I would like to thank my grandfather, Dr. V. R. Perry, for his enthusiasm in my

return to school to study Epidemiology

iv

Table of Contents Abstract ..................................................................................................................................................... i

Acknowledgements .................................................................................................................................. iii

Table of Contents ..................................................................................................................................... iv

List of Acronyms and Abbreviations ........................................................................................................ vi

List of Symbols....................................................................................................................................... vii

List of Tables ......................................................................................................................................... viii

List of Figures .......................................................................................................................................... x

Chapter 1 Introduction ....................................................................................................................... 1

1.1 Background ............................................................................................................................ 1

1.1.1 Real-Time Syndromic Surveillance ................................................................................ 1

1.1.2 Applications of Syndromic Surveillance......................................................................... 2

1.2 Study Objectives..................................................................................................................... 3

Chapter 2 Literature Review and Study Rationale ............................................................................... 5

2.1 Previous Research on the Telehealth Ontario Call-Emergency Department Visit Relationship

for Respiratory Illness .......................................................................................................................... 5

2.2 Time Series Forecasting .......................................................................................................... 7

2.3 Previous Research on Health Service Demand Forecasting ...................................................... 8

2.4 Gaps in Existing Knowledge ..................................................................................................13

2.5 Study Rationale .....................................................................................................................15

2.5.1 Conceptual Framework .................................................................................................15

2.5.2 Addressing Gaps in Knowledge ....................................................................................16

Chapter 3 Study Design and Methods ................................................................................................19

3.1 Study Population, Setting, and Design ....................................................................................19

3.2 Data Sources and Ethics Approval .........................................................................................19

3.3 Definitions.............................................................................................................................20

3.4 Emergency Department Visits: the NACRS Database .............................................................25

3.4.1 Coverage and Data Quality ...........................................................................................25

3.4.2 Inclusion/Exclusion ......................................................................................................26

3.5 Telehealth Ontario Calls ........................................................................................................27

3.5.1 Coverage and Data Quality ...........................................................................................27

3.5.2 Inclusion/Exclusion ......................................................................................................27

3.6 Confounders ..........................................................................................................................28

3.7 Geographic Grouping of Telehealth Calls and Emergency Visits ............................................29

3.8 Analytic Techniques for Establishing the Relationship between Calls and Visits .....................30

3.8.1 Background ..................................................................................................................30

3.8.2 Numerical Algorithms for Subspace State Space System Identification ..........................33

v

3.8.3 Fast Orthogonal Search .................................................................................................35

3.8.4 Parallel Cascade Identification ......................................................................................38

3.8.5 Model Implementation ..................................................................................................40

3.9 Measures ...............................................................................................................................47

Chapter 4 Results ..............................................................................................................................62

4.1 Summary Statistics of Telehealth Ontario Calls and Emergency Department Visits by Health

Unit ..............................................................................................................................................62

4.2 Plots of Daily Calls and Daily Visits over Study Period ..........................................................68

4.3 Qualitative Forecast Assessment ............................................................................................71

4.4 Quantitative Forecast Assessment ..........................................................................................85

4.5 Ability to Predict Increases ....................................................................................................98

4.5.1 Increases in Emergency Visits Aggregated over a Seven Day Window ........................ 102

4.5.2 Increases in Emergency Visits Aggregated over Four Day Windows ........................... 104

Chapter 5 Discussion and Conclusions ............................................................................................ 106

5.1 Summary of Key Findings ................................................................................................... 106

5.1.1 Forecast Accuracy ...................................................................................................... 106

5.1.2 Usefulness of Telehealth Ontario Calls versus Knowledge of Upcoming Holidays and

Weekends to Predict Future Visits for Respiratory Illness .............................................................. 108

5.1.3 Comparison of Forecasting Methods ........................................................................... 109

5.2 Results in the Context of the Existing Literature ................................................................... 110

5.3 Study Strengths ................................................................................................................... 113

5.4 Study Limitations ................................................................................................................ 115

5.5 Application of Results and Implications for Future Research ................................................ 118

References ............................................................................................................................................. 121

Appendices ............................................................................................................................................ 130

APPENDIX A: Ethics Approval ....................................................................................................... 130

APPENDIX B: Ability of Forecasts to Predict Increases in Emergency Department Visits ................. 131

APPENDIX C: One-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency

Visits for All Ontario Health Units .................................................................................................... 155

vi

List of Acronyms and Abbreviations

Acronym/Abbreviation Definition

ARIMA AutoRegressive Integrated Moving Average

ARX AutoRegressive with Exogenous Input

ARMAX AutoRegressive Moving Average with Exogenous Input

AUROC Area Under the Receiver Operating Characteristic

CIHI Canadian Institute of Health Information

ED Emergency Department

FOS Fast Orthogonal Search

FN False Negative

FP False Positive

FSA Forward Sortation Area

FWER Family-Wise Error Rate

GARCH Generalized Autoregressive Conditional Heterokedasticity

ICD International Classification of Disease Codes

LN Linear Nonlinear

MA Moving Average

MAPE Mean Absolute Percentage Error or Mean Absolute Prediction Error

MCC Matthew’s Correlation Coefficient

MSE Mean Square Error

N4SID Numerical Methods for Subspace State Space System Identification

NACRS National Ambulatory Care Reporting System

NHS National Health Service

NPV Negative Predictive Value

PCI Parallel Cascade Identification

PEM Prediction Error Method

PHLS Public Health Laboratory Service

PHU Public Health Unit

PPV Positive Predictive Value

QPHI Queen’s Public Health Informatics Group

RMS Root Mean Square

ROC Receiver Operating Characteristic

RSV Respiratory Syncytial Virus

Sn Sensitivity

Sp Specificity

SS Subspace

TN True Negative

TP True Positive

UK United Kingdom

vii

List of Symbols

Note: The following list is not exhaustive and provides a reference only to symbols

found in the body of the text with no associated equation. Symbols used in equations are

defined immediately following the equation.

Symbol Definition

C Number of candidate terms in the Fast Orthogonal Search model

cm Candidate term in the Fast Orthogonal Search difference equation model

j0 Number of y factors in a term pm of the Fast Orthogonal Search difference equation

model

j1 Number of u1 factors in a term pm of the Fast Orthogonal Search difference equation

model

j2 Number of u2 factors in a term pm of the Fast Orthogonal Search difference equation

model

K Kalman gain matrix

k Time index shift

pm mth term in the Fast Orthogonal Search difference equation model

M Number of terms in the Fast Orthogonal Search difference equation model

N Sample size/Total number of time values in a time series

n Time index

w1(n) Error in Telehealth Ontario calls at time index n

w2(n) Error in Holidays/Weekends at time index n

wm(n) Orthogonal basis function for the set of terms in the Fast Orthogonal Search

difference equation model

u1(n) Telehealth Ontario calls time series at time index n

u2(n) Indicator variable time series for holidays/weekends at time index n

vy(n) Error in Emergency department visits time series at time index n

y(n) Actual emergency department visits time series at time index n

z(n) Predicted emergency department visits time series at time index n

viii

List of Tables Table 1: Literature on Forecasting Health Services Demand (1996-2008).................................................. 9

Table 2: ICD-10CA Codes Used to Identify Emergency Visits for Respiratory Complaints from the

NACRS Data Set .....................................................................................................................................22

Table 3: Guidelines Used to Identify Calls for Respiratory Complaints from the Telehealth Ontario Data

Set ...........................................................................................................................................................24

Table 4: NACRS Fields Used in Analysis of Emergency Department Visits .............................................25

Table 5: Telehealth Ontario Call Database Fields Used in Analysis ..........................................................27

Table 6: Structure Choices Required for each Type of Prediction Model ..................................................43

Table 7: Total Telehealth Ontario Calls and Emergency Department Visits for Respiratory Complaints by

Health Unit over Study Period ..................................................................................................................64

Table 8: Summary Statistics of Daily Telehealth Ontario Call and Emergency Department Visit Activity

for Respiratory Complaints by Health Unit over Study Period...................................................................65

Table 9: Ratio of the Median Number of Daily Telehealth Ontario Calls to Median Number of Daily

Hospital Emergency Department Visits by Health Unit .............................................................................66

Table 10: Ages of Individuals Telehealth Ontario Calls were Concerning and Ages of Emergency

Department Visit Patients by Health Unit .................................................................................................67

Table 11: Summary Statistics of the Error (Predicted-Actual) in Daily Forecasts for the (Approximate) City

of Toronto Health Unit over the Validation Dataset ..................................................................................86

Table 12: Summary Statistics of the Error (Predicted-Actual) in Daily Forecasts for the (Approximate)

Grey Bruce Health Unit over the Validation Dataset .................................................................................86

Table 13: Summary Statistics of the Error (Predicted-Actual) in the Forecasted Aggregate Number of

Weekly Hospital Emergency Department Visits for Respiratory Illness for the (Approximate) City of

Toronto Health Unit over the Validation Dataset ......................................................................................87

Table 14: Summary Statistics of the Error (Predicted-Actual) in the Forecasted Aggregate Number of

Weekly Hospital Emergency Department Visits for Respiratory Illness for the (Approximate) Grey Bruce

Health Unit over the Validation Dataset ...................................................................................................87

Table 15: %MSE (MAPE) for 0-Day-Ahead Forecasts of Hospital Emergency Department Visits for

Respiratory Illness for Each of the 36 Health Units in Ontario over the Validation Dataset ........................88









ix

Table 20: Parameter Estimates for the Multilevel Regression Model of Transformed %MSE, MSET ........95

Table 21: Health Units where Forecasts Show Ability to Discriminate between Increases and Decreases in

the Aggregate Number of Visits over the Next Four Days ....................................................................... 104

Table 22: Health Units where Forecasts Show Ability to Predict 10% Nominal Increases in the Aggregate

Number of Visits over the Next Four Days ............................................................................................. 105

x

List of Figures Figure 1: Hypothetical Framework Illustrating the Temporal Relationship between Telehealth Ontario

Calls and Emergency Department Visits at the Individual Level ...............................................................15

Figure 2: Hypothetical Framework Illustrating the Temporal Relationship between Telehealth Ontario

Calls and Emergency Department Visits at the Population Level...............................................................16

Figure 3: Inclusion/Exclusion of Hospital Emergency Department Visits for Respiratory Complaints .......26

Figure 4: Inclusion/Exclusion of Telehealth Ontario Calls for Respiratory Complaints .............................28

Figure 5: The Dynamic Relationship between Calls and Visits Time Series Framed as a System

Identification Problem..............................................................................................................................32

Figure 6: Prediction of Aggregate Hospital Visits over a period of 1-7 Days in the Future (1 Window

Ahead) and a period of 8-14 Days in the Future (2 Windows Ahead) ........................................................49

Figure 7: Ability to Predict Important Increases in Visits over a Seven-Day Window ...............................55

Figure 8: Threshold used for Flagging an Important Increases in the Number of Emergency Department

Visits .......................................................................................................................................................57

Figure 9: Plot of the Daily Number of Emergency Department Visits and Telehealth Ontario Calls for

Respiratory Complaints for the Approximate City of Toronto Health Unit from June 1, 2004 to March 31,

2006 ........................................................................................................................................................69

Figure 10: Plot of the Daily Number of Emergency Department Visits and Telehealth Ontario Calls for

Respiratory Complaints for the Approximate Grey Bruce Health Unit from June 1, 2004 to March 31, 2006

................................................................................................................................................................70

Figure 11: Zero-Day Ahead Emergency Department Visit Forecast for Respiratory Complaints over the

Validation Dataset for the (Approximate) City of Toronto Health Unit (using all three Forecasting Methods)

................................................................................................................................................................72

Figure 12: Forecasting Errors (Predicted - Actual) for Zero-Day Ahead Emergency Department Visit

Prediction for Respiratory Complaints over the Validation Dataset for the (Approximate) City of Toronto

Health Unit ..............................................................................................................................................73

Figure 13: Zero-Day-Ahead Emergency Department Visit Forecast for Respiratory Complaints over the

Validation Dataset for the (Approximate) Grey Bruce Health Unit (using all three Forecasting Methods) ..74

Figure 14: Forecasting Errors (Predicted - Actual) for Zero-Day Ahead Emergency Department Visit

Prediction for Respiratory Complaints over the Validation Dataset for the (Approximate) Grey Bruce

Health Unit ..............................................................................................................................................75

Figure 15: Five-Day-Ahead Emergency Department Visit Forecast for Respiratory Complaints over the

Validation Dataset for the (Approximate) City of Toronto Health Unit (using all three Forecasting Methods)

................................................................................................................................................................76

Figure 16: Forecasting Errors (Predicted - Actual) for Five-Day Ahead Emergency Department Visit

Prediction for Respiratory Complaints over the Validation Dataset for the (Approximate) City of Toronto

Health Unit ..............................................................................................................................................77

xi

Figure 17: Five-Day-Ahead Emergency Department Visit Forecast for Respiratory Complaints over the

Validation Dataset for the (Approximate) Grey Bruce Health Unit (using all three Forecasting Methods) ..78


Prediction for Respiratory Complaints over the Validation Dataset for the (Approximate) Grey Bruce

Health Unit ..............................................................................................................................................79

Figure 19: One-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency Visits

using all Three Forecasting Methods and the Corresponding Weekly Aggregate Number of Actual Visits for

the (Approximate) City of Toronto Health Unit ........................................................................................81

Figure 20: Two-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency Visits


the (Approximate) City of Toronto Health Unit ........................................................................................82

Figure 21: One-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency Visits


the (Approximate) Grey Bruce Health Unit ..............................................................................................83

Figure 22: Two-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency Visits


the (Approximate) Grey Bruce Health Unit ..............................................................................................84

Figure 23: Regression Model Estimates for the %MSE versus Prediction Lead for Each Forecasting

Method for a Ratio of Median Daily Number of Calls to Median Daily Number of Visits of 0.6 ................96





Figure 26: Plot Illustrating Analyses of PCI-Predicted versus Actual Sequence of Increases/Decreases in

Emergency Department Visits One Week in Advance for the City of Toronto Health Unit ...................... 101

1

Chapter 1 Introduction

1.1 Background

This thesis investigates the use of a nursing telephone help line, Telehealth Ontario, as a

source of real-time data for syndromic surveillance of respiratory illness in Ontario. This

builds on past research by members of the Queen’s Public Health Informatics (QPHI)

group (1-5). Specifically, it investigates a practical application of syndromic surveillance

using Telehealth Ontario to predict demand for hospital emergency department services

for the treatment of respiratory illness.

1.1.1 Real-Time Syndromic Surveillance

In the context of public health, surveillance is the continuous monitoring of the

occurrence and distribution of disease in a population and it involves the collection,

analysis, interpretation and dissemination of information for this purpose (6). Timeliness,

sensitivity, and specificity of detected events are key characteristics of an effective

surveillance system (7)(8)(9). Timeliness can pose one of the greatest challenges to

surveillance as gathering and assimilating the information from its various sources can be

slow (7)(3). Threats of an influenza pandemic and bioterrorism, and events such water

contamination in Walkerton, Ontario, and North Battleford, Saskatchewan, and the SARS

outbreak in Hong Kong and Toronto, have generated interest in the development of more

timely surveillance systems(10-13)(9)(14).

2

Syndromic surveillance systems rely on the ―…detection of clinical case features…‖ or

health behaviours ―…that are discernable before confirmed diagnoses are made...‖ and

exploit the fact that ―…ill persons may exhibit behavioural patterns, symptoms, signs, or

laboratory findings that can be tracked through a variety of sources‖(15). This approach

combined with real-time automated data collection and anomaly detection methods has

given rise to real-time syndromic surveillance systems. The strength of these systems is

that they address the issue of timeliness (7)(9). Ideally, real-time syndromic surveillance

systems collect data that are leading indicators of disease, provide good coverage of the

target population, accurately reflect the level of disease in the target population, and are

readily available from electronic sources. Examples of such data sources include calls to

nursing help lines, over-the-counter drug sales, emergency medical services dispatch, and

emergency department triage information (7)(5). Real-time syndromic surveillance

systems automatically integrate and process these data into syndrome categories, scan the

resulting time series for unusual numbers of events, and provide rapid dissemination of

any anomalies to the appropriate individuals (7,16)(5)(17)(15).

1.1.2 Applications of Syndromic Surveillance

Early detection of respiratory illnesses, including influenza, has obvious benefits to public

health and the health of individuals. Timely warning of increased illness in a population

could be used in planning public health interventions to prevent further spread of disease

such as vaccination (18), especially vaccination in vulnerable populations which fall short

of national targets (19), and screening of health care staff (20). Less obvious, by giving

3

an estimate of the prevalence of respiratory illness, these systems could also help

facilitate clinical health professionals’ diagnostic and treatment decisions by providing a

measure of the pre-test probability of respiratory illness. Influenza can be difficult to

diagnose and knowing the prevalence of disease can substantially increase the utility of a

set of symptoms (21).

Early disease detection could also have benefits to the health system. Respiratory illness

can place a significant burden on hospitals. It is a leading cause of hospitalization in

Canada (22), accounts for 12-16% of emergency department visits in Canada (23,24), and

has been linked to emergency department overcrowding (25,26)(21). Canada is not alone

in this predicament. In fact, it has been recommended that the British National Health

Service (NHS) use its disease surveillance systems and its telephone nursing line, the

NHS Direct system, to anticipate sudden increases in hospital admissions in winter

months, of which respiratory infections are a major factor (27). Anticipating increased

visits could help hospitals better manage patient load and reduce wait times for

emergency services (28). Doing so might also help improve the efficiency of hospital

spending by reducing demand uncertainty (29).

1.2 Study Objectives

The objective of this thesis was to examine whether calls to a nursing helpline, Telehealth

Ontario, could be used to generate accurate forecasts for the number of emergency

department visits for respiratory illness for each of the 36 health regions (health units) in

4

Ontario. This tests the hypothesis that Telehealth Ontario calls are a leading indicator of

emergency department visits for respiratory illness. The accuracy of the forecasts

provides a measure of the degree of association between Telehealth Ontario calls and

emergency visits for different lead times.

This thesis compared the accuracy of three methods for generating emergency department

visit forecasts from the Telehealth calls time series. Nonlinearity in the temporal

relationship between calls and visits was considered. Two of the methods used were

capable of modeling nonlinearities while the third was not. The forecasting methods were

applied in such a way that they could be deployed as part of a real-time syndromic

surveillance system: the forecasts used only data that would be available to such a real-

time system in making predictions. By using this approach, it was hoped that the study

results would have practical significance and real-life application.

The three modeling techniques compared have never, to my knowledge, been previously

applied to health services demand forecasting or in the context of syndromic surveillance.

These methods were developed primarily for time series analysis and modeling in the

context of engineering. They represent a progression from the ARIMA (Autoregressive

Integrated Moving Average) models used previously in research on forecasting health

services demand. Two of them were novel non-linear techniques: Parallel Cascade

Identification (PCI) and Fast Orthogonal Search (FOS); and one of them a well-

established and widely-used linear technique: Numerical Methods for Subspace State

Space System Identification (abbreviated in the literature as N4SID or 4SID pronounced

―forsid‖ (30)).

5

Chapter 2 Literature Review and Study Rationale

2.1 Previous Research on the Telehealth Ontario Call-Emergency Department Visit Relationship for Respiratory Illness

Telehealth Ontario is a 24-hour, 7 days-a-week, free telephone helpline providing health

advice from trained Registered Nurses in English, French, and with translation support

available in other languages to callers across Ontario (31)(5). Telehealth receives an

average of 2700 calls each day that are captured in a central database. Advice offered

includes self-care, referral to physician, referral to the hospital emergency department

(ED), and immediate referral to 911 emergency services (1,4,5).

Previous research by the QPHI group has characterized Telehealth Ontario calls and

emergency department visits for respiratory complaints between mid-2004 and mid-2006.

At the provincial level, the majority of calls to Telehealth occurred during January and

March, on weekends, and in the late afternoon or evening (4). Compared to the hospital

emergency visit demographic, Telehealth calls for respiratory symptoms over-represent

children 0-4 years old and under-represent older age groups (5-17 years old, 18-65 years

old, and older than 65 years old). Specifically, ages 0-4 represent approximately 49% of

calls but only 24% of visits, while individuals older than 65 years represent

approximately only 3% of calls but 11% of visits (1). Intensities of both emergency

department use and Telehealth Ontario use is known to vary widely across Ontario based

on geographic location (3).

6

One previous approach to the assessment of data sources for real-time syndromic

surveillance used by several researchers involves cross-correlation analyses of the

candidate data source, after applying a syndromic mapping, with a gold-standard measure

of the outcome or disease being monitored, such as laboratory results (1,32,33). The

syndromic mapping classifies events in the candidate data source into syndrome

categories. The goal is to evaluate the strength of the correlation between the time series

of events assigned to a specific syndrome in the candidate data source and the gold-

standard measure of outcome, and to determine how far in advance this correlation is

significant. In this way, one can assess the candidate data source as a ―leading indicator‖

of disease and the usefulness of the syndromic mapping.

This type of analysis has been carried out for Telehealth Ontario for monitoring

respiratory illness by the QPHI group (1) based on methods developed by an earlier study

(32). The Telehealth calls time series was compared to the emergency department visits

time series obtained from the National Ambulatory Care Reporting System (NACRS)

database for respiratory complaints. Telehealth Ontario calls for respiratory syndrome

were identified according to set of call guidelines developed by QPHI (the syndromic

mapping). Emergency department visits for respiratory illness were identified by ICD-

10CA codes (International Classification of Disease Codes Revision 10 Canadian

Enhancement) for reason for visit. A number of steps to remove the effects of

confounding created by repeating patterns in the time series data, in particular weekends

which are associated both with increased call(4) and emergency visit (23) activity, were

required before assessing the cross-correlation of the time series. To do this, an ARIMA

(AutoRegressive Integrated Moving Average) model was fit to the time series to remove

7

autocorrelation, including that generated by weekends. Fitting an ARIMA model requires

stationary time series, which was achieved through differencing (34)(35). These steps

were performed for both time series. Cross-correlation was then performed on the

residuals and statistical significance was assessed for the different lags (1).

This study concluded that, at the provincial level, Telehealth Ontario calls for respiratory

complaints were significantly correlated with emergency department visits for respiratory

illness, with strong correlations at zero lag and weak correlations at lags of 15 days (1).

2.2 Time Series Forecasting

Forecasting health service demand can be done on a long- or short-term basis. Whereas

short-term forecasting can rely exclusively on time series analyses (34), longer-term

forecasting must account for other factors such as change in the age structure of the

population, the development of alternate forms of care, new procedures, and many other

factors that short-term forecasting assumes remain constant (36).

Generally, short-term forecasting methods can take three approaches: i) ―univariate time-

series forecasting‖ methods that rely on past values of a time series to predict its future

values, ii) ―causal models‖ that use the relationship between the variable to be forecast

and one or several independent variables to make the forecast, or iii) a combination of

both (34). When only a univariate approach is taken, it is assumed that past values of the

time series contain information that allow future values to be determined. This thesis is

8

concerned with short-term forecasting and employs a causal (as defined above) approach:

Telehealth Ontario calls were assumed to be a leading indicator of visits. The influence

of holidays/weekends on visits was also accounted for.

A popular approach to time series modeling and forecasting involves the use of ARIMA

(AutoRegressive Integrated Moving Average) models. ARIMA models can take either a

univariate or a causal form. If a causal form is chosen, an exogenous input is used and

the model is sometimes referred to as an ARMAX (AutoRegressive Moving Average

with eXogenous input(s)) model.

2.3 Previous Research on Health Service Demand Forecasting

A Medline search for studies in peer reviewed journals from July 2008 dating back to

1996 using subject headings ―Forecasting‖, ―Hospitalization/ or Patient Admission‖ and

―Health Services Needs and Demand‖ and a Google search (also for studies dating back

to 1996) of the world-wide web using the same search terms were performed to determine

what methods had been used previously to create short-term health services demand

forecasts. Only studies using time series methods to forecast health care contact were

included. Five such studies were identified. The study objectives relevant to forecasting

and the forecasting methods used are summarized in Table 1.

9

Table 1: Literature on Forecasting Health Services Demand (1996-2008)

Author, Date Study Objective Relevant to Forecasting Analytic Techniques Used

Abdel-Aal, 1998 (37) Forecast monthly patient volume of a primary health care clinic in

Saudi Arabia

Univariate time-series forecasting using ARIMA models

and ad-hoc extrapolation

Diaz, 2001(38) Forecast emergency admissions for organic disease, circulatory

disease, and respiratory disease in a Madrid hospital using

environmental variables

ARIMA models with environmental variables as exogenous

inputs

Jones, 2002 (28) Forecast daily bed occupancy and emergency admissions in an acute

hospital in UK

ARIMA with exogenous inputs and GARCH (Generalized

Autoregressive Conditional Heteroskedasticity) to forecast

volatility

Reis, 2003 (39) Generated forecasts for number of emergency department visits in

order to establish an expected number of visits that could be used in

statistical tests for outbreaks in syndromic surveillance

Trimmed mean seasonal model combined with univariate

ARIMA models

Upshur, 2005(40) Examined seasonality and predictability of hospital admissions for

various health outcomes in Ontario

Regression techniques using sinusoidal terms and spectral

analysis

10

Abdel-Aal et al.(37) fit a univariate ARIMA model to 108 months of monthly patient visit

volume data for a primary care clinic using univariate Box-Jenkins methods. This model

was used to forecast visits over the following 24 month period. The clinic served a

population of 13,000 and no particular age range or patient population details were

discussed. The visits data showed a very regular repeating pattern with increasing trend

in the monthly visits. Visits ranged from approximately 400 to 850 patients over the 11

year study period. The study found that the ARIMA models had a forecasting accuracy

with a mean absolute percentage error of 1.86% and a maximum absolute percentage

error of 4.23% over the last two years of data. Because the visit pattern was so regular,

this study also considered a simple ad-hoc extrapolation method for generating forecasts

(referred to as extrapolating the growth curve) which involved using values of past visits

multiplied by a factor determined using the ratio of past visits indicating anticipated

growth. This ad-hoc method produced more accurate forecasts with mean absolute

percentage error of 0.55% and a maximum absolute percentage error of 1.17% over the

last two years of data.

Diaz et al. (38) used an ARIMA model with exogenous inputs including levels four air

pollutants, air temperature, humidity, and day of week in order to establish a relationship

between air pollutants and daily hospital admissions for total organic-disease, circulatory

disease, and respiratory system disease for a single teaching hospital in Madrid over a

1004 day period. Specific details on the demographic characteristics of the patients

visiting the hospital examined in the study were not given, but 13% of the Madrid

population is over 65. While the authors did not explicitly attempt to forecast with the

model, they did suggest that the model might be used to ―detect variations in the number

11

of hospital admissions well in advance‖ and thereby ―ensure optimal management and

allocation of hospital health care resources‖. A mean error of 15% is reported for the

ability to accurately model daily hospital admissions, although it was not clear whether

this measurement was made over a separate validation data set or over the set used to fit

the model. The lack of precise description of the methods used and the models developed

in this study made it difficult to interpret the results.

Jones et al. (28) used 2182 days of hospital emergency admissions data in an attempt to

build time series models for forecast emergency admissions and bed occupancy in a 540

bed hospital in the Britain serving what the authors describe as ―an older than average

population‖. The study examined the relationship between the Public Health Laboratory

Service’s (PHLS) influenza surveillance program estimate of new influenza and

influenza-like illness cases and emergency admissions and bed occupancy. Both

admissions and bed occupancy were found to be correlated with these estimates up to two

weeks in advance. It also examined the relationship between outside temperature and bed

occupancy and admissions and found that temperature was related to current bed

occupancy but not to admissions. The authors used knowledge of these relationships to

attempt to build predictive models of bed occupancy and admissions. To build and test

models for bed occupancy and admissions (each treated separately), the available data

was divided into 10 segments and an ARIMA model was fit for each segment. The next

32 days was used to assess the model performance. ARIMA models for bed occupancy

incorporating exogenous inputs for temperature and the PHLS influenza surveillance

program were developed. Errors were calculated as percentages relative to the mean and

standard deviations of visits: the mean number (and standard deviation) of daily occupied

12

beds was 441.06 (standard deviation of 32.48) and the mean number of daily admissions

was 51.71 (standard deviation of 13.39). These models had root mean square (RMS) error

of 23 beds (5.2%) (standard deviation 4.2) compared with 15.1 (3.4%) (standard deviation

2.09) when no exogenous inputs were included. They noted that the forecasts were poor

during times of ―bed crisis‖. A simple moving average (MA) model was used to predict

admissions and tested in the same way. This model was found to have an RMS error of

12.6 (standard deviation of 2.5) or 24% relative to a mean of 51.71 admissions. The

authors note that using the mean level of admissions to forecast future admissions was

almost as good as the moving average model. This study also examined forecasting

volatility in admissions and bed occupancy. The authors suggest that future research

should consider nonlinearity as it may improve forecasting.

Reis et al. (39) attempted to find a systematic method for forecasting the expected number

of daily emergency department visits for respiratory complaints in order to be able to

reliably detect abnormal visit patterns for the purpose of surveillance. The forecasting

models used to generate the expected number of visits incorporated both a trimmed

seasonal model and an ARIMA model. The trimmed seasonal model generated the

expected number of visits using a sum of the overall mean, a mean for day of week, and a

mean for day of year. The authors fit an ARIMA model to the residuals of this trimmed

model and the actual time series. Combining these two models improved overall fit.

Models were constructed using 2775 days of data and validated over a period of 730 days.

The study found a mean absolute percentage error (MAPE) of 9.37% for prediction of

overall visits and an MAPE of 27.54% for respiratory visits. This study also investigated

the ability to detect outbreaks using a scheme that looked at the difference between the

13

expected number of visits forecast by the developed model and actual visits; however,

these results are not relevant to the current study.

Upshur et al. (40) developed a regression model including sinusoidal terms of monthly

hospital admissions for 52 of the most common admission diagnoses for all of Ontario for

the period from April 1988 to December 2001. The first 148 monthly observations for

each series were used to fit the models and the last 12 were used to assess the adequacy of

fit. The only measurement of fit provided by the authors is the number of samples in the

12 month validation set that fell outside the 95% confidence interval which was not

specified.

2.4 Gaps in Existing Knowledge

Based on the research reviewed above, the following gaps are noted:

1) Although the QPHI group has investigated the relationship between Telehealth calls

and emergency visits in Ontario at the provincial level, the fact that Telehealth Ontario

calls and emergency department visits each vary in intensity by region suggests the call-

visits relationship may also vary by geographical region. A preliminary evaluation of the

relationship between calls and visits at the health unit level was done by creating plots of

weekly calls and visits (3), but the relationship has not been quantitatively evaluated.

14

2) The cross-correlation analyses used in past studies to measure the association between

a data source for syndromic surveillance and the outcome it was intended to monitor

(32,33), including that performed for Telehealth Ontario calls and emergency department

visits for respiratory illness at the provincial level (1), ignore the possibility that there

may be important information in the calls time-series about visits in the form of a

nonlinear relationship.

3) To be useful, knowledge of the relationship between Telehealth Ontario calls and

emergency department visits must eventually lead to practical applications. However, to

date, studies of the Telehealth calls/emergency department visits relationship have not

addressed how information about the call-visits relationship might be applied to public

health and clinical practice.

4) While it has been suggested that respiratory illness has an impact on demand for

hospital services and that surveillance systems, including telephone help lines, might be

used to anticipate demand for these services, there have been few attempts to study how

this can be done. Of the literature reviewed, Jones et al. (28) was the best attempt at this

specific task. The forecasts obtained by this study for admissions and bed occupancy

were not as good as the authors had hoped, and they suggested that nonlinear

relationships might be investigated in order to improve results. The study was done at the

level of a single hospital and it is unknown whether better or worse results might have

been achieved if a larger number of hospitals had been included. The study was

performed in the UK and it is not clear how results might differ in Canada.

15

2.5 Study Rationale

2.5.1 Conceptual Framework

Figure 1 and Figure 2 illustrate a hypothetical framework for the temporal relationship

between Telehealth Ontario calls and Emergency Department visits for respiratory illness.

This relationship can be thought of at two levels: an individual level and a population

level.

Figure 1 presents a framework at the individual level. Individuals are infected with a

respiratory pathogen. The onset of symptoms occurs after some incubation period.

Symptoms cause individuals to initiate some sort of behaviour, in this case a call to

Telehealth Ontario, which precedes seeking care at the emergency department.

Figure 1: Hypothetical Framework Illustrating the Temporal Relationship between Telehealth

Ontario Calls and Emergency Department Visits at the Individual Level

Time

Exposure

and Infection

Onset of

Symptoms

Initiation of

BehaviourSeek Care

Call to

Telehealth

Ontario

Emergency

Department

Visit

Delay 1

Although the delay (labeled ―Delay 1‖ in Figure 1) between a call to Telehealth and a

visit to the emergency department may be short for a given individual (hours or a single

day), there may be a longer delay between when some individuals make calls and other

members of the population seek care (―Delay 2‖ in Figure 2). Telehealth Ontario calls for

16

respiratory complaints are primarily for younger individuals (1,4). There is evidence that

younger individuals start to use health services for respiratory illness sooner than older

individuals (41), meaning that the delay between the majority of calls may occur before

the majority of visits.

Figure 2: Hypothetical Framework Illustrating the Temporal Relationship between Telehealth

Ontario Calls and Emergency Department Visits at the Population Level

Time

Infection of

Younger

Individuals

Infection of

Older

Individuals

Telehealth

Ontario Calls

from Younger

Individuals

Emergency

Visits from

Older

Individuals

Delay 2

Research also indicates that health care contact for the 0-4 year old age group showed the

highest seasonal variability in rates (41). Since the majority of calls to Telehealth are

from this age group, this might mean that Telehealth calls have good signal-to-noise

properties, where ―signal‖ is defined as the difference in means between the time when

there is respiratory illness present to when there is not, and ―noise‖ is defined as the

standard deviation during the time there is no illness present (42).

2.5.2 Addressing Gaps in Knowledge

The objectives of this study address the knowledge gaps summarized in section 2.4.

17

1) This study examines the Telehealth call/emergency visit relationship at the health unit

level which has not been formally done. Because some health interventions may be

coordinated at the health unit level, health units are involved in monitoring infectious

disease, and there is geographic variability in the intensity of emergency department and

Telehealth Ontario use, it would be helpful to make the assessment of the relationship

between Telehealth Ontario calls and emergency department visits for respiratory illness

at the health unit level.

2) This study uses methods that are capable of capturing a nonlinear relationship between

calls and visits. Furthermore, because three methods of modeling the call-visits

relationship are compared, two of which are capable of modeling nonlinear relationships

and a third that is not, the results of the study may demonstrate the potential importance

of accounting for nonlinearity in these models. If methods capable of modeling

nonlinearity perform better than those that do not, the difference might be attributable to

important nonlinearity in the relationship captured in the time series models.

3) By measuring the ability of calls to forecast visits, this study examines a practical

application of the calls-visits relationship. Currently, there is no published research

investigating practical application of the known relationship between Telehealth Ontario

calls and emergency department visits for respiratory illness. Although the forecasting of

respiratory illness using surveillance information has been suggested in the literature, it

appears that few studies have been done to examine its feasibility. Furthermore, past

18

studies do not attempt to use nonlinear relationships to generate forecasts, but it has been

suggested that doing so might be of value (28).

4) Finally, the literature has recognized the need for more integration between the areas

of health services research and informatics in order to improve health care delivery (43).

This thesis attempts to bring new approaches to Epidemiology. It suggests a new

application for syndromic surveillance systems in forecasting health services demand.

19

Chapter 3 Study Design and Methods

3.1 Study Population, Setting, and Design

This study examined Telehealth Ontario calls and emergency department visits for

respiratory complaints for all of Ontario from June 1, 2004 to March 31, 2006 (669 days).

Time-series analyses of the relationship between calls and visits were carried out at the

health unit level for each of the 36 health units in Ontario. Forecasting models were

constructed using roughly half of the approximately two years of time series data, and

then validated on the remaining data. Individual Telehealth calls were not linked to

corresponding individual emergency department visits.

3.2 Data Sources and Ethics Approval

Hospital emergency department (ED) visits for respiratory complaints were obtained from

the Canadian Institute of Health Information (CIHI) National Ambulatory Care Reporting

System (NACRS) database using data from the fiscal years 2004-2005 and 2005-2006

(44,45). All institutions in Ontario providing hospital care are mandated by the Ontario

Ministry of Health and Long Term care to submit emergency data to CIHI on a yearly

basis (45).

Telehealth Ontario calls were obtained from Clinidata Inc. which was contracted by the

Ontario Ministry of Health and Long-Term Care to administer the Telehealth Ontario

nursing call line over the study period.

20

Ethics approval for the project was obtained from the Queen’s University Ethics Review

Board in accordance with the Tri-Council Policy Statement on the Ethical Conduct of

Research Involving Humans (refer to Appendix A for a copy of the ethics approval).

3.3 Definitions

Respiratory illness was defined as sickness caused by respiratory pathogens. Pathogens

responsible for the majority of respiratory illness screened for by laboratory tests in

Canada include respiratory syncytial virus (RSV), parainfluenza viruses, adenoviruses,

influenza A and influenza B (46). Specific definitions used in identifying emergency

visits and Telehealth Ontario calls are as follows:

Emergency Department Visits: Using a gold-standard of laboratory test results for

respiratory pathogens, a study by Marsden-Haug et. al. (42) developed a set of

International Classification of Disease version 9 (ICD-9) codes for use in syndromic

surveillance that were highly correlated with respiratory illness. These ICD-9 codes were

translated to ICD-10CA codes, the Canadian enhancement to the ICD-10 codes published

by the World Health Organization, using a conversion file and by matching definitions

(47)(48). Emergency department visits for respiratory illness were identified from the

NACRS database using this set of ICD10-CA codes. Both the ICD-9 codes developed by

Marsden-Haug et al. and the corresponding ICD10-CA codes are given in the third

column of Table 2. This set of ICD10-CA codes is similar to that used by van Dijk et al.

21

(1,3) in a previous study of the Telehealth Ontario call emergency department visit

relationship discussed in section 2.1.

22

Table 2: ICD-10CA Codes Used to Identify Emergency Visits for Respiratory Complaints from the NACRS Data Set

ICD9 Codes Developed by Marsden-Haug et al. Corresponding ICD10-CA Codes

ICD9 Code ICD9 Description ICD10-CA Code ICD10-CA Description

079.9 Unspecified viral and chlamydial infections B34.9 Viral infection, unspecified

382.9 Unspecified otitis media H66.9 Otitis media, unspecified

460 Acute nasopharyngitis [common cold] J00 Acute nasopharyngitis (common cold)

461.9 Acute sinusitis, unspecified J01.9 Acute sinusitis, unspecified

465.8 Acute upper respiratory infections of multiple or unspecified sites J06.8 Other acute upper respiratory infections of multiple sites

465.9

Acute upper respiratory infections of multiple or unspecified sites

J39.9 Disease of upper respiratory tract, unspecified

J06.9 Acute upper respiratory infection, unspecified

466.0

Acute bronchitis

J20.0 Acute bronchitis due to mycoplasma pneumoniae

J20.1 Acute bronchitis due to haemophilus influenzae

J20.2 Acute bronchitis due to streptococcus

J20.3 Acute bronchitis due to coxsackievirus

J20.4 Acute bronchitis due to parainfluenza virus

J20.5 Acute bronchitis due to respiratory syncytial virus

J20.6 Acute bronchitis due to rhinovirus

J20.7 Acute bronchitis due to echovirus

J20.8 Acute bronchitis due to other specified organisms

J20.9 Acute bronchitis, unspecified

486

Pneumonia, organism unspecified

J18.8 Other pneumonia, organism unspecified

J18.9 Pneumonia, unspecified

487.0

Influenza w/ pneumonia

J10.0 Influenza with pneumonia, influenza virus identified

J11.0 Influenza with pneumonia, virus not identified

487.1

Influenza w/ other respiratory manifestations

J10.1 Influenza with other respiratory manifestations, influenza virus identified

J11.1 Influenza with other respiratory manifestations,virus not identified

487.8

Influenza w/ other manifestations

J10.8 Influenza with other manifestations, influenza virus identified

J11.8 Influenza with other manifestations, virus not identified

490 Bronchitis, not specified as acute or chronic J40 Bronchitis, not specified as acute or chronic

780.6

Fever (general symptoms, pyrexia of unknown origin)

R50.0 Fever with chills

R50.1 Persistent fever

23

Telehealth Ontario Calls: The reason for each call to Telehealth Ontario is mapped to

one of 486 clinical guidelines (1,4,5). The study by van Dijk et. al. (1) discussed in

section 2.1 identified a set of Telehealth Ontario call guidelines (i.e. a syndromic mapping

for Telehealth calls) that resulted in a strong correlation at the provincial level between

the Telehealth calls and the emergency visits for respiratory complaints. This set of

guidelines (Table 3) was used to identify calls due to respiratory complaints from the

Telehealth Ontario call data set.

24

Table 3: Guidelines Used to Identify Calls for Respiratory Complaints from the Telehealth Ontario

Data Set

Upper Respiratory Syndrome

Colds (adult after hours)

Colds (pediatric after hours)

Congestion – guideline selection(pediatric after hours)

Croup (pediatric after hours)

Ear, congestion (adult after hours)

Ear, congestion (pediatric after hours)

Ear, discharge (adult after hours)

Ear, discharge (pediatric after hours)

Earache (adult after hours)

Earache (pediatric after hours)

Hoarseness (adult after hours)

Hoarseness (pediatric after hours)

Respiratory multiple symptoms – guideline selection (adult after hours)

Respiratory multiple symptoms – guideline selection (pediatric after hours)

Sinus pain and congestion (adult after hours)

Sinus pain or congestion (pediatric after hours)

Sore throat (adult after hours)

Sore throat (pediatric after hours)

Lower Respiratory Syndrome

Cough, acute non-productive (adult after hours)

Cough, acute productive (adult after hours)

Cough, chronic (adult after hours)

Cough (pediatric after hours)

Coughing up blood (adult after hours)

Wheezing, other than asthma (pediatric after hours)

25

3.4 Emergency Department Visits: the NACRS Database

Fields available from the NACRS database used to identify and characterize emergency

department visits for respiratory complaints in this study are given in Table 4. ICD10-CA

codes were used to identify visits for respiratory complaints, date information was used to

generate daily visit counts, age information was used for comparison with the age

demographics of callers, and forward sortation area (FSA) of the postal code of the

patient was used to allocate the calls to the various geographic regions in Ontario.

Table 4: NACRS Fields Used in Analysis of Emergency Department Visits

Variable Database Fields Used Data Quality

Time Registration date No Information

Reason for seeking care ICD-10-CA 3 digits for main

problem

Re-abstraction Study/original

agreement 78.5% exact ICD-10-CA

agreement; 88.8% category level (49)

Geographic location of

patient

Patient’s postal code forward

sortation area (first three

characters of postal code)

1.71% unknown postal code; 0.79%

invalid postal codes(45)

Demographic

information

Patient age in years 0.02% unknown/partial birth date (45)

3.4.1 Coverage and Data Quality

All institutions in Ontario are required to provide their records on emergency department

visits to CIHI. In the 2004-2005 and 2005-2006 fiscal years, 177 and 180 institutions in

Ontario reported their emergency visits to CIHI, respectively (44,45). The missing record

rate was estimated to be less than 0.10% and the duplicate record rate less than 0.20%

26

(44,45). Data quality in NACRS is verified using various cross-checking and data

validation algorithms, which are incorporated in the data collection software (44).

Quality of the emergency department visits data has been assessed using re-abstraction

studies (49). The last column of Table 4 provides information on the data quality related

to the NACRS fields used in the current analysis.

3.4.2 Inclusion/Exclusion

The inclusion/exclusion criteria employed for emergency visits is summarized in Figure

3. All visits with ICD-10CA codes for main reason for visit given in Table 2 and visit

dates between June 1, 2004 and March 31, 2006 were included. Calls from patients with

missing and out-of-province forward sortation area (FSA) information were excluded

from the analysis. This resulted in 555,171 emergency department visits for respiratory

complaints being included for all of Ontario over the study period.

Figure 3: Inclusion/Exclusion of Hospital Emergency Department Visits for Respiratory Complaints

NACRS Fiscal Year 2004/2005,

2005/2006 ED Visits for

ICD-10CA Chapters 1, 6, 10

N=1,473,276

N=1,354,181

Select Visits Between 6/1/2004

and 3/31/2006

N=570,014

Select Visits for ICD-10CA Codes for

Respiratory Illness given in Table 1

Visits used in Analyses N=555,171

Exclude Records with Missing or

Out-of-Province FSAN=14,843

27

3.5 Telehealth Ontario Calls

Information used from the Telehealth Ontario calls database is provided in Table 5.

Table 5: Telehealth Ontario Call Database Fields Used in Analysis

Variable Database Field Used

Time Call date

Geographic location of caller Patient forward sortation area (first three characters of postal

code)

Reason for call Clinical call guideline assigned

Demographic information Patient age in years

3.5.1 Coverage and Data Quality

The Telehealth service is freely available to anyone in Ontario and does not require the

caller to provide health insurance information (50). Information on the quality of the data

in the Telehealth Ontario database was not available.

3.5.2 Inclusion/Exclusion

The inclusion/exclusion criteria for Telehealth Ontario calls are summarized in Figure 4.

All calls with assigned call guidelines given in Table 3 occurring between June 1, 2004

28

and March 31, 2006 were included. Calls with missing or invalid patient forward

sortation areas were excluded.

Figure 4: Inclusion/Exclusion of Telehealth Ontario Calls for Respiratory Complaints

Telehealth Ontario Calls between

6/1/2004 and 3/31/2006

N=1,799,862

N=194,331

Select Calls with Guidelines

Related to Respiratory

Complaints given in Table 2

N=184,129

Exclude Records with Missing or

Invalid FSA

N=10,202

3.6 Confounders

Emergency department visits for all causes are known to vary across time, with a greater

number of visits on holidays and weekends (23). As discussed in section 2.1, Telehealth

calls are also known to vary in a weekly pattern, with a higher number on weekends (4).

Because increased visits are associated with weekends and holidays and increased

Telehealth call volume is associated with weekends and holidays, holidays and weekends

may be a potential source of confounding of the relationship between Telehealth calls and

ED visits. To attempt to control for this, weekends and all Canadian statutory holidays

were included in the analysis. When a statutory holiday fell on a weekend, the next

closest regular weekday was assigned as a holiday as most jurisdictions would follow a

similar holiday practice.

29

3.7 Geographic Grouping of Telehealth Calls and Emergency Visits

The analysis performed in this study was carried out by geographical regions that

correspond approximately to each of the 36 public health unit regions in Ontario. Both

the NACRS and Telehealth Ontario data sets contain patient/caller postal code forward

sortation area (FSA) information, which is the first three digits of the individual’s postal

code (to protect the identities of individuals, complete postal code information was not

available).

An exact mapping between FSA and health unit is not possible as the region

corresponding to a single FSA may overlap with the regions corresponding to two or

more health units. To address this issue, a mapping between FSA and approximate public

health unit region was created as follows:

In Canada, census geography is broken down into census subdivisions, dissemination

areas, and blocks (from least to most granular) (51)(52). Statistics Canada provides a

correspondence file between dissemination areas and health region boundaries (53)(54)

and a postal code conversion file which includes all the dissemination areas for a given

FSA (51). A dissemination area only falls into one health region; however, each

dissemination area can be linked to more than one FSA. For each FSA in Ontario, the

associated dissemination areas in the postal code conversion file were used to match the

FSA to one or more health regions. The correspondence file also included the 2001

census population for the dissemination area. For a given FSA, the population was

30

summed for each unique health region sharing a geographic area with that FSA. The

public health unit with the largest census population was assigned to that FSA. In this

way, a one-to-one mapping between FSA and PHU was established. As the developed

FSA groupings do not represent exact PHU areas, the approximation should be

noted in all the results that follow.

Time series of calls and visits for respiratory complaints for each approximate PHU

region were obtained using this FSA mapping and the date information in the call and

visit data.

3.8 Analytic Techniques for Establishing the Relationship between Calls and Visits

3.8.1 Background

This thesis attempted to establish a useful relationship between the time series of

Telehealth Ontario calls for respiratory complaints and the time series of emergency

department visits time series for respiratory illness, for each health unit in Ontario.

Specifically, it was desired to know if the number of calls from the current day and those

from several days in the past were predictive of future emergency department visits. For

example, one may want to know how well the calls for the past 10 days predicted the

number of visits 3 days in advance. One could also ask the same question for 4, 5, 6 or

more days in advance. Therefore, this thesis examined multiple associations between

calls and visits, one association for each day in advance. This can be thought of an

analogous but different and more sophisticated approach to looking at the correlation

31

between time series at multiple lags as was done by van Dijk et al., discussed in section

2.1.

Characterization of this association can be thought of as a type of dynamic regression

problem (34). A mathematical model to quantify the relationship between a variable and

a set of predictor variables can be built from a first-principles understanding of the

relationship between variables, or by taking an empirical approach using measurement

data. In this latter approach, the model is sometimes referred to as a ―black-box‖ model

as we ignore how the physical process works (i.e. ignore the first principles approach) in

building the model and care only that the relationship between variables is accurately

described. In the engineering literature the term ―system identification‖ is used to refer to

the process of building a mathematical model describing the dynamic relationship

between two or more time series from observed data. The ―system‖ is a physical or

hypothetical process that transforms one or more input time series (independent variables)

into one or more output time series (dependent variables). System identification seeks to

build a mathematical description of how the system transforms its inputs into outputs.

There have been many methods developed to do this (55)(56). All three analytic time

series techniques applied in this thesis have been developed and used for the development

of black-box models of systems. A brief introduction to these methods is presented here.

Figure 5 illustrates the relationship between Telehealth Ontario calls and emergency

department visits framed as a system identification problem. This follows a standard

representation (57). The emergency department visit time series, y(n) (the system output

or dependent variable), is assumed to depend on past and present Telehealth Ontario calls,

32

u1(n), and holidays/weekends, u2(n) (the system inputs or independent variables). The

variable n indicates the time in days. All three of these time series are subject to

measurement error (for example miscoded reason for calls or visits). The error in calls is

represented by the noise series w1(n), w2(n) represents the error in holiday/weekends, and

vy(n) represents the error in emergency visits.

Figure 5: The Dynamic Relationship between Calls and Visits Time Series Framed as a System

Identification Problem

+ Unknown

Deterministic,

Time-Invariant

Relationship

Telehealth

Calls, u1(n)

Call Noise,

w1(n)

+

Process

Noise, wp(n)

+

Output

Measurement

Noise vy(n)

Emergency Visits,

y(n)

Measured

Emergency Visits,

ym(n)+Holidays,

u2(n)

Holiday

Noise, w2(n)

Shaping

Filter

The process noise, wp(n), and the associated shaping filter account for the fact that our

description of the relationship between time series may be imperfect because of missing

explanatory time series, a poorly chosen model structure, or a poor parameterization of

the chosen model structure (these are sometimes referred to as disturbances) (35,56).

33

Some model structures attempt to include a description of these disturbances effects.

Incorporation of these can improve results when the models are used to generate forecasts

(56). Other model structures do not include a description of the process noise: ignoring it

may have negligible impact on model performance if the inputs we have chosen to model,

u1(n) and u2(n), describe enough of the variation in the output (i.e. high signal-to-noise

ratio) (56).

3.8.2 Numerical Algorithms for Subspace State Space System Identification

ARMA (AutoRegressive Moving Average, univariate), ARX (AutoRegressive with

eXogenous input), and ARMAX (AutoRegressive Moving Average with eXogenous

input) time-series models are all variants of the ARIMA class of models. This class of

models has been investigated by previous studies as methods to create forecasts for health

services demand (Table 1). The ARIMA class of models have been shown to be special

cases of the state-space representation for a dynamic deterministic-stochastic process

(34,35,56,58). The state space representation uses a set of auxiliary variables, called state

variables, to describe the relationship between the predictor time series and the dependent

time series. The state variables allow the relationship to be expressed as a set of first-

order difference equations; the number of state variables in the model is referred to as the

order of the system (56).

34

The innovations form of the state-space representation is given by (56):

)()()()1( nKenBunAxnx

Equation 3-1

)()()()( nenDunCxny

Equation 3-2

where

x(n) is a vector of state variables

u(n) is a vector of system inputs

y(n) is a vector of system outputs

e(n) is the innovation vector

A, B, C, D, K are matrices of appropriate dimensions

Two general approaches for the identification of A, B, C, D, K using input-output data are

the Prediction Error Method (PEM) and Numerical Algorithms for Subspace State Space

System Identification (N4SID)(30,59). Briefly, N4SID methods construct an estimate of

a sequence of state vectors, x(n), of the state-space model given in Equation 3-1 and

Equation 3-2 from the observed input-output data. To do this, an approach similar to a

principal components analysis is taken (30). Once this state sequence is obtained, it is

possible to use regression to obtain a least-squares estimate of the system matrices C, D

and then A, B and K (56). The PEM method takes a fundamentally different approach

and suffers from a number of problems which are not encountered using the N4SID class

methods; therefore, the N4SID approach is preferable under many circumstances (59,60).

35

The N4SID algorithms have been implemented in the MATLAB System Identification

Toolbox (56). This software was used to fit linear state space models describing the

relationship between calls and visits. Further description of how this was done is given in

section 3.8.5.

Since the ARIMA, and in particular the ARMAX, model structures are a special case of

the state-space representation, the state space representation of the time series model was

included in the analysis to serve as a baseline with which to compare the novel non-linear

approaches. In this way, the advantage of accounting for possible non-linearity in the

calls to visits relationship could be assessed.

3.8.3 Fast Orthogonal Search

Fast Orthogonal Search (FOS) is a novel method for developing a non-linear difference

equation or other model of unknown structure (in contrast to the linear difference

equation described by the state-space method) of a dynamic system proposed by

Korenberg (61). The nonlinear difference equation model describes the relationship

between the current number of emergency department visits, y[n], current and past

number of Telehealth Ontario calls, u1[n], u1[n-1], u1[n-2], etc…, and current and past

values of an indicator variable for holidays/weekends, u2[n], u2[n-1], u2[n-2], etc… , and

past values of the number of emergency visits, y[n-1], y[n-2], etc… and is of the form:

36

M

m

mm nenpany0

][][][

Equation 3-3

where

n is the time index, n=0, 1, 2, …

p0=1

][]...[][]...[][]...[][210 2121111 jjjm qnuqnulnulnumnymnynp is the

general form of each term in the difference equation for m>0

am is a scalar coefficient

e[n] represents the error

and

j0 is the number of y factors, j0≥0

j1 is the number of u1 factors j1≥0

j2 is the number of u2 factors j2≥0

mk is the lag in kth

y factor, mk≥1

lk is the lag in the kth u1 factor, lk≥0

qk is the lag in the kth u2 factor, qk≥0

FOS searches a set of possible candidate terms, {cm[n]}, for the most significant terms

that minimize the mean square error over a training data set. Each of the candidate terms

in the set {cm[n]} has the general form shown above for pm.

37

Briefly, the method works by adding terms to the difference equation model from a set of

candidate terms, {cm[n]}, one at a time. Terms are selected based on a measure of the

reduction in the mean square error (MSE) of the model fit over the training data after

having added that particular term to the model. The candidate offering the largest

reduction in MSE is chosen and the process is repeated until the MSE has been reduced to

a predetermined level or a certain number of terms have been accepted into the model.

The reduction in MSE by adding a given candidate term to the model can be simply stated

in terms of a set of orthogonal basis functions {wm[n]} spanning the set {pm[n]} of all

terms in the model plus the candidate under consideration. One key to FOS is how the

measure of the reduction in MSE is calculated for each candidate term. Because the

number of potential candidate terms can be extremely large, calculating the MSE

reduction for each term could take a great deal of time as finding {wm[n]} is

computationally intensive. With the FOS algorithm, the set {wm[n]} are never actually

calculated, yet a measure of the reduction in MSE is still obtained, leading to faster

identification of the candidate which reduces the MSE by the largest amount (hence the

―Fast‖ in Fast Orthogonal Search) (62). To choose M terms for the model out of C

candidates using N values of time, FOS requires on the order of MNC multiplications.

FOS is related to an algorithm by Desrochers (63) for approximating static nonlinear

models where, amongst various differences, the computational and memory requirements

are proportional to the square of the number of candidates, whereas in FOS they depend

linearly on the number.

38

3.8.4 Parallel Cascade Identification

Parallel Cascade Identification (PCI) is a novel method for developing a model of a non-

linear time-invariant system and has been applied successfully in a wide-range of system

identification problems (64-70).

Consider a discrete-time (i.e. data are sampled at evenly spaced instants in time), time-

invariant (i.e. the relationship between time series does not change over time), causal (i.e.

output only depends on present and past values of the input(s)), finite-memory system

(i.e. output only depends on inputs up to a finite time in the past). Suppose that its output

is a continuous mapping of its input, in that ―small‖ changes in the input result in ―small‖

changes in the output. Then it follows from the Stone-Weierstrass theorem that such a

system can, over a uniformly bounded set of input signals, be approximated to an

arbitrary degree of accuracy by a discrete-time, finite-memory Volterra series of

sufficient, but finite, order. Korenberg further showed that any discrete-time, finite-

memory, finite-order Volterra series can be represented with an arbitrary degree of

accuracy by a parallel array of ―LN cascades‖. Each ―LN cascade‖ is a system consisting

of a dynamic linear element (L), described by a finite duration impulse response,

followed by a static non-linearity (N). PCI is a method of identifying these cascades using

observed input-output data from the system(64). To summarize, PCI has been

theoretically proven capable of modeling a class of non-linear time series associations

(more specifically any system having a Wiener series expansion).

39

A simplistic explanation of how PCI works for a single-input single-output system is as

follows (64,69,70):

In this explanation we note a discrepancy between the language used in signal processing

and that used in biostatistics. In this section ―cross-correlation‖ refers actually to the

cross-covariance familiar to statisticians and biostatisticians. Starting with the first

cascade, identify the impulse response of the dynamic linear element (L) in the cascade

using the first-order cross-correlation of the system input with the system output or a slice

of a second- or higher-order cross-correlation, determined at random, of the system input

with the system output. In the case of the latter, addition or subtraction (also determined

at random) of weighted delta functions is required for diagonal elements of the cross-

correlations, in order to ensure convergence. Calculate the output of the linear element

and fit a polynomial that minimizes the mean square error of the residual, where the

residual for the first cascade is the difference between the polynomial’s output and the

output of the system to be identified. Determine the reduction in mean-square error in the

residual as a result of adding this cascade (which is equal to the mean-square of the

cascade output); reject the cascade if this reduction is not above a certain threshold (to

avoid fitting noise). Subsequent cascades are added in a similar fashion using the residual

between the system output and the output of the model consisting of all the cascades

identified to date.

The above algorithm is easily modified to accommodate multiple inputs, as is done in this

thesis, by randomly selecting one of the inputs or cross-products of the inputs when

computing the first- or higher-order cross-correlation (64,70).

40

There are no special requirements for the input-output data required to fit (train) a PCI

model as there is with some other methods of non-linear system identification. The only

requirement is that the training data should be sufficiently ―rich‖. Practically, ―rich‖

means that it should cover the range of inputs and outputs expected when the model is

applied (69,70). Since respiratory illness shows a yearly pattern that peaks during

influenza season (46), in terms of the current study, this means that the training set cover

at least one year of data. For this reason approximately half of the two years of study data

was used to train the models and the other half was used to validate (assess the fit) of the

models.

3.8.5 Model Implementation

There were two inputs (independent variables) to the models created: 1) an indicator

variable coded 1 for weekends/statutory holidays and 0 otherwise, u2, and 2) the number

of Telehealth Ontario calls, u1, for each 24 hour period. There was one output (dependent

variable): the number of emergency department visits, y, for each 24 hour period.

Process noise is ignored when constructing a state space model for the system for two

reasons. First, we do not have access to the error for past predictions due to the fact that

NACRS visit data is not available until the end of the fiscal year (44,45), therefore we

cannot use the sequence of past errors to improve future forecasts. Second, the Parallel

Cascade Identification and Fast Orthogonal Search algorithms are not implemented to

41

include these effects, although it is possible to do so, and so to make them comparable

with the Subspace Identification Algorithm, the latter ignores fitting a disturbance model.

Forecasting models were constructed for each of the days ahead to be predicted. This

included a zero-day-ahead forecaster that produced a forecast of the number of visits for

the current day (i.e. at the end of that day, assuming all calls for that day had been

received). A total of 15 models were created to produce such forecasts up to 14 days in

advance. Note that a 14-day-ahead predictor could have been used to produce an estimate

for all of these cases (i.e. 0, 1, 2, 3, 4, …, 14 days in advance), but as will be shown later,

in general, a n-day-ahead predictor was more accurate than an n+k-day-ahead predictor,

where k>0.

To fit (train) a model using the Parallel Cascade Identification (PCI), Fast Orthogonal

Search (FOS), or N4SID (SS) algorithms to create an k-day ahead predictor, visits were

advanced in time (i.e. left-shifted) by k-days relative to the calls. The weekend/holiday

indicator variable input was also advanced in time by the same k-days. This is reasonable

to do as upcoming weekends/holidays are known in advance. Specifically, a k-day ahead

predictor described the mathematical relationship between the following input and output

time series:

42

Input (predictor) time series:

Telehealth Ontario calls time series: u1(n)

Holiday/weekend indicator variable time series: u2(n+k)

Output (dependent) time series:

Emergency department visits time series: y(n+k)

where n=0, 1, … could take on any time index over the training data.

When generating the forecasted values in this thesis, it was NOT assumed that the actual

past emergency department visits are available to the forecasting model. This is realistic

if the algorithms were to be applied in a real-time, prospective manner as hospitals in

Ontario are only required to submit the information used to generate the NACRS database

to CIHI at the end of the fiscal year (44,45). The specific implications of this are as

follows:

1) When making forecasts in the case of Fast Orthogonal Search, it may be necessary to

use past values of the system output (emergency department visits) to make future

forecasts of visits; this can be seen from Equation 3-3 which includes the possibility

of past values of the output in the difference equation. In this case, the forecasted values

of the past output, rather than the actual values of that output were used.

2) The state space model in Equation 3-1 allows for a disturbance model by specifying

the K matrix which uses values of the error to compute the next state vector. Since the

error is not available to the model, the term containing K in Equation 3-1 is ignored.

43

All models were programmed in the MATLAB software package (71). The state-space

model was identified using the Numerical Methods for Subspace State Space System

Identification (N4SID) algorithm implemented in the MATLAB system identification

toolbox (71). The Parallel Cascade Identification algorithm and Fast Orthogonal Search

algorithm were implemented by the author. Both implementations were validated using

test data generated from a known non-linear difference equation prior to using them to

build models in this thesis.

A trial-and-error process was used to determine appropriate model structure for the three

types of models. Specifically, this included defining structural choices given in Table 6.

Table 6: Structure Choices Required for each Type of Prediction Model

Model Type Structure Choices

State Space Number of state variables (system order)

Difference Equation Determined by Fast

Orthogonal Search

Total number of terms in the difference equation

Maximum number of u1, u2, and y factors in each term

in the difference equation

Maximum lag for each u1, u2, and y factor

Parallel Cascade Number of cascades

Memory length of the linear elements

Order of the static nonlinearity in each cascade

44

3.8.5.1 State Space Model using Subspace Identification

A state-space model with 10 state variables was used for all health units. Preliminary

investigation indicated that models of order 8 or 9 allowed good fits to be achieved for

most health units when the algorithm was allowed to choose automatically. This

dimension of the state space model was chosen to ensure rough comparability with other

techniques: the number of state variables (also referred to as the system order) is related

to the number of lags in the difference equation representation of a state space model

(56). The Parallel Cascade model allowed lags from 0 to 9 in the inputs (i.e. the linear

element in each cascade had a memory length of 10) and the difference equation

generated by FOS allowed for lags up to 10 in any factor in any term (refer to

implementation details of FOS and PCI implementations below). The N4SID algorithm

properties were set to optimize for prediction and stability (56); without specifying these

options, many of the models found were unstable (i.e. resulted in extremely large/small

forecasts and/or large oscillations in the forecasts).

3.8.5.2 Parallel Cascade Model Using Parallel Cascade Identification

The Parallel Cascade Identification algorithm was implemented as described by

Korenberg (64). This section describes the model structure chosen and modifications to

the algorithm.

The Parallel Cascade models incorporated a maximum of three cascades (it was observed

that the mean square error did not decline much after three cascades had been added to

45

the model over the training data). The linear element in each cascade had a memory

length of 10 samples (i.e. calls from 0 to 9 days in the past were used by the model). A

maximum second-order cross-correlation was used in identifying the impulse response of

the linear element. The static non-linearity was limited to only a second-order

polynomial. Limiting the order of the polynomial prevented over-fitting (loss of

generalizability of the model) on the training data which could result in large deviations

in the forecasts over the validation data.

An adaptive approach was taken to the cutoff value used in determining whether a given

cascade was accepted in the model in an effort to only accept the best cascades into the

model. Specifically, the suggested cutoff given by Korenberg (64):

46

)(1

4)( 22

1 nyRT

nz ii

Equation 3-4

where:

zi+1(n) is the output of the i+1 cascade to be added

yi(n) is the residual after adding the ith cascade to the model

T is the length of the training time series

R is the maximum lag used by the linear element

the over-bar represents the time average from n = R to n = T

was modified to:

)(1

)( 22

1 nyRT

nz ii

Equation 3-5

where λ was adjusted from an initial value of 40, and then decreased in an exponential

fashion (by dividing by a factor of 1.5) each time more than 100 cascades were rejected.

3.8.5.3 Difference Equation Model Using Fast Orthogonal Search

The Fast Orthogonal Search algorithm was implemented as described by Korenberg (61).

Ten terms were allowed in the difference equation model generated by the Fast

Orthogonal Search algorithm. Each term, other than the constant term, contained factors

with lags of up to 10 days.

47

Trial and error indicated that the model generalized better when a number of linear terms

were allowed in the model first before adding terms containing cross-products. Linear

terms are terms which have only one factor; in terms of pm in Equation 3-3 this means

only one of j0, j1, j2 is non-zero. Therefore the model included a constant term, seven

linear terms, and two cross-product terms. The two non-linear or cross-product terms had

up to one factor in the call input, one factor related to the holiday input, and at most one

output factor. This approach to forcing certain terms into the model before others was

similar to that used by Minz and Korenberg (72), but did not involve pre-screening of the

terms. Cross-product terms were not limited to cross-products of the linear terms

previously accepted into the model.

3.9 Measures

For each health unit, the time series was divided into a training dataset (45% of the data

or 301 days) and a validation dataset (55% of the data or 368 days). Refer to section

3.8.4 for justification of this split in the data. The training dataset was used to fit the

models. Each model was assessed using the measures described below over the

validation dataset. Models producing forecasts of the number of daily visits up to 14 days

in advance were created. A maximum 14-day lead was chosen because previous work

indicated a significant correlation in the calls/visits time series of up to 15 days in

advance (1). A period of time equal to the maximum lag used in the models (10 days)

48

was excluded at the beginning of the validation datasets to allow the models to settle

before performance was assessed.

3.9.1.1 Qualitative Assessment of Model Fit

Model fit was assessed qualitatively by visually comparing plots of the forecasted time

series of hospital emergency visits and the actual time series of visits. Plots can provide

more information than statistical summary measures of error. For example, summary

measures such as average error can fail to capture short-term but important deviations of

the predictions from the actual time series, and also fail to adequately characterize the

temporal variation in the error. Plots of the error between the predicted and actual

number of visits were also generated.

Plots for both the daily number of visits and weekly aggregate number of visits were

generated. Plots for daily number of visits can be generated for any one of the k-day

ahead predictors. Plots of the weekly aggregate number of visits can be generated for one

week ahead (the next 1 to 7 days) or two weeks ahead (the next 8-14 days). The one

week ahead aggregate number of visits is generated by summing the 1-day, 2-day, 3-day,

4-day, 5-day, 6-day, and 7-day ahead (lead) predictions. A concrete example of this is

illustrated in Figure 6 assuming it is Monday and all calls have been received for the day

(note that the start of the week was arbitrary and the choice of Tuesday is only for

illustrative purposes). The two week ahead aggregate number of visits is generated in a

similar manner by summing 8-day through 14-day ahead predictions. Plots of the weekly

49

aggregate number of visits remove the cyclical weekly pattern of visits which can act to

visually obscure trends. Aggregation keeps the plots more interpretable than if filtering

methods such as a moving average were applied. Because they remove the cyclical

weekly pattern, this approach controls for the possible confounding effect of weekends,

allowing the ability of Telehealth calls to predict emergency department visits to be

assessed (rather than assessing the ability of Telehealth AND weekends/holidays to

predict visits). Note that there will still be some residual confounding as aggregating on a

weekly basis does not control for the effect of holidays.

Figure 6: Prediction of Aggregate Hospital Visits over a period of 1-7 Days in the Future (1 Window

Ahead) and a period of 8-14 Days in the Future (2 Windows Ahead)

Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu

1 Week Ahead 2 Week Ahead

Current Date

1-D

ay L

ea

d

Pre

dic

tor

2-D

ay L

ea

d

Pre

dic

tor

3-D

ay L

ea

d

Pre

dic

tor

4-D

ay L

ea

d

Pre

dic

tor

5-D

ay L

ea

d

Pre

dic

tor

6-D

ay L

ea

d

Pre

dic

tor

7-D

ay L

ea

d

Pre

dic

tor

8-D

ay L

ea

d

Pre

dic

tor

9-D

ay L

ea

d

Pre

dic

tor

10

-Da

y L

ea

d

Pre

dic

tor

11

-Da

y L

ea

d

Pre

dic

tor

12

-Da

y L

ea

d

Pre

dic

tor

13

-Da

y L

ea

d

Pre

dic

tor

14

-Da

y L

ea

d

Pre

dic

tor

Predictors

Used:

3.9.1.2 Quantitative Assessment of Model Fit

Quantitative assessment of the fit of the model over the validation set was made by

capturing the median, 25th percentile, and 75

th percentile of the error in the daily and

weekly aggregate number of emergency department visits for each of the prediction

methods over the validation data.

Two summary measures of the error were used. The percent mean square error (%MSE)

is defined as (73):

50

%100))()((

))()((%100

))()((

))()((%

22

2

2

2

nyny

nzny

nyny

nznyMSE

Equation 3-6

The percent mean absolute prediction (percentage) error (MAPE) is defined as (34):

%100)(

)()(

ny

nznyMAPE

Equation 3-7

where

the over-bar represents the time average

y(n) is the value of the actual time series at time n

z(n) is the value of the forecast (predicted) time series at time n

Justification for the choice of these measures follows.

The MAPE has been used as a standard measure of forecasting accuracy (34) and is an

intuitive measure of the size of the forecasting errors. It allows the results of this study to

be compared with past studies that have used this measure. However, it has two

drawbacks. First, Equation 3-7 is not defined when the actual number of visits is zero.

Zero visits occurred in some of the smaller health units: Perth District Health Unit,

Timiskaming Health Unit, Brant County Health Unit, and Elgin-St. Thomas Health Unit.

This measure cannot be used in these health units. A second drawback is that forecasting

errors of equal magnitude above and below the actual value give different percentage

errors, with over-predictions being penalized more than under-prediction. For example if

51

the actual value is 130 and the forecast value is 100 (a difference of 30—under-

prediction), the MAPE is 23%, but when the actual value is 100 and the forecast is 130

(also a difference of 30—over-prediction), the MAPE is 30% (74,75). Although

corrections for this have been proposed, there are still issues (76). For this reason the

comparison between forecasting methods was performed using %MSE, and MAPE is

presented as an alternative mainly for comparison with past and future studies.

%MSE is an alternative measurement of accuracy without this problem, but with the

drawback that it is less intuitive. %MSE is a relative measure of the variation in the

prediction error (sum of squares of the error) compared to the variance in the time series

we are trying to predict. A model that fits well will have an error variance much lower

than the variance in the actual time series to be predicted.

3.9.1.3 Ability to Predict Increases

While plots of the actual and forecasted time series, the error time series, summary error

statistics, and summary measures of fit (%MSE and MAPE) discussed in the previous

sections provide a way to compare models, using any of these to interpret the usefulness

of the models can still be difficult. A model might appear to fit well according to these

measures, but be of little practical value.

52

Usefulness of the predictions was assessed in terms of the ability to predict future

increases in emergency department visits. Since it is desirable to be able to forecast

hospital bed shortages or the presence of a disease outbreak, the ability to warn of

increases in the aggregate number of future visits over a given window of time is

important. Aggregation of visits over a window of several days was necessary because

the day-to-day volatility in the number of visits was high and it was desirable to capture

increases in longer term trend rather than in daily volatility.

3.9.1.3.1 Windowing Method

Visits were aggregated over four- or seven-day windows that did not overlap. Analyses

were carried out over the validation dataset. A window size of seven-days eliminates the

cyclical weekly pattern in visits and calls on weekends thereby providing control for the

confounding effect of weekends (as discussed in section 3.9.1.1). The four-day window

was chosen to assess the combined predictive ability of calls and holidays/weekends over

a shorter time frame as a comparison.

The ability to predict an increase in the aggregate number of visits over a seven-day

window (i.e. weekly aggregate) either 1-7 days in the future (referred to as a one-window-

ahead/one-week-ahead forecast), or 8-14 days in the future (referred to as a two-

windows-ahead/two-week-ahead forecast) was assessed. Increases were relative to a

baseline seven-day window, consisting of the immediately preceding seven-day period.

The aggregate number of visits predicted for one-window-ahead (1-7 days in the future)

53

consists of the sum of the forecasted visits using the 1-day, 2-day, 3-day, 4-day, 5-day, 6-

day and 7-day lead predictors. Note that it is possible to use only a seven-day ahead

predictor to generate forecasts for the 1-window-ahead aggregate forecast; however, as

will be shown later, the %MSE generally increases as the prediction lead increases,

meaning that the forecasted visits generated by the n-day ahead predictor more accurately

reflect the true visits than those generated by the n+k-day, where k>0, ahead predictor.

Therefore, we should use the predictor with the smallest lead possible if we wish to

generate the most accurate forecasts. Forecasts for the two-windows-ahead were

generated in a similar manner.

The aggregate number of visits over the baseline window was calculated using forecasted

visits because data from the NACRS database are not available immediately; hospitals are

only required to submit their data to CIHI for inclusion in the NACRS database by the

end of July for the preceding fiscal year (44). Although it might seem intuitive to use the

zero-lead predictor to do this because it is more accurate that an k-lead predictor, where

k>0, as will be shown later, in practice this can give poor results because the predicted

time series generated by the zero-lead predictors can be quite different from the 1-7 or 8-

14 day predictors (for example one sequence can be slightly offset from the other). For

this reason, the corresponding past 1-7 or 8-14 day predictors were used in creating the

baseline visits.

An analogous approach was taken for the four-day window, the only difference being that

visits were aggregated over four days instead of seven.

54

3.9.1.3.2 Method used to Flag Increases

Increases in the aggregate number of visits for the nth

-window-ahead over the baseline

window was calculated as follows:

100%

B

BWn

A

AAI

Equation 3-8

where

%I is the percentage increase for the nth

-window-ahead over baseline

AWn is the aggregate number of forecasted visits over the nth-window-ahead

AB is the aggregate number of predicted visits over the baseline window.

To clarify this scheme, consider Figure 7. Suppose it is currently Tuesday (not yet the

end of the day) and it is desired to predict whether there will be an important increase in

the number of visits over the following seven-day window (1 window ahead, or Tuesday

(today) through Monday, inclusive) relative to the baseline aggregate number of visits

over the previous seven-day window (baseline window). Predictions from the 1-day lead

predictor through to the 7-day lead predictor are summed to give AW1 (AWn with n=1) in

Equation 3-8. This sum from the previous period (i.e. that generated last Tuesday) is used

for AB in Equation 3-8. The percentage increase over baseline is calculated using

Equation 3-8 and compared to a threshold (this threshold is discussed below). If the

percentage increase is above the threshold, the window is flagged. The actual visits time

series is flagged in a similar manner except that AWn and AB in Equation 3-8 are

calculated using actual visits rather than forecasted values.

55

Figure 7: Ability to Predict Important Increases in Visits over a Seven-Day Window

Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu

1 Window Ahead 2 Windows Ahead

Current Date

1-D

ay L

ea

d

Pre

dic

tor

2-D

ay L

ea

d

Pre

dic

tor

3-D

ay L

ea

d

Pre

dic

tor

4-D

ay L

ea

d

Pre

dic

tor

5-D

ay L

ea

d

Pre

dic

tor

6-D

ay L

ea

d

Pre

dic

tor

7-D

ay L

ea

d

Pre

dic

tor

8-D

ay L

ea

d

Pre

dic

tor

9-D

ay L

ea

d

Pre

dic

tor

10

-Da

y L

ea

d

Pre

dic

tor

11

-Da

y L

ea

d

Pre

dic

tor

12

-Da

y L

ea

d

Pre

dic

tor

13

-Da

y L

ea

d

Pre

dic

tor

14

-Da

y L

ea

d

Pre

dic

tor

Predictors

Used:

Su Mo Tu We Th Fr Sa

Baseline Window

Estimate of Baseline Visits using Previous 1

Window Ahead or 2 Window Ahead Forecasts

Two different types of thresholds were specified: 1) a fixed threshold at a zero percent

increase, and 2) a threshold whose percentage increase cutoff depended on the number of

visits.

The purpose of using a fixed threshold at zero was to test whether or not the forecasts

could accurately predict the sequence of increases and decreases in actual visits.

As discussed above, for planning purposes it might be desirable to warn of a large

upcoming increase in the number of emergency visits rather than simply an increase. A

large change in the number of visits may also indicate disease outbreak. The threshold

defining a ―large percentage increase‖ was variable depending on the actual number of

visits over the baseline window. A variable threshold was necessary because for a small

number of visits, a few additional visits can represent a large percentage increase, but this

increase is not important. For example, if the baseline window had 10 visits and the

window ahead being considered had 12 visits this would be a 20% increase. If the

baseline window for the same health unit now had 100 visits and the window ahead being

considered had 120 visits, this also represents a 20% increase. While the former case is

probably not an important increase, the latter might be.

56

The threshold used for flagging an increase, T, as a function of vbaseline (expressed as a

percentage) was defined as follows:

imum

ref

baseline

alnoimumimumbaseline Tv

vTTTvT minminmaxmax ,)(max)(

Equation 3-9

where

otherwiseb

baifaba

,

,,max

vbaseline is the number of visits over the baseline window

Tnominal is a nominal threshold

vref is a reference number of baseline visits where the threshold is the nominal

threshold and was set to the average number of daily visits over the training data

times the window size (for example to give the average number of weekly visits in

the case of a seven-day window),

Tmaximum is the threshold if there were zero visits

Tmininum is the minimum threshold, regardless of the number of visits

Figure 8 illustrates this variable threshold as a function of the baseline number of visits.

57

Figure 8: Threshold used for Flagging an Important Increases in the Number of Emergency

Department Visits

T

vref

Tmaximum

Tminimum

Tnominal

vbaseline

For the window of 1-7 days in advance, Tnominal was set to 10, Tminimum was set to 10, and

Tmaximum was set to 30. For the window of 8-14 days in advance, the thresholds were set

higher (as we are trying to detect ―important‖ increases after a longer period of time—

with more time, we expect higher variation). In this case Tnominal was set to 15, vref was

again set to the average number of daily visits over the training data times the window

size (to give the average number of weekly visits in the case of a 7-day window size),

Tminimum was set to 10, and Tmaximum was set to 60.

For example, using the values for 1-7 days in advance, with an average weekly number of

visits of 100, the nominal threshold with 100 visits in the baseline window is a 10%

increase; with 50 visits in the baseline window, the threshold is a 20% increase; with 20

visits in the baseline window, the threshold is a 26% increase.

This procedure for flagging increases was repeated using the time series for the actual

visits employing the same threshold and window definition—the only difference was that

the actual values for the number of emergency visits was used in the calculations

described above instead of those generated by k-day-ahead predictors.

58

3.9.1.3.3 Assessment of Predictions

A two-by-two contingency table giving the number of true positives (TP), false positives

(FP), true negatives (TN), and false negatives (FN) (also referred to as a confusion

matrix) can be constructed to compare if the sequence of increases flagged by the

forecasting models agrees with the sequence of actual increases. Statistical tests can then

be applied to measure whether the predictions are better than those made by chance.

Measures of the predictive ability are useful in helping a user assess how much trust to

place in the prediction can be generated. These measures include sensitivity, specificity,

negative predictive value, and positive predictive value of predicted increases.

Assuming the windows are independent, the ability of the predictor to correctly flag

increases above chance can be assessed using Fisher’s exact test (since sometimes there

can be a small number of counts in one of the cells in the contingency table). Where the

predictions were found to be statistically better than chance, the degree of agreement

between actual and forecasted increases was measured using the Matthew’s Correlation

Coefficient (MCC):

))()()((

)()(

FNTNFPTNFNTPFPTP

FNFPTNTPMCC

Equation 3-10

Matthew’s Correlation Coefficient is actually a special case of the Pearson correlation

coefficient when values take on only 1 or 0 or other binary values (77,78).

59

The usefulness of better than chance predictions was assessed by calculating the

sensitivity (Sn), specificity (Sp), positive predictive value (PPV), and negative predictive

value (NPV). The exact 95% confidence intervals for these values were calculated.

Note that the calculated confidence intervals also assume that the windows are

independent.

FNTP

TPSn

Equation 3-11

TNFP

TNSp

Equation 3-12

FPTP

TPPPV

Equation 3-13

FNTN

TNNPV

Equation 3-14

It should be noted than in assessing whether or not the predictions are better than chance

across three methods and all 36 health units, we are performing the same hypothesis test

on a family of values. Therefore the problem of multiple comparisons arises, and we

expect some significant results just by chance. With a type I error rate, α, of α=0.05

allowed, the family-wise error rate (FWER), which is the probability of at least one

significant result, is given by (79):

60

kFWER )1(1

Equation 3-15

where

k is the number of independent hypothesis tests performed

α is the probability of a type I error in the predictions over the validation set for

any single combination of health unit, method, and windows ahead

Conservative adjustment of the p-values for multiple comparisons can be made by the

Bonferroni method, which involves multiplication of the Fisher’s exact test p-values by

the number of experiments, k.

kppadj

Equation 3-16

Note that the Bonferroni correction is a very conservative adjustment (79).

3.9.1.3.4 Comparison to Using Telehealth Alone

It has been shown that Telehealth calls appear to precede visits at the provincial level (1).

Therefore, Telehealth calls time series alone may predict future increases in visits. A

comparison between the method of flagging increases discussed above and a simple

method using Telehealth calls alone should be made to ensure that the models used to

generate the forecasts provide value. In other words, is it really necessary to go to all of

the trouble to use forecasted values produced by the models to predict increases or could

a simple predictor be built using only the raw Telehealth calls time series?

61

To answer this question, Telehealth calls were aggregated over a given window and again

over the window prior to that. The percentage change in calls over these two windows

was calculated using Equation 3-8 (with AWn representing calls for the given window,

and AB calls from the previous window) and this percentage increase was considered to

be a ―test value‖. Because Telehealth calls appear to represent a scaled version of the

number of visits for many health units (refer to Figure 9 for example), the same threshold

used for actual increases (described above) was not employed for flagging increases using

the ―test value‖. Instead, it was desired to know if using a threshold at any level yielded a

useful predictor. Answering this question can be accomplished by calculating the area

under the receiver operating characteristic (ROC) and associated confidence limits

(78,80,81). In other words, the problem is viewed as that of determining the usefulness of

a diagnostic variable as is done in clinical epidemiology.

As just noted, in addition to a point estimate, confidence limits can be generated for the

area under the receiver operating characteristic (AUROC) (80,81). In this analysis, the

confidence limits of the AUROC were not adjusted for possible correlation between

windows, nor were they adjusted to account for multiple comparisons.

62

Chapter 4 Results

Although analyses were carried out for each of the 36 Health Units in Ontario, it would

be overwhelming to present detailed results such as plots for each health unit. Therefore

in sections where it is not feasible to include results for all health units, the City of

Toronto Health Unit and the Grey Bruce Health Unit are chosen as representative

examples. Justification for the choice of these two health units is as follows. According

to total and daily median number of emergency department visits, these two Health Units

are most important. These two health regions also provide contrast: the City of Toronto

Health Unit is an urban area with a relatively high ratio of daily Telehealth calls to daily

visits while the Grey Bruce Health Unit is a rural area with a relatively low ratio of daily

calls to daily visits.

4.1 Summary Statistics of Telehealth Ontario Calls and Emergency Department Visits by Health Unit

Table 7, Table 8, and Table 9 present summary statistics for the number of Telehealth

Ontario calls and emergency department visits for respiratory complaints for each

approximate health unit area in Ontario over the study period.

Table 7 gives the total number of hospital emergency department visits and Telehealth

Ontario calls over the study period for each health unit. Health units were ranked in

importance based on the overall number of emergency visits. Table 7 and all subsequent

tables are sorted according to this ranking. Note that ranking Health Units by the overall

63

number of ED visits is similar (with some exceptions) to ranking them by median number

of daily visits as can be seen by examining Table 8.

Table 8 presents summary statistics (25th percentile, median, 75

th percentile) of the daily

number of Telehealth Ontario calls and emergency department visits for respiratory

complaints over the study period. Health units with a higher median number of visits do

not necessarily also have a higher median number of calls. To highlight this, Table 9

presents the ratio of the median number of daily calls to the median number of daily ED

visits.

Table 10 presents basic descriptive statistics of the age of the individuals Telehealth calls

for respiratory complaints were about and of the age of patients visiting the emergency

department for respiratory illness.

64

Table 7: Total Telehealth Ontario Calls and Emergency Department Visits for Respiratory

Complaints by Health Unit over Study Period

Hospital Emergency

Department Visits

Telehealth Ontario Calls

Public Health Unit Name (Approximate1)

2001

Census

Population2

Total

Number

of Visits

Percentage

of Total

Total

Number

of Calls

Percentage

of Total

City of Toronto Health Unit 2,481,494 59,048 10.6 35,808 19.4

Grey Bruce Health Unit 152,965 32,018 5.8 2,457 1.3

Simcoe Muskoka District Health Unit 430,156 29,647 5.3 8,361 4.5

Niagara Regional Area Health Unit 410,574 25,261 4.6 5,405 2.9

Peel Regional Health Unit 988,948 25,111 4.5 17,032 9.3

City of Ottawa Health Unit 791,477 24,502 4.4 14,727 8.0

City of Hamilton Health Unit 490,268 23,280 4.2 6,261 3.4

York Regional Health Unit 729,254 18,633 3.4 14,069 7.6

Leeds, Grenville and Lanark District Health Unit 159,101 17,989 3.2 2,345 1.3

Middlesex-London Health Unit 403,185 17,862 3.2 6,166 3.3

Durham Regional Health Unit 506,901 17,647 3.2 8,732 4.7

The Eastern Ontario Health Unit 185,968 16,230 2.9 2,698 1.5

Peterborough County-City Health Unit 125,856 15,105 2.7 2,884 1.6

Hastings and Prince Edward Counties Health Unit 150,816 14,703 2.6 3,036 1.6

Waterloo Health Unit 438,515 14,413 2.6 8,853 4.8

The District of Algoma Health Unit 117,185 14,192 2.6 1,758 1.0

Renfrew County and District Health Unit 96,467 13,306 2.4 1,621 0.9

Thunder Bay District Health Unit 155,462 12,812 2.3 3,120 1.7

Porcupine Health Unit 88,205 12,558 2.3 1,696 0.9

Haliburton, Kawartha, Pine Ridge District Health

Unit

161,761 12,295 2.2 1,827 1.0

North Bay Parry Sound District Health Unit 120,353 12,230 2.2 2,663 1.4

Oxford County Health Unit 99,270 11,725 2.1 1,606 0.9

Lambton Health Unit 126,971 11,549 2.1 1,336 0.7

Chatham-Kent Health Unit 107,709 11,451 2.1 1,557 0.8

Haldimand-Norfolk Health Unit 104,575 11,297 2.0 1,431 0.8

Halton Regional Health Unit 375,229 11,210 2.0 7,451 4.0

Windsor-Essex County Health Unit 374,975 9,991 1.8 3,965 2.2

Northwestern Health Unit 77,823 9,906 1.8 1,673 0.9

Kingston, Frontenac, Lennox, Addington Health Unit 235,664 9,806 1.8 2,820 1.5

Sudbury and District Health Unit 190,474 8,183 1.5 2,723 1.5

Huron County Health Unit 59,701 6,857 1.2 798 0.4

Wellington-Dufferin-Guelph Health Unit 238,326 6,835 1.2 3,132 1.7

Perth District Health Unit 73,675 5,131 0.9 1,172 0.6

Brant County Health Unit 118,580 4,814 0.9 1,706 0.9

Timiskaming Health Unit 35,245 4,072 0.7 498 0.3

Elgin-St. Thomas Health Unit 81,553 3,458 0.6 742 0.4

Total 555,127 184,129

1 Refer to section 3.7

2 Calculated using the 2001 census population of the dissemination areas making up the approximate health unit grouping

65

Table 8: Summary Statistics of Daily Telehealth Ontario Call and Emergency Department Visit

Activity for Respiratory Complaints by Health Unit over Study Period Public Health Unit Name Number of Daily Visits Number of Daily Calls

Median (25th Percentile,

75th Percentile)


75th Percentile)

City of Toronto Health Unit 75 (60, 103) 48 (35, 65)

Grey Bruce Health Unit 44 (30, 58) 3 (2, 5)

Simcoe Muskoka District Health Unit 41 (31, 52) 11 (8, 16)

Niagara Regional Area Health Unit 33 (25, 44) 7 (5, 10)

Peel Regional Health Unit 33 (25, 45) 23 (16, 32)

City of Ottawa Health Unit 33 (25, 45) 21 (13, 28)

City of Hamilton Health Unit 30 (24, 40) 8 (5, 12)

York Regional Health Unit 25 (19, 33) 19 (13, 26)

Leeds Grenville and Lanark District Health Unit 25 (18, 34) 3 (2, 5)

Middlesex-London Health Unit 23 (16, 32) 8 (5, 12)

Durham Regional Health Unit 24 (17, 32) 12 (8, 17)

The Eastern Ontario Health Unit 23 (16, 29) 3 (2, 6)

Peterborough County-City Health Unit 20 (14, 27) 4 (2, 6)

Hastings and Prince Edward Counties Health Unit 20 (13, 27) 4 (2, 6)

Waterloo Health Unit 19 (14, 26) 11 (8, 16)

The District of Algoma Health Unit 19 (14, 26) 2 (1, 4)

Renfrew County and District Health Unit 18 (13, 25) 2 (1, 3)

Thunder Bay District Health Unit 17 (12, 23) 4 (2, 6)

Porcupine Health Unit 17 (12, 23) 2 (1, 4)

Haliburton Kawartha Pine Ridge District Health 17 (11, 23) 2 (1, 4)

North Bay Parry Sound District Health Unit 16 (12, 22) 3 (2, 5)

Oxford County Health Unit 15 (11, 22) 2 (1, 3)

Lambton Health Unit 15 (11, 21) 2 (1, 3)

Chatham-Kent Health Unit 15 (10, 20) 2 (1, 3)

Haldimand-Norfolk Health Unit 15 (11, 21) 2 (1, 3)

Halton Regional Health Unit 15 (11, 20) 10 (6, 14)

Windsor-Essex County Health Unit 13 (9, 19) 5 (3, 8)

Northwestern Health Unit 13 (10, 18) 2 (1, 3)

Kingston Frontenac, Lennox and Addington

Health Unit 13 (9, 19) 4 (2, 6)

Sudbury and District Health Unit 11 (8, 15) 4 (2, 5)

Huron County Health Unit 9 (6, 13) 1 (0, 2)

Wellington-Dufferin-Guelph Health Unit 9 (6, 13) 4 (2, 6)

Perth District Health Unit 6 (4, 10) 1 (1, 3)

Brant County Health Unit 6 (4, 9) 2 (1, 4)

Timiskaming Health Unit 5 (3, 8) 0 (0, 1)

Elgin-St. Thomas Health Unit 4 (3, 7) 1 (0, 2)

66

Table 9: Ratio of the Median Number of Daily Telehealth Ontario Calls to Median Number of Daily

Hospital Emergency Department Visits by Health Unit Public Health Unit Name Ratio of Median Number of

Daily Calls to Median

Number of Daily Visits

City of Toronto Health Unit 0.64

Grey Bruce Health Unit 0.07

Simcoe Muskoka District Health Unit 0.27

Niagara Regional Area Health Unit 0.21

Peel Regional Health Unit 0.70

City of Ottawa Health Unit 0.64

City of Hamilton Health Unit 0.27

York Regional Health Unit 0.76

Leeds, Grenville and Lanark District Health Unit 0.12

Middlesex-London Health Unit 0.35

Durham Regional Health Unit 0.50

The Eastern Ontario Health Unit 0.13

Peterborough County-City Health Unit 0.20

Hastings and Prince Edward Counties Health Unit 0.20

Waterloo Health Unit 0.58

The District of Algoma Health Unit 0.11

Renfrew County and District Health Unit 0.11

Thunder Bay District Health Unit 0.24

Porcupine Health Unit 0.12

Haliburton, Kawartha, Pine Ridge District Health Unit 0.12

North Bay Parry Sound District Health Unit 0.19

Oxford County Health Unit 0.13

Lambton Health Unit 0.13

Chatham-Kent Health Unit 0.13

Haldimand-Norfolk Health Unit 0.13

Halton Regional Health Unit 0.67

Windsor-Essex County Health Unit 0.38

Northwestern Health Unit 0.15

Kingston, Frontenac, Lennox and Addington Health Unit 0.31

Sudbury and District Health Unit 0.36

Huron County Health Unit 0.11

Wellington-Dufferin-Guelph Health Unit 0.44

Perth District Health Unit 0.17

Brant County Health Unit 0.33

Timiskaming Health Unit 0.00

Elgin-St. Thomas Health Unit 0.25

67

Table 10: Ages of Individuals Telehealth Ontario Calls were Concerning and Ages of Emergency

Department Visit Patients by Health Unit

Emergency Department Visits Telehealth Ontario Calls

Public Health Unit Patient Age Age of Individual Call was

Concerning


25th Percentile)


25th Percentile)

City of Toronto Health Unit 21 (3, 51) 5 (1, 30)

Grey Bruce Health Unit 30 (9, 52) 7 (1, 30)

Simcoe Muskoka District Health Unit 24 (4, 48) 5 (1, 30)

Niagara Regional Area Health Unit 26 (5, 49) 6 (1, 31)

Peel Regional Health Unit 11 (2, 11) 4 (1, 25)

City of Ottawa Health Unit 19 (3, 49) 5 (1, 28)

City of Hamilton Health Unit 25 (5, 49) 5 (1, 28)

York Regional Health Unit 18 (3, 18) 3 (1, 23)

Leeds, Grenville and Lanark District Health

Unit

27 (7, 27) 7 (1, 31)

Middlesex-London Health Unit 18 (3, 18) 7 (1, 28)

Durham Regional Health Unit 20 (3, 20) 4 (1, 27)

The Eastern Ontario Health Unit 27 (7, 27) 6 (1, 30)

Peterborough County-City Health Unit 30 (10, 52) 13 (2, 31)

Hastings and Prince Edward Counties Health

Unit

27 (7, 49) 6 (1, 29)

Waterloo Health Unit 22 (3, 46) 6 (1, 28)

The District of Algoma Health Unit 28 (9, 51) 10 (2, 32)

Renfrew County and District Health Unit 27 (6, 49) 6 (1, 27)

Thunder Bay District Health Unit 22 (5, 46) 8 (2, 29)

Porcupine Health Unit 27 (6, 49) 9 (2, 29)

Haliburton, Kawartha, Pine Ridge District

Health Unit

30 (8, 53) 10 (2, 35)

North Bay Parry Sound District Health Unit 28 (7, 52) 8 (2, 31)

Oxford County Health Unit 24 (5, 45) 5 (1, 28)

Lambton Health Unit 26 (7, 48) 7 (1, 30)

Chatham-Kent Health Unit 23 (5, 46) 5 (1, 27)

Haldimand-Norfolk Health Unit 21 (5, 46) 6 (1, 29)

Halton Regional Health Unit 16 (2, 44) 3 (1, 26)

Windsor-Essex County Health Unit 22 (3, 50) 3 (1, 25)

Northwestern Health Unit 22 (4, 45) 8 (1, 30)

Kingston, Frontenac, Lennox and Addington

Health Unit

27 (6, 51) 10 (1, 30)

Sudbury and District Health Unit 30 (6, 56) 6 (2, 28)

Huron County Health Unit 28 (6, 53) 6 (1, 31)

Wellington-Dufferin-Guelph Health Unit 22 (4, 44) 5 (1, 28)

Perth District Health Unit 20 (4, 46) 6 (1, 27)

Brant County Health Unit 23 (5, 47) 5 (1, 27)

Timiskaming Health Unit 33 (13, 52) 19 (3, 34)

Elgin-St. Thomas Health Unit 19 (3, 44) 5 (1, 26)

68

4.2 Plots of Daily Calls and Daily Visits over Study Period

Figure 9 and Figure 10 show plots of the daily number of emergency department visits

and Telehealth Ontario calls for respiratory complaints for the approximate City of

Toronto Health Unit and the approximate Grey Bruce Health Unit over the study period

from June 1, 2004 to March 31, 2006.

69

Figure 9: Plot of the Daily Number of Emergency Department Visits and Telehealth Ontario Calls

for Respiratory Complaints for the Approximate City of Toronto Health Unit from June 1, 2004 to

March 31, 2006

70

Figure 10: Plot of the Daily Number of Emergency Department Visits and Telehealth Ontario Calls

for Respiratory Complaints for the Approximate Grey Bruce Health Unit from June 1, 2004 to

March 31, 2006

71

4.3 Qualitative Forecast Assessment

Figure 11 and Figure 13 show the predicted emergency department visits using each of

the three forecasting methods for zero days in the future (i.e. using calls to predict the

current number of visits) for the approximate City of Toronto Health Unit and the

approximate Grey Bruce Health Unit respectively over the validation dataset (days 302 to

669, or March 29, 2005 to March 31, 2006). Note that the time scale in these plots is with

respect to a reference of June 1, 2004 being day 1, and therefore the actual dates of

plotted values can be determined by adding the independent axis value less one to this

reference date. Days that were coded as weekends or statutory holidays (discussed in

section 3.6) are indicated at the bottom of the plots. Figure 12 and Figure 14 show plots

of the corresponding forecasting errors, with positive error indicating forecasts that are

too high and negative errors indicating forecasts that are too low. A zero-days-ahead

forecast is presented as a reference point, as it will be later shown that forecasting error

generally increases with lead time.

Figure 15 through Figure 18 show the same thing for a five-day-ahead forecast rather than

a zero-day-ahead forecast.

72

Figure 11: Zero-Day Ahead Emergency Department Visit Forecast for Respiratory Complaints over

the Validation Dataset for the (Approximate) City of Toronto Health Unit (using all three Forecasting

Methods)

73

Figure 12: Forecasting Errors (Predicted - Actual) for Zero-Day Ahead Emergency Department

Visit Prediction for Respiratory Complaints over the Validation Dataset for the (Approximate) City

of Toronto Health Unit

74

Figure 13: Zero-Day-Ahead Emergency Department Visit Forecast for Respiratory Complaints over

the Validation Dataset for the (Approximate) Grey Bruce Health Unit (using all three Forecasting

Methods)

75

Figure 14: Forecasting Errors (Predicted - Actual) for Zero-Day Ahead Emergency Department

Visit Prediction for Respiratory Complaints over the Validation Dataset for the (Approximate) Grey

Bruce Health Unit

76

Figure 15: Five-Day-Ahead Emergency Department Visit Forecast for Respiratory Complaints over

the Validation Dataset for the (Approximate) City of Toronto Health Unit (using all three Forecasting

Methods)

77


Prediction for Respiratory Complaints over the Validation Dataset for the (Approximate) City of

Toronto Health Unit

78

Figure 17: Five-Day-Ahead Emergency Department Visit Forecast for Respiratory Complaints over

the Validation Dataset for the (Approximate) Grey Bruce Health Unit (using all three Forecasting

Methods)

79

Figure 18: Forecasting Errors (Predicted - Actual) for Five-Day Ahead Emergency Department

Visit Prediction for Respiratory Complaints over the Validation Dataset for the (Approximate) Grey

Bruce Health Unit

80

Plots of the forecasted aggregate weekly hospital emergency department visits one week

in advance (aggregate of the forecasted visits for 1-7 days in advance) and two weeks in

advance (aggregate of the forecasted visits for 8-14 days in advance) for the City of

Toronto Health Unit and the Grey Bruce Health Unit are given in Figure 19 through

Figure 22 respectively. These plots better show the trends in the actual and forecasted

time series than do plots of the daily visits.

Appendix C provides plots for the forecasted aggregate weekly hospital emergency

department visits one week in advance for each of the 36 health units.

81

Figure 19: One-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency

Visits using all Three Forecasting Methods and the Corresponding Weekly Aggregate Number of

Actual Visits for the (Approximate) City of Toronto Health Unit

82

Figure 20: Two-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency



83

Figure 21: One-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency


Actual Visits for the (Approximate) Grey Bruce Health Unit

84

Figure 22: Two-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency



85

4.4 Quantitative Forecast Assessment

Summary statistics for the error in the daily forecasts over the validation dataset for the

City of Toronto Health Unit and the Grey Bruce Health Unit for each of the methods are

given in Table 11 and Table 12 respectively. Summary statistics for the error in the

aggregate weekly forecasts over the validation dataset for these two health units are given

in Table 13 and Table 14. Note in these tables the interquartile range is generally less for

models developed using the Fast Orthogonal Search or Parallel Cascade Identification

methods. Specifically in the case of the Toronto Health Unit, the models developed using

Fast Orthogonal Search had the lowest interquartile range (with some exceptions where

the Subspace models had a lower interquartile range). In the case of the Grey Bruce

Health Unit, the models developed with Parallel Cascade Identification generally had the

lowest interquartile range (with some exceptions where the Subspace models had a lower

interquartile range).

86

Table 11: Summary Statistics of the Error (Predicted-Actual) in Daily Forecasts for the

(Approximate) City of Toronto Health Unit over the Validation Dataset

Parallel Cascade Identification Fast Orthogonal Search Subspace Identification

Lead Median (25th Percentile,

75th Percentile)


75th Percentile)


75th Percentile)

0 -5 (-15, 4) -2 (-11, 5) -2 (-15, 9)

1 -4 (-14, 5) 0 (-9, 9) -2 (-13, 9)

2 -5 (-14, 5) -1 (-9, 9) -1 (-13, 10)

3 -5 (-13, 5) -2 (-11, 6) -2 (-12, 10)

4 -2 (-11, 8) -2 (-11, 6) 0 (-12, 10)

5 0 (-11, 8) 1 (-10, 9) -3 (-14, 8)

6 0 (-10, 11) 5 (-6, 15) -2 (-13, 10)

7 2 (-9, 12) 4 (-7, 13) -1 (-14, 11)

8 2 (-11, 12) 7 (-4, 17) 0 (-11, 10)

9 2 (-11, 13) 6 (-5, 17) 0 (-13, 11)

10 5 (-8, 15) 5 (-6, 15) 0 (-14, 15)

11 4 (-9, 18) 3 (-9, 11) 2 (-21, 20)

12 4 (-9, 18) 3 (-9, 12) 2 (-15, 16)

13 4 (-10, 16) 11 (-2, 22) 2 (-19, 17)

14 5 (-10, 16) 11 (-3, 22) 1 (-17, 20)

Table 12: Summary Statistics of the Error (Predicted-Actual) in Daily Forecasts for the

(Approximate) Grey Bruce Health Unit over the Validation Dataset


Lead Median (25th Percentile,

75th Percentile)


75th Percentile)


75th Percentile)

0 0 (-8, 8) 17 (3, 26) 1 (-6, 8)

1 0 (-6, 8) 9 (-5, 18) 10 (-1, 19)

2 0 (-7, 8) 9 (-5, 17) 2 (-6, 9)

3 0 (-7, 8) 9 (-5, 17) 2 (-7, 7)

4 0 (-8, 8) 12 (0, 24) 5 (-5, 13)

5 1 (-8, 9) 11 (0, 22) 4 (-6, 12)

6 1 (-7, 10) 11 (0, 22) 0 (-13, 11)

7 2 (-7, 9) 19 (8, 33) 3 (-6, 11)

8 2 (-6, 10) 21 (8, 34) 5 (-7, 14)

9 2 (-7, 9) 19 (7, 32) 3 (-5, 10)

10 3 (-6, 10) 19 (7, 31) 3 (-6, 10)

11 3 (-6, 11) 1581 (61, 261)

1 3 (-6, 12)

12 4 (-5, 11) 38 (27, 52) 6 (-6, 17)

13 4 (-5, 11) 30 (20, 44) 5 (-7, 13)

14 5 (-6, 12) 17 (5, 35) 7 (-5, 19)

1This model had poor fit over the training data (%MSE > 100%) and should not be used for prediction, explaining this observation

87

Table 13: Summary Statistics of the Error (Predicted-Actual) in the Forecasted Aggregate Number

of Weekly Hospital Emergency Department Visits for Respiratory Illness for the (Approximate) City

of Toronto Health Unit over the Validation Dataset


Weeks

Ahead


75th Percentile)


75th Percentile)


75th Percentile)

1 -5 (-57, 31) 18 (-31, 52) -28 (-70, 42)

2 5 (-66, 97) 48 (0, 97) -17 (-86, 58)

Table 14: Summary Statistics of the Error (Predicted-Actual) in the Forecasted Aggregate Number

of Weekly Hospital Emergency Department Visits for Respiratory Illness for the (Approximate) Grey

Bruce Health Unit over the Validation Dataset


Weeks

Ahead


75th Percentile)


75th Percentile)


75th Percentile)

1 5 (-38, 55) 76 (0, 141) 28 (0, 62)

2 21 (-23, 64) 3231 (259, 382)

1 45 (-21, 84)

1This model had poor fit over the training data (%MSE > 100%) and should not be used for prediction, explaining this observation

The %MSE and MAPE, shown in brackets, of the models over the validation dataset for

leads of 0, 5, 8, 11, and 14 days are given for each of the 36 Ontario health units in Table

15 through Table 19. Results for models with a %MSE of greater than 100% over the

training dataset were excluded from the analysis (indicated by footnotes in the tables).

When %MSE was greater than 100% over the training data, it was deemed that no

suitable forecasting model could be found using that method.

88


Respiratory Illness for Each of the 36 Health Units in Ontario over the Validation Dataset 0-Day Lead

Approximate Public Health Region

Parallel Cascade

Identification

Fast Orthogonal

Search

Subspace

Identification

City of Toronto Health Unit 40 (14) 36 (14) 48 (19)

Grey Bruce Health Unit 42 (56) 135 (56) 39 (23)

Simcoe Muskoka District Health Unit 39 (17) 42 (17) 45 (17)

Niagara Regional Area Health Unit 44 (26) 47 (26) 53 (28)

Peel Regional Health Unit 43 (19) 43 (19) 50 (23)

City of Ottawa Health Unit 39 (20) 41 (20) 42 (21)

City of Hamilton Health Unit 46 (26) 49 (26) 67 (31)

York Regional Health Unit 52 (22) 48 (22) 54 (25)

Leeds Grenville and Lanark District Health Unit 69 (26) 64 (26) 71 (27)

Middlesex-London Health Unit 73 (24) 59 (24) 64 (25)

Durham Regional Health Unit 50 (23) 47 (23) 55 (24)

The Eastern Ontario Health Unit 72 (32) 70 (32) 67 (27)

Peterborough County-City Health Unit 70 (36) 68 (36) 91 (41)

Hastings and Prince Edward Counties Health Unit 55 (30) 50 (30) 52 (32)

Waterloo Health Unit 59 (26) 59 (26) 68 (29)

The District of Algoma Health Unit 88 (30) 71 (30) 79 (32)

Renfrew County and District Health Unit 61 (54) 103 (54) 116 (51)

Thunder Bay District Health Unit 90 (43) 107 (43) 81 (32)

Porcupine Health Unit 68 (43) 72 (43) 123 (47)

Haliburton Kawartha Pine Ridge District Health 85 (37) 87 (37) 91 (38)

North Bay Parry Sound District Health Unit 68 (31) 65 (31) 73 (32)

Oxford County Health Unit 50 (34) 49 (34) 59 (37)

Lambton Health Unit 61 (48) 86 (48) 67 (41)

Chatham-Kent Health Unit 102 (75) 164 (75) 180 (75)

Haldimand-Norfolk Health Unit 60 (42) 62 (42) 59 (40)

Halton Regional Health Unit 61 (27) 56 (27) 65 (28)

Windsor-Essex County Health Unit 91 (46) 82 (46) 100 (51)

Northwestern Health Unit 111 (29) 108 (29) 95 (30)

Kingston Frontenac Lennox and Addington Health Unit 66 (35) 65 (35) 68 (34)

Sudbury and District Health Unit 78 (41) 79 (41) 125 (49)

Huron County Health Unit 69 (56) 71 (56) 71 (53)

Wellington-Dufferin-Guelph Health Unit 68 (47) 66 (47) 86 (50)

Perth District Health Unit 73 –1 77 –

1 82 –

1

Brant County Health Unit 81 –1 76 –

1 84 –

1

Timiskaming Health Unit 128 –1 101 –

1 151 –

1

Elgin-St. Thomas Health Unit 72 –1 75 –

1 78 –

1

Average 67 72 78

1 MAPE is not defined when there are zero visits; refer to section 3.9.1.2

89




Parallel Cascade

Identification

Fast Orthogonal

Search

Subspace

Identification


















Thunder Bay District Health Unit 93 (35) 234 (76) 85 (32)
















1 95 –

1


1 77 –

1


1 152 –

1


1 183 –

1

Average 69 212 82


90




Parallel Cascade

Identification

Fast Orthogonal

Search

Subspace

Identification









Leeds Grenville and Lanark District Health Unit 72 (31) –2 –

2 99 (35)






Waterloo Health Unit 66 (31) –2 –

2 90 (33)



Thunder Bay District Health Unit 112 (39) –2 –

2 93 (34)
















1 119 –

1


1 81 –

1


1 215 –

1


1 90 –

1

Average 74 95 112


2 Model had >100% MSE over training data and was considered inappropriate

91

Table 18: %MSE (MAPE) for 11-Day-Ahead Forecasts of Hospital Emergency Department Visits

for Respiratory Illness for Each of the 36 Health Units in Ontario over the Validation Dataset 11-Day Lead


Parallel Cascade

Identification

Fast Orthogonal

Search

Subspace

Identification


Grey Bruce Health Unit 54 (28) –2 –

2 60 (30)












Hastings and Prince Edward Counties Health Unit 61 (33) –2 –

2 83 (38)





2 116 (35)
















1 93 –

1


1 85 –

1


1 147 –

1


1 97 –

1

Average 79 92 93



92

Table 19: %MSE (MAPE) for 14-Day-Ahead Forecasts of Hospital Emergency Department Visits

for Respiratory Illness for Each of the 36 Health Units in Ontario over the Validation Dataset 14-Day Lead


Parallel Cascade

Identification

Fast Orthogonal

Search

Subspace

Identification



Simcoe Muskoka District Health Unit 61 (22) –2 –

2 73 (23)












Waterloo Health Unit 80 (36) –2 –

2 82 (34)




2 107 (33)



North Bay Parry Sound District Health Unit 72 (36) –2 –

2 75 (35)

Oxford County Health Unit 64 (39) –2 –

2 78 (43)












1 116 –

1


1 80 –

1


1 159 –

1


1 91 –

1

Average 84 97 101



93

An informal comparison of the forecasting accuracy for the three different methods and

the effect of lead time on forecasting accuracy can be made by looking at the ―Average‖

row at the bottom of each table. To formally compare the performance of the different

methods of forecasting visits, and examine the effect of lead time on forecasting

accuracy, a multi-level regression model was used to fit the transformed MSE. The

transformation was chosen to normalize the distribution of the residuals of the regression

model. The following power transformation was used for the %MSE:

1

1100

%1

MSE

MSET

Equation 4-1

MSET was modeled as a function of the prediction lead time in days and prediction

method, both treated as fixed effects. Health Unit was treated as a random effect in the

model to allow for the differences in the health units. Specifically, the multilevel

regression model was:

94

ijijijijjijT PCIFOSleadMSE 3210 (level 1—individual prediction

model)

jjj ratio 400 (level 2—health unit)

),0(~2

Nj

),0(~ 2 N

Equation 4-2

where

the subscript j is used to denote the jth health unit

the subscript i is the ith observation (for a prediction model) in health unit j

lead is the lead time of the predictor in days (0 through 14 days)

FOS is an indicator variable for the Fast Orthogonal Search algorithm (1 if the

prediction model used FOS, 0 otherwise)

PCI is an indicator variable for the Parallel Cascade Identification algorithm (1 if

the prediction model used PCI, 0 otherwise)

ratio is the ratio of the median number of daily calls to median number of daily

visits

β0 is the overall intercept

β1, β2, β3, β4 are the regression coefficients for lead, FOS, PCI and ratio

respectively

αj describes the variation attributable to the health unit

ε describes the unexplained variation in MSET

95

Observations where the MSE exceeded 100% over the training data were excluded from

the regression analysis (a total of 27 observations were excluded—all from Fast

Orthogonal Search); therefore the model was estimated from a total of 1593 observations

(each corresponding to a prediction model; 36 health units × 3 methods × 15 leads (0 to

14 days) – 27 excluded observations).

Table 20: Parameter Estimates for the Multilevel Regression Model of Transformed %MSE, MSET Variable (Coefficient) Parameter Estimate Standard Error p-value

Intercept (β0) 0.9884 0.000796 <0.0001

Prediction lead time (days) (β1) 0.000263 0.000024 <0.0001

Ratio of Median Number of Daily Calls to Median Number of Daily

Visits (β4)

-0.01030 0.002225 <0.0001

Method Subspace (SS) Identification referent n/a n/a

Fast Orthogonal Search (FOS) (β2) -0.00097 0.000277 0.0005

Parallel Cascade Identification (PCI) (β3) -0.00205 0.000275 <0.0001

Covariance Parameters σα2

7.112×10-6

σ2

5.646×10-6

This analysis indicates that %MSE increases with prediction lead in days. In terms of

forecasting accuracy as measured by %MSE, Parallel Cascade Identification produced the

most accurate forecasts followed by Fast Orthogonal Search and Subspace Identification.

A model including the interaction between method and lead found these interaction terms

not to be statistically significant. The covariance parameter estimates suggest about 55%

of the total unexplained variation is attributable to differences in health units.

To provide a visual interpretation of the results of the multilevel regression model, plots

of the model estimates for the %MSE for each of the forecasting methods for several

values of the ratio of the median number of daily calls to the median number of daily

visits are given in Figure 23 to Figure 25. This was done as it can be difficult to interpret

the regression coefficients as the dependent variable in the model is the transformed

96

%MSE--the corresponding equation for %MSE is nonlinear. In these plots, three

different values of the ratio of the median number of daily calls to median number of

daily visits is given: 0.6, 0.3, and 0.1. These values were chosen based on the range of

the calls-to-visits ratio given in Table 9.


Method for a Ratio of Median Daily Number of Calls to Median Daily Number of Visits of 0.6

REGRESSION MODEL ESTIMATES FOR

%MSE VERSUS PREDICTION LEAD BY FORECAST METHOD

(CALL-TO-VISITS RATIO OF 0.6)

40

50

60

70

80

90

100

0 2 4 6 8 10 12 14

Prediction Lead (Days)

%M

SE

Parallel Cascade Identification

Fast Orthogonal Search

Subspace Identification

97






40

50

60

70

80

90

100

0 2 4 6 8 10 12 14


%M

SE




98






40

50

60

70

80

90

100

0 2 4 6 8 10 12 14


%M

SE




4.5 Ability to Predict Increases

Figure 26 illustrates an example of analyses carried out of the ability to predict increases

in emergency department visits above a certain threshold. This example is for the City of

Toronto Health Unit for a seven-day window size and a threshold of 0% (i.e.

discrimination between visit increases/decreases). It was desired to predict increases

above this threshold one week in advance.

The information in this plot is as follows.

99

The validation data set is divided into windows as indicated by the vertical yellow lines.

The actual daily number of emergency visits is shown in red at the bottom of the plot.

The forecasted number of daily visits is plotted in green at the bottom of the plot. The

daily forecasted number of visits time series shown in each window is constructed using

the 1-day-ahead predictor through 7-day-ahead predictor as discussed in 3.9.1.1. The 1-

day-ahead prediction is the left-most point in the window and actually lies on the vertical

yellow line indicating the left boundary of the window. The 2-day-ahead prediction is the

second-from-the-left point in the window, and so on. Days that were holidays or

weekends are given at the bottom of the plot.

The horizontal red bars indicate the aggregate number of actual visits for a given window.

This is the sum of each of the daily visits over that window including the visit that falls on

the left boundary of the window, but not including that on the right side of the window.

The horizontal green bar indicates sum of the forecasted daily visits (generated as

discussed above) over that window. Note that the level of these bars in Figure 26 can be

directly compared to the points plotted in Figure 19.

If the actual aggregate number of visits for the current window exceeds the threshold

percentage increase relative to the baseline window then the window is flagged. In the

case of Figure 26, the baseline window is the immediately preceding window. The

procedure is repeated for the forecasted series. Flagged windows are indicated by an ―×‖

at the top of the plot: red ―×‖s indicate flagged windows for the actual time series, while

lower placed green ―×‖s indicate flagged windows for the forecasted time series.

100

The results of this analysis can be summarized in a two-by-two contingency table

(confusion matrix) giving the number of windows where actual increases were flagged as

increases (true positives), number of windows where actual increases were not flagged

(false negatives), number of windows that were flagged as increases but were not actual

increases (false positives), and number of windows that were not flagged where there was

no actual increase (true negatives). From these tables, the Matthew’s Correlation

Coefficient, sensitivity, specificity, positive predictive value, and negative predictive

value can be calculated with corresponding confidence intervals as discussed in section

3.9.1.3. These results for each of the 36 Ontario Health Units and each of the three

forecasting methods are given in Appendix B for each combination of window size (four-

day or seven-day) and threshold (ability to discriminate between increases and decreases

and nominal thresholds of 10% or 15% increases). It would be overwhelming to present

figures such as Figure 26 for each of these and so the results are presented in tabular

format only.

101

Figure 26: Plot Illustrating Analyses of PCI-Predicted versus Actual Sequence of

Increases/Decreases in Emergency Department Visits One Week in Advance for the City of Toronto

Health Unit

102

4.5.1 Increases in Emergency Visits Aggregated over a Seven Day Window

4.5.1.1 Ability to Discriminate Between Increases and Decreases in the

Aggregate Number of Emergency Visits One and Two Weeks in

Advance

After adjustment for multiple comparisons, none of the predictions for any of the 36

health units showed an ability to discriminate between increases or decreases in the

aggregate number of emergency visits 1-7 or 8-14 days in advance. Table B.1 in

Appendix B presents results for each health unit. The adjustment used the highly

conservative Bonferroni correction, and multiplied each observed p-value by 3 ×36 × 2 =

216 since three methods were tried on 36 PHUs to predict both one- and two-weeks

ahead. Values for Matthew’s Correlation Coefficient, sensitivity, specificity, positive

predictive value, and negative predictive value are reported in the tables when the p-value

for Fisher’s exact test is significant before applying the Bonferroni correction.

Another issue concerned the ability of Telehealth Ontario calls alone to predict increases

and decreases without use of FOS, PCI, or Subspace methods. The lower 95%

confidence intervals for the area under the receiver operating characteristic (AUROC)

were above 0.5 in only four of 72 cases: Niagara Regional Area Health Unit, City of

Ottawa Health Unit, Middlesex-London Health Unit, and Peterborough County-City

Health Unit. In these four cases the lower 95% confidence intervals were very close to

0.5. This provides no evidence for the ability of Telehealth calls to be used directly to

103

discriminate between increases or decreases in aggregate numbers of future emergency

department visits. Table B.2 in Appendix B presents the results for each health unit.

4.5.1.2 Ability to Predict Increases in the Aggregate Number of Future

Visits above a 10% Nominal Threshold One Week in Advance or

a Nominal 15% Threshold Two Weeks in Advance

After adjusting for multiple comparisons (again by the Bonferroni correction), none of the

predictions for any forecasting method for any of the 36 health units showed an ability to

warn of increases in the seven-day aggregate number of emergency visits above a

nominal 10% increase for the next week or above a nominal 15% increase for two weeks

in advance. Table B.3 in Appendix B presents results for each health unit.

The next issue concerned the ability of Telehealth calls alone to predict increases above

this same threshold one- and two-weeks in advance. Six of the 36 health units had lower

95% confidence intervals for area under the receiver operating characteristic that were

above 0.5. Results for the City of Toronto Health Unit indicated that Telehealth Ontario

Calls alone might be predictive of hospital emergency department visits one and two

weeks in advance. Grey Bruce Health Unit, Peel Regional Health Unit, City of Ottawa

Health Unit, Middlesex-London Health Unit, and Durham Region Health Unit had lower

95% confidence intervals AUROC above 0.5. However, this is only seven of 72 cases.

In five of these cases the lower 95% confidence interval was very close to 0.5. Table B.4

in Appendix B presents the results for each health unit.

104

4.5.2 Increases in Emergency Visits Aggregated over Four Day Windows

4.5.2.1 Ability to Discriminate Between Increases and Decreases in the

Aggregate Number of Future Visits over the Next Four Days

Table B.5 in Appendix B presents the results of the ability to discriminate between

increases and decreases in the aggregate number of future visits over the next four days

for each health unit. After conservative adjustment of the p-values for multiple

comparisons, there was evidence for ability to distinguish between increases and

decreases in the aggregate visits for a window consisting of the next four days over a

baseline of the aggregate number of visits over the previous four days in eight of the 36

health units: City of Toronto Health Unit, Peel Regional Health Unit, York Regional

Health Unit, Durham Regional Health Unit, Waterloo Health Unit, Renfrew County and

District Health Unit, Halton Regional Health Unit, and Windsor-Essex County Health

Unit. These results are summarized in Table 21.

Table 21: Health Units where Forecasts Show Ability to Discriminate between Increases and

Decreases in the Aggregate Number of Visits over the Next Four Days Health Unit Best Performing

Method

MCC Sensitivity Specificity PPV NPV

City of Toronto Health Unit FOS 0.68 0.85 0.83 0.85 0.83

Peel Regional Health Unit PCI/FOS

(PCI reported)

0.48 0.77 0.71 0.72 0.76

York Regional Health Unit PCI/SS

(PCI reported)

0.60 0.81 0.78 0.78 0.82

Durham Regional Health Unit PCI 0.46 0.74 0.72 0.71 0.75

Waterloo Health Unit FOS 0.44 0.70 0.74 0.75 0.69

Renfrew County and District Health Unit FOS 0.42 0.70 0.71 0.70 0.71

Halton Regional Health Unit FOS 0.51 0.79 0.72 0.69 0.82

Windsor-Essex County Health Unit FOS/PCI

(PCI reported)

0.51 0.79 0.72 0.72 0.79

Concerning the ability of Telehealth calls alone to predict four-day increases and

decreases, without the use of FOS, PCI, or Subspace methods, only two of the 36 health

105

units had lower 95% confidence intervals for AUROC that were above 0.5: the Eastern

Ontario Health Unit and Huron County Health Unit. Table B.6 in Appendix B presents

the results for each of the 36 health units.

4.5.2.2 Ability to Predict Increases in the Aggregate Number of Future

Visits above a 10% Nominal Threshold over the Next Four Days

Table B.7 in Appendix B presents the results for each health unit. After adjusting for

multiple comparisons and when a nominal threshold of 10% was considered, there was

evidence for ability to flag increases in the aggregate number of future hospital

emergency visits over a period of the next four days compared to the past four days in six

of the 36 health units: City of Toronto Health Unit, Peel Regional Health Unit, York

Regional Health Unit, Durham Regional Health Unit, Halton Regional Health Unit, and

Windsor Essex Health units. A summary of these results is given in Table 22.

Table 22: Health Units where Forecasts Show Ability to Predict 10% Nominal Increases in the

Aggregate Number of Visits over the Next Four Days Health Unit Best Performing

Method

MCC Sensitivity Specificity PPV NPV

City of Toronto Health Unit FOS 0.41 0.54 0.85 0.63 0.80

Peel Regional Health Unit PCI 0.49 0.74 0.77 0.59 0.87

York Regional Health Unit FOS 0.42 0.59 0.82 0.67 0.80

Durham Regional Health Unit PCI 0.46 0.66 0.81 0.66 0.81

Halton Regional Health Unit PCI 0.41 0.69 0.75 0.53 0.85

Windsor-Essex County Health Unit PCI 0.40 0.57 0.75 0.62 0.71

In the case of flagging a nominal 10% increase in visits using Telehealth calls only, the

95% confidence limits for AUROC did not include 0.5 in only two of the 36 health units:

the district of Algoma Health Unit and the Sudbury and District Health Unit. Table B.8

in Appendix B presents the results for each of the 36 health units.

106

Chapter 5 Discussion and Conclusions

5.1 Summary of Key Findings

5.1.1 Forecast Accuracy

This thesis found limited evidence that predictive models employing Telehealth Ontario

Calls and knowledge of upcoming statutory holidays and weekends could be used to

accurately predict future emergency department visits for respiratory illness. Assuming

that it is appropriate to apply the highly conservative Bonferroni correction for multiple

hypothesis testing, then there was no evidence of the ability of such models to accurately

predict increases in the number of weekly emergency department visits either one or two

weeks in advance. However, the Bonferroni correction assumes that each of the multiple

tests is independent and that there is the same probability of obtaining a significant result

in each test. If this were really the case here, then the significant results for individual

tests should occur at random positions in Table B1. In fact, there appears to be a much

greater tendency for significant results to occur near the top of the table, for health units

with larger populations and higher ratios of calls-to-visits. Moreover, the models could

predict increases in the aggregate number of emergency department visits over the next

four days in eight of the health units when discrimination between increases and

decreases was considered and in six of these eight health units when a nominal 10%

threshold was considered even after highly conservative adjustment of p-values. While

six out of 36 health units seems to be a small number, these six health units account for

approximately half of the Ontario population (refer to Table 7). The call-to-visits ratio

was highly correlated with whether a health unit had a significant result or not (in other

107

words these six health units had a significantly higher ratio of calls to visits than the

remaining ones).

Despite the fact that increases were arguably not predicted accurately for the weekly

number of emergency department visits, the forecasts did appear to follow the trend in

these visits for some health units (refer to Figure 19 through Figure 22 and Appendix C).

Unfortunately, plots of both the forecasted and actual time series and the forecasting

errors show that the forecasts sometimes miss important increases in the visits time series

(see for example Figure 15 and Figure 16) which is consistent with the poor ability to

predict increases.

Estimates of the accuracy of the forecasts in terms of the percent mean square error were

not consistent across health units. This is evident in the wide range of values of the mean

square errors in Table 15 through Table 19. This might be expected as the intensity of

Telehealth Ontario call and emergency department visit use had been found to vary

widely by geographic region (3). Some of the variation in the forecast error between

health units was explained by the ratio of the median daily number of calls to the median

daily number of visits as indicated by the results of the multilevel regression model.

Health units with a larger ratio of daily calls-to-visits generally had a lower %MSE.

Analysis of the area under the receiver operating characteristic (AUROC) showed that

Telehealth calls on their own (i.e. without use of a forecasting model) have little or no

ability to predict increases in emergency department visits as most of the confidence

intervals contain 0.50.

108

A lack of consistent ability to predict future increases across health units suggests some

chance findings, in particular for a predictor using only Telehealth calls alone.

5.1.2 Usefulness of Telehealth Ontario Calls versus Knowledge of Upcoming Holidays and Weekends to Predict Future Visits for Respiratory Illness

The prediction models have two independent variables: the number of Telehealth Ontario

calls and an indicator variable for upcoming holiday/weekends. Therefore some of the

ability of such a model to predict increases in hospital emergency department visits is due

to the calls variable and some is due to the weekends/holidays variable. It is well-known

that hospital emergency department visits increase during holidays and weekends (23), so

ability to predict increases based on this knowledge alone would provide little additional

information to those monitoring disease or trying to manage hospital resources.

The fact that prediction accuracy decreases with lead time (i.e. the coefficient for lead

time is statistically significant and greater than zero in the multi-level regression model

used to assess the impact of method and lead time on prediction accuracy) provides

indirect evidence that Telehealth Ontario calls do carry some advance information about

emergency department visits. Longer-lead predictors use Telehealth Ontario call

information further in the past than shorter lead predictors when generating predictions.

However, both longer- and shorter-lead predictors have upcoming weekends/holidays as

far ahead as they are predicting as this information is always available in advance.

109

Therefore, of the two independent variables in the model, only the Telehealth Ontario

calls variable could potentially have less useful information for longer lead predictors.

We would also not expect that the forecasts would follow trends in the visits as shown in

Figure 19 through Figure 22 using only information in an indicator variable for

weekends/holidays.

The fact that, even after applying the Bonferroni correction, predictions of the aggregate

number of visits over a four-day window show accuracy beyond chance for some health

units while predictions of the aggregate number of visits over a seven day window do not

show this suggests two possibilities: 1) that weekend/holiday information is more useful

for predicting visits than call information and/or 2) only accurate short-term forecasts are

possible (i.e. the delay in the conceptual framework in Figure 1 and Figure 2 is short and

calls from longer ago have little use in predicting visits).

5.1.3 Comparison of Forecasting Methods

The coefficients of the indicator variables for modeling method in the multi-level

regression model are all significant suggesting that both the Parallel Cascade

Identification and Fast Orthogonal Search algorithms generally produce more accurate

forecasting models (as measured by %MSE) than the Subspace Identification method.

This finding was further supported by the fact that methods that provided the best ability

to flag increases in the aggregate number of hospital visits (highest MCC) over a four-day

110

period were Parallel Cascade Identification and Fast Orthogonal search (refer to Table 21

and Table 22).

There are two possible reasons why these non-linear methods gave better results than the

subspace method. First, they may have been better able to capture linear information in

the call-to-visits relationship. Second, there may be important nonlinear information in

the call-to-visits relationship which could only be captured using nonlinear modeling

techniques.

5.2 Results in the Context of the Existing Literature

In each of the 36 health units the median patient age of callers for respiratory complaints

was less than the median patient age of patients visiting the emergency department for

respiratory complaints (refer to Table 10). This is consistent with previous research

examining calls and visits at the provincial level (1,4) and supports the conceptual

framework suggested in section 2.5.1.

The fact that Telehealth Ontario calls can be used to generate forecasts of both the daily

and the aggregated number of emergency department visits for respiratory illness that

generally follow the trend in emergency department visits (refer to Figure 19 through

Figure 22) for some of the health units is consistent with the finding by van Dijk et al.

that Telehealth Ontario calls are correlated with emergency department visits at the

provincial level (1). The study by van Dijk found that Telehealth Ontario calls and

111

Emergency Department visits were strongly correlated at a zero day lag and weakly

correlated at up to a 15 day lag. This is consistent with the finding that over all health

units in Ontario, there was a statistically significant decrease in forecasting accuracy with

increasing lead time as discussed in section 5.1.1.

None of the previous literature looked specifically at forecasting hospital emergency

department visits in Ontario or Canada, so there is nothing to directly compare the results

with in terms of forecasting accuracy. However, it is possible to make a general

comparison of the forecasting accuracy for other health services for the five studies

discussed in section 2.3. In order to compare the results obtained in this study with

previous literature, forecasting error must be measured in the same way. Unfortunately,

these five studies did not describe explicitly how the forecasting error they report was

calculated. This makes it difficult to be certain error was measured in the same way it

was in this thesis or to convert the error measurements into the same format. Only the

studies by Reis et al. and Jones et al. are compared below as these are the most relevant in

terms of the health service investigated and because the authors reported enough on the

methods used to make an attempt at comparison feasible.

The study by Reis et al. (39) found an MAPE of 27.5% in forecasts for emergency

department visits for respiratory complaints. Comparing this result (although it is not

possible to make a true comparison) to the MAPE results in Table 15 and Table 16 we

find that the predictions produced by the methods in this thesis are better for some of the

health units than what Reis had found. If the PCI- and FOS-found models perform better

that ARIMA methods, a modified version of the methods used in this thesis might be

valuable in developing models for the expected number of emergency department visits

112

for use in a conventional syndromic surveillance application; the need to do this has been

acknowledged in the syndromic surveillance literature (7).

The study by Jones et al. (28) reported a percent root mean square (%RMS) error of 5.2%

(relative to the mean number of beds occupied) for their forecasting model for bed

occupancy when exogenous inputs were used and 3.2% if no exogenous inputs were used.

It appears that the %RMS error was calculated as:

)(

))()((%

2

ny

nznyRMS

where

y(n) is the actual number of beds occupied

z(n) is the forecasted number of beds occupied

Using the standard deviations in the actual number of beds occupied reported by the

authors, the %RMS errors can be converted into %MSE errors. The %RMS error of 3%

(15.1 beds out of a mean of 441.06 beds with a standard deviation of 32.48 beds) in their

study corresponds to a %MSE of 15.12/32.48

2×100%=21.6% as calculated in this thesis.

The prediction models in Jones et al. therefore may appeared to perform better in that

application as a %MSE of 22% was lower than any of the prediction models in this thesis.

However, note that the authors in their study only test over a validation set consisting of

the next 32 days whereas in this thesis the models are tested over a validation set of

approximately one year. Shorter validation sets make it more likely that the relationship

will not change over the validation set and that better forecasting results will be obtained.

113

5.3 Study Strengths

The study undertaken in this thesis had strengths in terms of quality of the data used to

verify the forecasts, its pragmatic approach to studying the Telehealth call-emergency

department visit relationship, the various measures it used in reporting forecasting errors,

its examination of the call-visits relationship at the health unit level, and the use of several

approaches in establishing a relationship between calls and visits.

The high quality and coverage of the NACRS database (discussed in section 3.4.1) put the

study in a good position to be able to conclude whether forecasts of emergency

department visits for respiratory illness using Telehealth Ontario calls were accurate and

useful. Good measurement of the true number of emergency department visits is

necessary before it is possible to conclude whether or not forecasts are accurate. If only a

poor measurement of the actual number of visits were used, then poor forecasts could

have been due to either the methods used or simply to the fact that measurement of the

truth was inaccurate.

Because this study used a variety of approaches to measure the association between calls

and visits, including those allowing for nonlinearity in this association, there was

increased chance of capturing an association if one existed. The fact that similar results

were obtained by several different approaches to modeling the association reinforces the

study findings about predictive ability of Telehealth and weekend/holiday information.

For example, the models developed by all three forecasting methods generated similar

114

forecasts and errors for the City of Toronto as shown in Figure 11 and Figure 12.

Similarly, measurements of the %MSE and MAPE reported in Table 15 through Table 19

are consistent across the methods for a given health unit relative to other health units.

This study took a pragmatic approach to studying the association between calls and visits:

the association between calls and visits was examined from the point of view of how

knowledge of its existence might be used in a practical application of forecasting future

visits. This is in contrast to previous work in investigating data sources for syndromic

surveillance data that yielded only a correlation coefficient between the data source under

study and a measure of the outcome (1,32,33). Such a pragmatic approach allows the

results of the study to be more easily interpreted in the context of how they may be

applied.

This study quantified the forecasting error in a number of ways including summary

measures of the error, %MSE and MAPE, measurement of the ability to discriminate

between increases and decreases, ability to predict increases above a threshold, and plots

and summary statistics of the errors for select health units. Previous research (discussed

in section 2.3) has reported only summary measures of the error, which can be difficult to

interpret.

The association between calls and visits was investigated at the health unit level allowing

for regional differences in the relationship that might be expected because of differing use

in Telehealth Ontario and emergency department services. Previous work by van Dijk (1)

et al. had only looked at the relationship at the provincial level, although geographical

115

analysis of the intensity of Telehealth Ontario call and emergency department visit use

had been found to vary widely by geographic region (3). This allows the usefulness of

Telehealth calls in each of the health units to be assessed.

5.4 Study Limitations

This thesis had several limitations in terms of how it handled the confounding effect of

holidays/weekends, how it statistically tested the ability of forecasts to predict increases

in visits, its adjustment of the AUROC for multiple comparisons and correlation when

assessing the usefulness of raw Telehealth Ontario calls in making predictions, the small

sample size used in model validation, how it assigned calls and visits to Health Units, and

the assumptions made about the static nature of the Telehealth calls to ED visits

relationship.

Assessment of the ability to predict increases in the aggregate number of visits over a

seven day window was used to control for confounding, specifically the fact that

emergency visits and Telehealth calls are both known to vary on a weekly basis. Some

residual confounding may be present because of the fact that this adjustment does not

account for holidays. With a seven-day window, increase due for example to Christmas

holidays would not be controlled for. Because of this confounding effect, it is difficult to

say whether ability to predict increases over a four day window beyond chance was due

only to the weekend/holiday information presented to the model or to both this

information and the Telehealth call information. Fitting a model that only used

Telehealth call information and comparing it to the performance of a model that used both

116

Telehealth call information and weekend/holiday information would not necessarily

resolve this problem because Telehealth call information implicitly contains information

about holidays/weekends (i.e. there is some collinearity between these variables) since

call volume is known to increase on weekends (5).

The statistical tests to determine whether predicted increases are better than chance

assume that observations are independent of one another as do computation of confidence

intervals for measures of sensitivity, specificity, positive predictive value and negative

predictive value. Note that windows are not necessarily independent of each other

because of autocorrelation in the time series. It is also noted that when the memory

length of the predictors is less than the window size, there will be some correlation

between the predictions. Therefore predictions may not be independent of each other

either. No attempt was made to correct for this. This type of correlation would reduce

the effective sample size and widen the confidence intervals reported for the sensitivity,

specificity, positive predictive value, and negative predictive value. This also would

affect the calculation of the confidence limits for the AUROC in assessing the predictive

value of raw counts of Telehealth Ontario calls for respiratory complaints.

Although the upper and lower 95% confidence intervals for the AUROC are provided, it

could be argued that a one-tailed test is actually more appropriate since we are only

interested in determining whether or not the AUROC is greater than 0.5, and that really

the cutoff to consider the AUROC significant was p=0.025 rather than p=0.05. However,

this is more than offset by the fact that the AUROC is not adjusted for multiple

117

comparisons or for possible correlation in the prediction windows as discussed above.

Therefore it is entirely likely that positive findings were due only to chance.

The confidence intervals for the sensitivity and specificity are wide indicating that one

year of validation data is insufficient to give a good assessment of the predictive ability of

the forecasting models. For example, the sensitivity of the fast orthogonal search forecast

for the City of Toronto Health Unit had a 95% confidence interval of 0.34 to 0.72. It is

difficult to assess in advance how much data is required to train the models to achieve

reasonable fit, and therefore to know how much data will be available for validation. It

was also not known in advance how many increases and non-increases there would be

over the validation data set. This makes calculating sample size and study power very

difficult. The results of this study could therefore be useful in designing future follow-up

studies.

Because only patient forward sortation area is available in the call and visit data and

forward sortation areas do not have a one-to-one correspondence with health units as

discussed in section 3.7, the Health Unit labeling of the geographical groupings in this

study are only approximate. In other words, calls and visits always come from the same

geographic region, but the regions may not exactly represent the health units. If an

individual health unit were interested in applying the results, an analysis would have to be

carried out to determine the overlap in geographical regions.

118

A related issue is that callers from one region may seek care in another region. This

could be particularly problematic where health unit boundaries separate a populated area

from the centre that provides care for that area.

This study assumes that the relationship between calls and visits is time invariant. The

models capture the relationship between calls and visits over the first year of data and

then assume that this remains the same over the second year of data. This may not be the

case, however. For example, call patterns may change due to shifts in behaviour or the

delays between calls and visits may vary depending on the underlying cause of the

respiratory complaints. Better predictive models might be achieved by using triage

information rather than NACRS database information to fit the models. Since this

information is available in real-time, the models could be updated on an on-going basis to

correct for the time-varying nature of the calls-visits relationship.

5.5 Application of Results and Implications for Future Research

The forecasts using Telehealth Ontario calls and the methods presented in this thesis do

not generate forecasts that are accurate enough to be used as a sole means of making

decisions about future increases in the number of emergency department visits for

respiratory illness. However, the models may generate accurate enough predictions at

short leads to be used as a crude proxy for the actual number of emergency department

visits. The plots given in Appendix C provide evidence for this. Even though accurate

flagging of increases only occurs in six of the 36 health units, these health units account

119

for approximately half of the Ontario population (refer to Table 7). This could still allow

the forecasts to contribute valuable information to a surveillance system that incorporates

data from multiple sources. The information provided by the forecasts might be especially

useful because aggregate emergency department visit information at the health unit level

may be difficult, time consuming, and costly to obtain from other sources.

The forecasts generated by the methods discussed in this thesis might be improved if

emergency visits for past days were available. Recall from section 3.8.5 that these were

not assumed to be available to the forecasting system as NACRS data is only available at

the end of the fiscal year. If information collection systems are improved, it is feasible

that such information would be available. This would allow for three things. First, it

would allow the models to be updated on an on-going basis to correct for changes in the

call-to-visits relationship. Doing so would reduce, to some extent, the assumption of a

time invariant relationship discussed in section 5.4 (the length of time over which the

relationship was assumed to be time invariant would be reduced). Second, it would allow

an accurate number of baseline visits to be estimated when predicting increases, possibly

improving accuracy of these predictions. Third, when FOS-found difference equation

models are employed to make predictions, it would allow actual values of past visits to be

used in the models rather than calculated values of past visits.

The methods developed here might be applied to other syndromic surveillance time series

or for forecasting demand for other health services with better success. As stated earlier,

poor forecasts could be due to either: 1) that the methods were inadequate to capture

information in the data that would allow accurate forecasts or 2) that the data used to

120

produce the forecasts does not contain enough information in order to generate accurate

forecasts. Forecasting accuracy of the models might be improved if they incorporate

information from other independent variables. For example, if real-time triage data were

available, this might improve model accuracy.

The threshold approach to flagging increases using the forecasted time series might be

improved by developing a predictor model directly. In other words, instead of outputting

the forecasted number of visits, the predictor might output a binary variable indicating an

increase or no increase. Such predictors have been developed using Parallel Cascade

Identification for other purposes (68).

121

References

(1) van Dijk A, McGuinness D, Rolland E, Moore KM. Can Telehealth Ontario

respiratory call volume be used as a proxy for emergency department respiratory visit

surveillance by public health? CJEM 2008 Jan;10(1):18-24.

(2) Moore K. Real-time syndrome surveillance in Ontario, Canada: the potential use of

emergency departments and Telehealth. Eur.J.Emerg.Med. 2004 Feb;11(1):3-11.

(3) Moore KM. Integrating Ontario’s Telehealth program into a province-wide public

health surveillance system: evaluation and recommendations. 2008.

(4) Rolland E, Moore KM. Ontario's Telehealth: A Novel Syndromic Surveillance Tool.

Advances in Disease Surveillance 2007;4:190-190.

(5) Rolland E, Moore KM, Robinson VA, McGuinness D. Using Ontario's "Telehealth"

health telephone helpline as an early-warning system: a study protocol. BMC Health

Serv.Res. 2006 Feb 15;6:10.

(6) Webb P, Bain C, Pirozzo S. Essential Epidemiology. New York: Cambridge

University Press; 2005.

(7) Buckeridge DL, Burkom H, Campbell M, Hogan WR, Moore AW. Algorithms for

rapid outbreak detection: a research synthesis. Journal of Biomedical Informatics, 2005

4;38(2):99-113.

(8) Kleinman K, Abrams A. Assessing Surveillance Using Sensitivity, Specificity and

Timeliness. Statistical Methods in Medical Research 2006;15:445-464.

(9) Lombardo J, Buckeridge D editors. Disease Surveillance: A Public Health Informatics

Approach. 1st ed. New Jersey: John Wiley & Sons, Inc.; 2007.

122

(10) Naylor D, Basrur S, Bergeron M, Brunham R, Butler-Jones D, Dafoe G, et al.

National Advisory Committee on SARS and Public Health: Learning from SARS:

Renewal of Public Health in Canada. 2003.

(11) Ontario Ministry of Health and Long-Term Care. For The Public's Health : Initial

Report of the Ontario Expert Panel on SARS and Infectious Disease Control. 2003.

(12) Ontario Ministry of Health and Long-Term Care. Ontario Health Pandemic Influenza

Plan June 2005. 2005:68-78.

(13) Walker D, Keon W, Laupacis A, Low D, Moore K, Kitts J, et al. The Ontario Expert

Panel on SARS and Infectious Disease Control: For the Public's Health: A Plan of

Action--Final Report of the Ontario Expert Panel on SARS and Infectious Disease

Control. 2004.

(14) Campbell A. Commission to Investigate the Introduction and Spread of SARS in

Ontario: Interim Report: SARS and Public Health in Ontario. 2004.

(15) Mandl K, Overhage M, Wagner M, Lober W, Sebastiani P, Mostashari F, et al.

Implementing Syndromic Surveillance: A Practical Guide Informed by the Early

Experience. Journal of the American Medical Informatics Association 2004;11(2):141-

150.

(16) Burkom, HS, Murphy, SP, Shmueli G. Automated Time Series Forecasting for

Biosurveillance. Statistics in Medicine 2007;26:4202-4218.

(17) Lombardo J, Burkom H, Elbert E, Magruder S, Lewis S, Loschen W, et al. A

Systems Overview of the Electronic Surveillance System for the Early Notification of

Community-Based Epidemics (ESSENCE II). Journal of Urban Health: Bulletin of the

New York Academy of Medicine 2003;80(2, Supplement 1):i32-i42.

(18) Gostin L. Public Health Strategies for Pandemic Influenza. The Journal of the

American Medical Association 2006;295(14):1700-1704.

123

(19) Kwong J, Rosella L, Johansen H. Trends in influenza vaccination in Canada,

1996/1997 to 2005. Health Reports 2007;18(4):1-11.

(20) Ontario Hospital Association and Ontario Medical Association Joint Committee on

Communicable Disease Surveillance Protocols. Influenza Surveillance Protocol for

Ontario Hospitals. 2005;306.

(21) Eskin B, Levy R. Does This Patient Have Influenza? Annals of Emergency Medicine

2007;49(1):103-105.

(22) Public Health Agency of Canada. Leading Causes of Death and Hospitalization in

Canada. 2008; Available at: http://www.phac-aspc.gc.ca/publicat/lcd-pcd97/index-

eng.php. Accessed 08/01, 2008.

(23) Chan B, Schull MJ, Schultz SE. Atlas Report: Emergency Department Services in

Ontario 1993-2000. 2001.

(24) Brown A, Alikhan M, Anderson G, Baker G, Croxford R, Daniel I, et al. Hospital

Report 2003: Emergency Department Care. 2004.

(25) Menec V, Bruce S, MacWilliam L. Exploring Reasons for Bed Pressures in

Winnipeg Acute Care Hospitals. Canadian Journal of Aging 2005;24(Supplement 1):121-

131.

(26) Menec V, Black C, MacWilliam L, Aoki F. The Impact of Influenza-Associated

Respiratory Illnesses on Hospitalizations, Physician Visits, Emergency Room Visits, and

Mortality. Canadian Journal of Public Health 2003;94(1):59-63.

(27) Hanratty B, Robinson M. Coping with Winter Bed Crises. BMJ 1999;319:1511-

1512.

(28) Jones S, Joy M. Forecasting Demand of Emergency Care. Health Care Management

Science 2002;5:297-305.

http://www.phac-aspc.gc.ca/publicat/lcd-pcd97/index-eng.php

http://www.phac-aspc.gc.ca/publicat/lcd-pcd97/index-eng.php

124

(29) Huges D, McGuire A. Stochastic Demand, Production Responses and Hospital

Costs. Journal of Health Economics 2003;22:999-1010.

(30) Viberg M. Subspace-based Methods for the Identification of Linear Time-invariant

Systems. Automatica 1995;31(12):1835-1851.

(31) Ontario Ministry of Health and Long-Term Care. Telehealth Ontario. 2006;

Available at:

http://www.health.gov.on.ca/english/public/program/telehealth/telehealth_mn.html.

Accessed 03/21, 2009.

(32) Doroshenko A, Cooper D, Smith G, Gerard E, Chinemana F, Verlander N, et al.

Evaluation of Sydromic Surveillance Based on National Health Service Direct Derived

Data -- England and Wales. Morbidity and Mortality Weekly Report

2005;54(Supplement):117-122.

(33) Zheng W, Aitken R, Muscatello D, Churches T. Potential for early warning of viral

influenza activity in the community by monitoring clinical diagnoses of influenza in

hospital emergency departments. BMC Public Health 2007;7(1):250.

(34) Makridakis S, Wheelwright S, Hyndman R. Forecasting: methods and applications.

3rd ed. New York: Wiley; 1998.

(35) Brockwell P, Davis R. Introduction to Time Series and Forecasting. 2nd ed. New

York: Springer-Verlag; 2002.

(36) Finarelli H, Johnson T. Effective Demand Forecasting in 9 Steps. Healthcare

Financial Management 2004;58(11):52-58.

(37) Abdel-Aal R, Mangoud A. Modeling and Forecasting Monthly Patient Volume at a

Primary Health Care Clinic using Univariate Time-Series Analysis. Computer Methods

and Programs in Biomedicine 1998;56:235-247.

http://www.health.gov.on.ca/english/public/program/telehealth/telehealth_mn.html

125

(38) Diaz J, Aberdi J, Pajares M, Lopez C, Lopez R, Lage M, et al. A Model for

Forecasting Emergency Hospital Admissions: Effect of Environmental Variables. Journal

of Environmental Health 2001;64(3):15.

(39) Reis B, Mandl K. Time Series Modeling for Syndromic Surveillance. BMC Medical

Informatics and Decision Making 2003;3(2).

(40) Upshur R, Moineddin R, Crighton E, Kiefier L, Mamdani M. Simplicity within

Complexity: Seasonality and Predictability of Hospital Admissions in the Province of

Ontario 1988-2001, A Population-Based Analysis. BMC Health Services Research

2005;5(13).

(41) Crighton E, Moineddin R, Mamdani M, Upshur REG. Influenza and pneumonia

hospitalizations in Ontario: a time-series analysis. Epidemiology and Infection

2004;132:1167-1174.

(42) Marsden-Haug N, Foster V, Gould P, Elbert E, Wang H, Pavlin J. Code-based

Syndromic Surveillance for Influenzalike Illness by International Classification of

Diseases, Ninth Revision. Emerging Infectious Diseases 2007;13(2):207-216.

(43) Mandl K, Lee T. Integrating Medical Informatics and Health Services Research: The

Need for Dual Training at the Clinical Health Systems and Policy Levels. Journal of the

American Medical Informatics Association 2002;9(2):127-132.

(44) Canadian Institute for Health Information. Database Background and General Data

Limitations Documentation: National Ambulatory Care Reporting System (NACRS) FY

2005-2006. 2006.

(45) Canadian Institute for Health Information. Database Background and General Data

Limitations Documentation: National Ambulatory Care Reporting System (NACRS) FY

2004-2005. 2005.

(46) Reyes F, Aziz S, Li Y, Macey J, Winchester B, Garner M, et al. Influenza in Canada:

2006-2007 Season. Canada Communicable Disease Report 2008;34(3):1-25.

126

(47) Ontario Ministry of Health and Long-Term Care. ICD10-CA to ICD-9 Conversion

Table. 2006.

(48) Canadian Institute for Health Information. 2006/2007 Conversion Tables: ICD-10-

CA/CCI to ICD-9-CM (July 2006 Update). 2007.

(49) Canadian Institute for Health Information. CIHI Data Quality Study of Ontario

Emergency Department Visits for Fiscal Year 2004-2005--Executive Summary. 2007.

(50) Ontario Ministry of Health and Long-Term Care. Telehealth Ontario. 2002;

Available at:

http://www.health.gov.on.ca/english/public/program/telehealth/telehealth_mn.html.

Accessed 02/14, 2009.

(51) Statistics Canada. Postal Code Conversion File (PCCF), Reference Guide.

2006;92F0153GIE.

(52) Mechanda K, Puderer H. How Postal Codes Map to Geographic Areas.

2007;92F0138MIE.

(53) Statistics Canada. Health Region Boundary File. 2005.

(54) Statistics Canada. Health Regions: Boundaries and Correspondence with Census

Geography. 2007;82-402-XIE.

(55) Ljung L. From Data to Model: A Guided Tour of System Identification. 1994;LiTH-

ISY-R-1652.

(56) Ljung L. MATLAB System Identification Toolbox User's Guide. Version 6 ed.

Natick, MA: The MathWorks; 2003.

(57) D. Westwick. Methods for the Identification of Multiple-Input Nonlinear

SystemsMcGill University; 1995.

http://www.health.gov.on.ca/english/public/program/telehealth/telehealth_mn.html

127

(58) Gilbert P. State Space and ARMA Models: An Overview of the Equivalence.

1993;Bank of Canada Working Paper 93-4.

(59) Katayama T. Subspace methods for system identification: a realization approach.

London: Springer-Verlag; 2005.

(60) Overschee P, DeMoor B. N4SID: Subspace Algorithms for the Identification of

Combined Deterministic-Stochastic Systems. Automatica 1994;30(1):75-93.

(61) Korenberg M. A Robust Orthogonal Algorithm for System Identification and Time-

Series Analysis. Biological Cybernetics 1989;60:267-276.

(62) Korenberg M. Identifying Nonlinear Difference Equation and Functional Expansion

Representations: The Fast Orthogonal Algorithm. Annals of Biomedical Engineering

1988;16:123-142.

(63) Desrochers A. On an improved model reduction technique for nonlinear systems.

Automatica 1981;17(2):407-409.

(64) Korenberg M. Parallel Cascade identification and Kernel Estimation for Nonlinear

Systems. Annals of Biomedical Engineering 1991;19:429-455.

(65) Korenberg M, David R, Hunter I, Solomon JE. Parallel Cascade Identification and its

Application to Protein Family Prediction. Journal of Biotechnology 2001;91:35-47.

(66) Korenberg M, Doherty P. Rapid DTMF signal classification via parallel cascade

identification. Electronics Letters 1996;32(20):1862-1863.

(67) Korenberg M, McGaughey D, Aitken G. Parallel cascade prediction of turbulence

induced wavefront tilt. Electronics Letters 1996;32(14):1315-1316.

(68) Korenberg M, Solomon JE, Regelson ME. Parallel Cascade Identification as a

Means for Automatically Classifying Protein Sequences into Structural/Functional

Groups. Biological Cybernetics 2000;82:15-21.

128

(69) Eklund M, Korenberg M. Simulation of Aircraft Pilot Flight Controls Using

Nonlinear System Identification. Simulation 2000;75(2):72-81.

(70) Green J, Korenberg M. On the Advantages of Multi-Input Single-Output Parallel

Cascade Classifiers. Annals of Biomedical Engineering 2006;34(4):709-716.

(71) The MathWorks Inc. MATLAB Version 7 Release 14. 2003.

(72) Minz I, Korenberg M. Modeling Cooperative Gene Regulation Using Fast-

Orthogonal Search. The Open Bioinformatics Journal 2008;2:80-89.

(73) Windhorst U, Johansson H editors. Modern Techniques in Neuroscience Research.

1st ed. New York: Springer; 1999.

(74) Markridakis S, Wheelwright S, McGee V. Forecasting: Methods and Applications.

2nd ed. Toronto: John Wiley & Sons; 1983.

(75) Makridakis S. Accuracy Measures: Theoretical and Practical Concerns. International

Journal of Forecasting 1993;9:527-529.

(76) Goodwin P, Lawton R. On the asymmetry of the symmetric MAPE. International

Journal of Forecasting 1999;15:405-408.

(77) Baldi P, Brunak S, Chauvin Y, Andersen C, Nielsen H. Assessing the Accuracy of

Prediction Algorithms for Classification: An Overview. Bioiinformatics 2000;16(5):412-

424.

(78) Lund O, Nielsen M, Lundegaard C, Kesmir C, Brunak S. Immunological

Bioinformatics. 1st ed. Cambridge, Massachusetts: The MIT Press; 2005.

(79) Bender R, Lange S. Adjusting for multiple testing--when and how? Journal of

Clinical Epidemiology 2001;54:343-349.

(80) Bewick V, Cheek L, Ball J. Statistics review 13: Receiver operating characteristic

curves. Critical Care 2004;8(6):508-512.

129

(81) Park S, Goo J, Jo C. Receiver Operating Characteristic (ROC) Curve: Practical

Review for Radiologists. Korean Journal of Radiology 2004;5(1):11-18.

130

Appendices

APPENDIX A: Ethics Approval

131

APPENDIX B: Ability of Forecasts to Predict Increases in Emergency Department Visits Table B.1: Ability to Discriminate Between Increases and Decreases in Weekly Aggregate Number of Hospital Emergency Department Visits One or

Two Weeks in Advance

TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

City of Toronto Health Unit 1 FOS 19 10 13 6 0.038 1.000 0.34 0.64 0.44 0.81 0.70 0.46 0.88 0.75 0.53 0.90 0.58 0.37 0.78

City of Toronto Health Unit 1.000 PCI 20 9 14 5 0.007 1.000 0.42 0.68 0.48 0.84 0.75 0.51 0.91 0.79 0.58 0.93 0.63 0.41 0.81

City of Toronto Health Unit 1.000 SS 17 12 9 10 0.770 1.000 0.11 0.61 0.41 0.78 0.50 0.27 0.73 0.63 0.42 0.81 0.48 0.26 0.70



City of Toronto Health Unit 2.000 SS 13 11 11 13 1.000 1.000 0.05 0.57 ed 0.77 0.48 0.28 0.69 0.50 0.30 0.70 0.55 0.32 0.76

Grey Bruce Health Unit 1 FOS 8 13 21 6 0.338 1.000 0.15 0.35 0.15 0.59 0.79 0.59 0.92 0.54 0.25 0.81 0.63 0.45 0.79

Grey Bruce Health Unit 1.000 PCI 12 9 16 11 0.383 1.000 0.16 0.55 0.32 0.77 0.61 0.41 0.78 0.50 0.28 0.72 0.65 0.44 0.83

Grey Bruce Health Unit 1.000 SS 11 10 16 11 0.561 1.000 0.11 0.50 0.27 0.73 0.61 0.41 0.78 0.48 0.26 0.70 0.63 0.42 0.81

Grey Bruce Health Unit 2 FOS* 19 1 11 17 0.008 1.000 0.40 0.95 0.74 1.00 0.41 0.24 0.61 0.51 0.34 0.69 0.92 0.64 1.00



Simcoe Muskoka District Health Unit 1 FOS 13 11 14 10 0.564 1.000 0.12 0.52 0.31 0.73 0.60 0.39 0.79 0.55 0.32 0.76 0.58 0.37 0.77

Simcoe Muskoka District Health Uni 1.000 PCI 13 11 13 11 0.773 1.000 0.08 0.52 0.31 0.73 0.56 0.35 0.76 0.52 0.31 0.73 0.56 0.35 0.76

Simcoe Muskoka District Health Uni 1.000 SS 13 11 14 10 0.564 1.000 0.12 0.52 0.31 0.73 0.60 0.39 0.79 0.55 0.32 0.76 0.58 0.37 0.77

Simcoe Muskoka District Health Uni 2 FOS* 17 5 13 13 0.074 1.000 0.28 0.76 0.53 0.92 0.52 0.32 0.71 0.55 0.36 0.74 0.74 0.49 0.91



Niagara Regional Area Health Unit 1 FOS 12 9 17 10 0.244 1.000 0.19 0.55 0.32 0.77 0.64 0.44 0.81 0.52 0.30 0.74 0.67 0.46 0.83

Niagara Regional Area Health Unit 1.000 PCI 10 11 16 11 0.771 1.000 0.06 0.45 0.23 0.68 0.61 0.41 0.78 0.45 0.23 0.68 0.61 0.41 0.78

Niagara Regional Area Health Unit 1.000 SS 9 12 10 17 0.244 1.000 -0.20 0.40 0.19 0.64 0.39 0.22 0.59 0.32 0.15 0.54 0.48 0.27 0.69



Niagara Regional Area Health Unit 2.000 SS 16 7 15 10 0.049 1.000 0.30 0.68 0.45 0.86 0.62 0.41 0.80 0.60 0.39 0.79 0.70 0.47 0.87

Peel Regional Health Unit 1 FOS 13 10 12 13 0.780 1.000 0.05 0.55 0.32 0.76 0.50 0.30 0.70 0.48 0.28 0.69 0.57 0.34 0.77

Peel Regional Health Unit 1.000 PCI 12 11 15 10 0.563 1.000 0.12 0.50 0.28 0.72 0.62 0.41 0.80 0.52 0.30 0.74 0.59 0.39 0.78

Peel Regional Health Unit 1.000 SS 13 10 14 11 0.564 1.000 0.12 0.55 0.32 0.76 0.58 0.37 0.77 0.52 0.31 0.73 0.60 0.39 0.79




City of Ottawa Health Unit 1 FOS 17 7 19 5 0.001 0.256 0.50 0.70 0.47 0.87 0.80 0.59 0.93 0.76 0.53 0.92 0.74 0.54 0.89

City of Ottawa Health Unit 1.000 PCI 14 10 19 5 0.017 1.000 0.42 0.61 0.39 0.80 0.80 0.59 0.93 0.74 0.49 0.91 0.69 0.49 0.85

City of Ottawa Health Unit 1.000 SS 16 8 15 9 0.082 1.000 0.34 0.70 0.47 0.87 0.64 0.43 0.82 0.64 0.43 0.82 0.70 0.47 0.87




Health Unit Weeks

Ahead

Forecasting

Method

Fisher's

Exact Test

p-Value

Specificity Positive Predictive Value Negative Predictive ValueSensitivityConfusion Matrix

for Flagging

Increases

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)

132

Table B.1 (Continued): Ability to Discriminate Between Increases and Decreases in Weekly Aggregate Number of Hospital Emergency Department

Visits One or Two Weeks in Advance

TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

City of Hamilton Health Unit 1 FOS 16 11 12 9 0.383 1.000 0.17 0.58 0.37 0.77 0.59 0.36 0.79 0.63 0.41 0.81 0.54 0.33 0.74

City of Hamilton Health Unit 1.000 PCI 15 12 12 9 0.561 1.000 0.13 0.54 0.33 0.73 0.59 0.36 0.79 0.61 0.39 0.80 0.52 0.31 0.72

City of Hamilton Health Unit 1.000 SS 13 14 11 10 1.000 1.000 0.05 0.50 0.30 0.70 0.55 0.32 0.76 0.57 0.34 0.77 0.48 0.28 0.69




York Regional Health Unit 1 FOS 14 9 15 10 0.248 1.000 0.21 0.59 0.36 0.79 0.62 0.41 0.80 0.57 0.34 0.77 0.64 0.43 0.82

York Regional Health Unit 1.000 PCI 15 8 14 11 0.161 1.000 0.21 0.64 0.41 0.83 0.58 0.37 0.77 0.56 0.35 0.76 0.65 0.43 0.84

York Regional Health Unit 1.000 SS 11 12 14 11 1.000 1.000 0.08 0.50 0.28 0.72 0.58 0.37 0.77 0.50 0.28 0.72 0.58 0.37 0.77




1 FOS* 14 13 11 10 1.000 1.000 0.05 0.50 0.30 0.70 0.55 0.32 0.76 0.57 0.34 0.77 0.48 0.28 0.69

1.000 PCI 14 13 9 12 0.776 1.000 -0.05 0.50 0.30 0.70 0.45 0.24 0.68 0.52 0.31 0.72 0.43 0.23 0.66

Leeds, Grenville and Lanark Distri 1.000 SS 18 9 10 11 0.380 1.000 0.16 0.65 0.44 0.83 0.50 0.28 0.72 0.61 0.41 0.78 0.55 0.32 0.77

Leeds, Grenville and Lanark Distri 2 FOS* 20 3 4 21 1.000 1.000 0.07 0.86 0.65 0.97 0.19 0.07 0.39 0.48 0.32 0.64 0.63 0.24 0.91

Leeds, Grenville and Lanark Distri 2.000 PCI 13 10 15 10 0.386 1.000 0.16 0.55 0.32 0.76 0.62 0.41 0.80 0.55 0.32 0.76 0.62 0.41 0.80


Middlesex-London Health Unit 1 FOS 10 13 14 11 1.000 1.000 -0.01 0.41 0.21 0.64 0.58 0.37 0.77 0.45 0.23 0.68 0.54 0.34 0.72

Middlesex-London Health Unit 1.000 PCI 13 10 17 8 0.145 1.000 0.24 0.55 0.32 0.76 0.69 0.48 0.86 0.60 0.36 0.81 0.64 0.44 0.81

Middlesex-London Health Unit 1.000 SS 14 9 17 8 0.081 1.000 0.28 0.59 0.36 0.79 0.69 0.48 0.86 0.62 0.38 0.82 0.67 0.46 0.83

Middlesex-London Health Unit 2 FOS 11 12 15 10 0.771 1.000 0.12 0.48 0.27 0.69 0.64 0.43 0.82 0.55 0.32 0.77 0.57 0.37 0.76



Durham Regional Health Unit 1 FOS 15 7 17 9 0.041 1.000 0.33 0.67 0.43 0.85 0.67 0.46 0.83 0.61 0.39 0.80 0.72 0.51 0.88

Durham Regional Health Unit 1.000 PCI 14 8 15 11 0.161 1.000 0.21 0.62 0.38 0.82 0.59 0.39 0.78 0.54 0.33 0.74 0.67 0.45 0.84

Durham Regional Health Unit 1.000 SS 14 8 18 8 0.041 1.000 0.32 0.62 0.38 0.82 0.70 0.50 0.86 0.62 0.38 0.82 0.70 0.50 0.86



Durham Regional Health Unit 2.000 SS 10 14 13 11 1.000 1.000 -0.05 0.39 0.20 0.61 0.56 0.35 0.76 0.45 0.23 0.68 0.50 0.31 0.69

The Eastern Ontario Health Unit 1 FOS 7 14 20 7 0.750 1.000 0.12 0.33 0.15 0.57 0.78 0.58 0.91 0.54 0.25 0.81 0.60 0.42 0.76

The Eastern Ontario Health Unit 1.000 PCI 9 12 14 13 0.776 1.000 -0.02 0.43 0.22 0.66 0.56 0.35 0.75 0.43 0.22 0.66 0.56 0.35 0.75

The Eastern Ontario Health Unit 1.000 SS 8 13 14 13 0.565 1.000 -0.06 0.38 0.18 0.62 0.56 0.35 0.75 0.40 0.19 0.64 0.54 0.34 0.72


The Eastern Ontario Health Unit 2.000 PCI 11 9 19 9 0.144 1.000 0.22 0.53 0.29 0.76 0.69 0.49 0.85 0.53 0.29 0.76 0.69 0.49 0.85

The Eastern Ontario Health Unit 2.000 SS 11 9 15 13 0.770 1.000 0.08 0.53 0.29 0.76 0.55 0.36 0.74 0.43 0.23 0.66 0.64 0.43 0.82

Leeds, Grenville and Lanark District

Health Unit

Positive Predictive Value Negative Predictive ValueHealth Unit Weeks

Ahead

Forecasting

Method

Confusion Matrix

for Flagging

Increases

Fisher's

Exact Test

p-Value

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)

Sensitivity Specificity

133



TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Peterborough County-City Health Unit 1 FOS 15 8 19 6 0.008 1.000 0.41 0.64 0.41 0.83 0.77 0.56 0.91 0.70 0.46 0.88 0.71 0.51 0.87

Peterborough County-City Health Un 1.000 PCI 16 7 15 10 0.049 1.000 0.30 0.68 0.45 0.86 0.62 0.41 0.80 0.60 0.39 0.79 0.70 0.47 0.87

Peterborough County-City Health Un 1.000 SS 12 11 14 11 0.773 1.000 0.08 0.50 0.28 0.72 0.58 0.37 0.77 0.50 0.28 0.72 0.58 0.37 0.77

Peterborough County-City Health Un 2 FOS 12 8 18 10 0.143 1.000 0.23 0.58 0.33 0.80 0.66 0.46 0.82 0.52 0.30 0.74 0.70 0.50 0.86



1 FOS 12 10 18 8 0.143 1.000 0.23 0.52 0.30 0.74 0.70 0.50 0.86 0.58 0.33 0.80 0.66 0.46 0.82

1.000 PCI 12 10 17 9 0.244 1.000 0.19 0.52 0.30 0.74 0.67 0.46 0.83 0.55 0.32 0.77 0.64 0.44 0.81

Hastings and Prince Edward Countie 1.000 SS 13 9 13 13 0.573 1.000 0.09 0.57 0.34 0.78 0.52 0.32 0.71 0.48 0.28 0.69 0.61 0.39 0.80

Hastings and Prince Edward Countie 2 FOS* 17 5 10 16 0.351 1.000 0.18 0.76 0.53 0.92 0.41 0.22 0.61 0.50 0.32 0.68 0.69 0.41 0.89

Hastings and Prince Edward Countie 2.000 PCI 12 10 17 9 0.244 1.000 0.19 0.52 0.30 0.74 0.67 0.46 0.83 0.55 0.32 0.77 0.64 0.44 0.81


Waterloo Health Unit 1 FOS* 17 7 11 13 0.371 1.000 0.18 0.70 0.47 0.87 0.48 0.28 0.69 0.55 0.36 0.74 0.63 0.38 0.84

Waterloo Health Unit 1.000 PCI 14 10 12 12 0.772 1.000 0.09 0.57 0.34 0.77 0.52 0.31 0.72 0.52 0.31 0.72 0.57 0.34 0.77

Waterloo Health Unit 1.000 SS 15 9 14 10 0.248 1.000 0.21 0.61 0.39 0.80 0.60 0.39 0.79 0.58 0.37 0.78 0.63 0.41 0.81




The District of Algoma Health Unit 1 FOS 10 11 18 9 0.380 1.000 0.18 0.48 0.26 0.70 0.70 0.50 0.86 0.56 0.31 0.78 0.63 0.44 0.80

The District of Algoma Health Unit 1.000 PCI 13 8 14 13 0.393 1.000 0.14 0.62 0.38 0.82 0.52 0.32 0.71 0.50 0.30 0.70 0.64 0.41 0.83

The District of Algoma Health Unit 1.000 SS 12 9 17 10 0.244 1.000 0.20 0.57 0.34 0.78 0.63 0.42 0.81 0.55 0.32 0.76 0.65 0.44 0.83




1 FOS 11 15 13 9 1.000 1.000 0.01 0.40 0.21 0.61 0.61 0.39 0.80 0.53 0.29 0.76 0.48 0.29 0.67

1.000 PCI 14 12 12 10 0.772 1.000 0.09 0.52 0.31 0.72 0.57 0.34 0.77 0.57 0.34 0.77 0.52 0.31 0.72

Renfrew County and District Health 1.000 SS 13 13 8 14 0.393 1.000 -0.13 0.48 0.28 0.69 0.39 0.20 0.61 0.46 0.27 0.67 0.41 0.21 0.64

Renfrew County and District Health 2 FOS 10 12 17 9 0.557 1.000 0.14 0.48 0.26 0.70 0.67 0.46 0.83 0.53 0.29 0.76 0.62 0.42 0.79

Renfrew County and District Health 2.000 PCI 12 10 19 7 0.077 1.000 0.32 0.57 0.34 0.78 0.74 0.54 0.89 0.63 0.38 0.84 0.69 0.49 0.85


Thunder Bay District Health Unit 1 FOS* 20 3 9 16 0.098 1.000 0.28 0.86 0.65 0.97 0.38 0.20 0.59 0.54 0.37 0.71 0.77 0.46 0.95

Thunder Bay District Health Unit 1.000 PCI 12 11 15 10 0.563 1.000 0.12 0.50 0.28 0.72 0.62 0.41 0.80 0.52 0.30 0.74 0.59 0.39 0.78

Thunder Bay District Health Unit 1.000 SS 10 13 14 11 1.000 1.000 -0.01 0.41 0.21 0.64 0.58 0.37 0.77 0.45 0.23 0.68 0.54 0.34 0.72



Thunder Bay District Health Unit 2.000 SS 13 11 16 8 0.244 1.000 0.20 0.52 0.31 0.73 0.68 0.46 0.85 0.60 0.36 0.81 0.61 0.41 0.78

Hastings and Prince Edward Counties

Health Unit

Renfrew County and District Health

Unit

Health Unit Weeks

Ahead

Forecasting

Method

Confusion Matrix

for Flagging

Increases

Fisher's

Exact Test

p-Value

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)

Sensitivity Specificity Positive Predictive Value Negative Predictive Value

134



TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Porcupine Health Unit 1 FOS 12 11 13 12 1.000 1.000 0.04 0.50 0.28 0.72 0.54 0.33 0.73 0.48 0.27 0.69 0.56 0.35 0.76

Porcupine Health Unit 1.000 PCI 9 14 11 14 0.265 1.000 -0.17 0.36 0.17 0.59 0.46 0.27 0.67 0.36 0.17 0.59 0.46 0.27 0.67

Porcupine Health Unit 1.000 SS 11 12 12 13 1.000 1.000 -0.05 0.45 0.24 0.68 0.50 0.30 0.70 0.43 0.23 0.66 0.52 0.31 0.72


Porcupine Health Unit 2.000 PCI 12 10 12 14 1.000 1.000 0.01 0.52 0.30 0.74 0.48 0.29 0.68 0.44 0.24 0.65 0.57 0.34 0.77

Porcupine Health Unit 2.000 SS 10 12 16 10 0.770 1.000 0.06 0.43 0.22 0.66 0.63 0.42 0.81 0.47 0.24 0.71 0.59 0.39 0.76

1 FOS 7 13 14 14 0.382 1.000 -0.11 0.37 0.16 0.62 0.52 0.33 0.71 0.33 0.15 0.57 0.56 0.35 0.75

1.000 PCI 14 6 17 11 0.045 1.000 0.30 0.68 0.43 0.87 0.62 0.42 0.79 0.54 0.33 0.74 0.75 0.53 0.90

Haliburton, Kawartha, Pine Ridge D 1.000 SS 13 7 13 15 0.555 1.000 0.11 0.63 0.38 0.84 0.48 0.29 0.67 0.44 0.25 0.65 0.67 0.43 0.85

Haliburton, Kawartha, Pine Ridge D 2 FOS 11 14 12 11 1.000 1.000 0.00 0.46 0.26 0.67 0.54 0.33 0.74 0.50 0.28 0.72 0.50 0.30 0.70

Haliburton, Kawartha, Pine Ridge D 2.000 PCI 17 8 14 9 0.081 1.000 0.29 0.67 0.45 0.84 0.63 0.41 0.81 0.64 0.43 0.82 0.65 0.43 0.84


1 FOS 14 12 15 7 0.153 1.000 0.22 0.52 0.31 0.72 0.70 0.47 0.87 0.65 0.41 0.85 0.57 0.37 0.76

1.000 PCI 12 14 10 12 0.772 1.000 -0.08 0.44 0.24 0.65 0.48 0.27 0.69 0.48 0.27 0.69 0.44 0.24 0.65

North Bay Parry Sound District Hea 1.000 SS 16 10 13 9 0.246 1.000 0.21 0.60 0.39 0.79 0.61 0.39 0.80 0.63 0.41 0.81 0.58 0.37 0.78

North Bay Parry Sound District Hea 2 FOS* 20 2 4 22 0.674 1.000 0.13 0.90 0.70 0.99 0.19 0.06 0.38 0.46 0.31 0.63 0.71 0.29 0.96

North Bay Parry Sound District Hea 2.000 PCI 12 10 16 10 0.384 1.000 0.15 0.52 0.30 0.74 0.63 0.42 0.81 0.52 0.30 0.74 0.63 0.42 0.81


Oxford County Health Unit 1 FOS 10 15 12 11 0.771 1.000 -0.04 0.42 0.22 0.63 0.54 0.33 0.74 0.48 0.26 0.70 0.48 0.29 0.68

Oxford County Health Unit 1.000 PCI 12 13 14 9 0.573 1.000 0.08 0.46 0.26 0.67 0.63 0.41 0.81 0.55 0.32 0.77 0.54 0.34 0.72

Oxford County Health Unit 1.000 SS 16 9 15 8 0.082 1.000 0.29 0.63 0.41 0.81 0.67 0.45 0.84 0.65 0.43 0.84 0.64 0.43 0.82

Oxford County Health Unit 2 FOS* 13 11 15 9 0.385 1.000 0.16 0.52 0.31 0.73 0.64 0.43 0.82 0.57 0.34 0.78 0.59 0.39 0.78



Lambton Health Unit 1 FOS 11 13 17 7 0.371 1.000 0.16 0.43 0.23 0.66 0.72 0.51 0.88 0.59 0.33 0.82 0.58 0.39 0.75

Lambton Health Unit 1.000 PCI 14 10 12 12 0.772 1.000 0.09 0.57 0.34 0.77 0.52 0.31 0.72 0.52 0.31 0.72 0.57 0.34 0.77

Lambton Health Unit 1.000 SS 13 11 15 9 0.385 1.000 0.16 0.52 0.31 0.73 0.64 0.43 0.82 0.57 0.34 0.78 0.59 0.39 0.78




Chatham-Kent Health Unit 1 FOS 10 13 16 9 0.769 1.000 0.12 0.43 0.23 0.66 0.68 0.46 0.85 0.56 0.31 0.78 0.57 0.37 0.75

Chatham-Kent Health Unit 1.000 PCI 9 14 12 13 0.401 1.000 -0.09 0.39 0.20 0.61 0.52 0.31 0.72 0.43 0.22 0.66 0.48 0.29 0.68

Chatham-Kent Health Unit 1.000 SS 10 13 16 9 0.769 1.000 0.12 0.43 0.23 0.66 0.68 0.46 0.85 0.56 0.31 0.78 0.57 0.37 0.75

Chatham-Kent Health Unit 2 FOS 8 14 15 11 0.771 1.000 -0.03 0.38 0.18 0.62 0.59 0.39 0.78 0.42 0.20 0.67 0.55 0.36 0.74



Haliburton, Kawartha, Pine Ridge

District Health Unit

Negative Predictive Value

North Bay Parry Sound District Health

Unit

Forecasting

Method

Confusion Matrix

for Flagging

Increases

Fisher's

Exact Test

p-Value

Health Unit Weeks

Ahead

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)

Sensitivity Specificity Positive Predictive Value

135



TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Haldimand-Norfolk Health Unit 1 FOS 11 9 15 13 0.770 1.000 0.13 0.58 0.33 0.80 0.55 0.36 0.74 0.46 0.26 0.67 0.67 0.45 0.84

Haldimand-Norfolk Health Unit 1.000 PCI 12 8 14 14 0.565 1.000 0.15 0.63 0.38 0.84 0.52 0.33 0.71 0.46 0.27 0.67 0.68 0.45 0.86

Haldimand-Norfolk Health Unit 1.000 SS 10 10 15 13 1.000 1.000 0.08 0.53 0.29 0.76 0.55 0.36 0.74 0.43 0.23 0.66 0.64 0.43 0.82




Halton Regional Health Unit 1 FOS 13 14 13 8 0.565 1.000 0.14 0.50 0.30 0.70 0.64 0.41 0.83 0.62 0.38 0.82 0.52 0.32 0.71

Halton Regional Health Unit 1.000 PCI 13 14 13 8 0.565 1.000 0.14 0.50 0.30 0.70 0.64 0.41 0.83 0.62 0.38 0.82 0.52 0.32 0.71

Halton Regional Health Unit 1.000 SS 16 11 12 9 0.383 1.000 0.17 0.58 0.37 0.77 0.59 0.36 0.79 0.63 0.41 0.81 0.54 0.33 0.74




Windsor-Essex County Health Unit 1 FOS 11 12 15 10 0.771 1.000 0.07 0.45 0.24 0.68 0.62 0.41 0.80 0.50 0.27 0.73 0.57 0.37 0.76

Windsor-Essex County Health Unit 1.000 PCI 12 11 12 13 1.000 1.000 0.00 0.50 0.28 0.72 0.50 0.30 0.70 0.46 0.26 0.67 0.54 0.33 0.74

Windsor-Essex County Health Unit 1.000 SS 11 12 15 10 0.771 1.000 0.12 0.50 0.28 0.72 0.62 0.41 0.80 0.52 0.30 0.74 0.59 0.39 0.78

Windsor-Essex County Health Unit 2 FOS 12 14 9 13 0.401 1.000 -0.13 0.44 0.24 0.65 0.43 0.23 0.66 0.46 0.26 0.67 0.42 0.22 0.63

Windsor-Essex County Health Unit 2.000 PCI 11 15 12 10 1.000 1.000 -0.04 0.40 0.21 0.61 0.57 0.34 0.77 0.50 0.27 0.73 0.46 0.28 0.66

Windsor-Essex County Health Unit 2.000 SS 8 18 10 12 0.143 1.000 -0.20 0.32 0.15 0.54 0.48 0.27 0.69 0.40 0.19 0.64 0.39 0.22 0.59

Northwestern Health Unit 1 FOS 12 11 10 15 0.771 1.000 -0.08 0.50 0.28 0.72 0.42 0.23 0.63 0.42 0.23 0.63 0.50 0.28 0.72

Northwestern Health Unit 1.000 PCI 6 17 10 15 0.023 1.000 -0.31 0.27 0.11 0.50 0.42 0.23 0.63 0.29 0.11 0.52 0.41 0.22 0.61

Northwestern Health Unit 1.000 SS 12 11 8 17 0.377 1.000 -0.16 0.50 0.28 0.72 0.35 0.17 0.56 0.39 0.22 0.59 0.45 0.23 0.68

Northwestern Health Unit 2 FOS* 16 9 16 7 0.025 1.000 0.38 0.67 0.45 0.84 0.71 0.49 0.87 0.70 0.47 0.87 0.68 0.46 0.85



1 FOS 9 13 15 11 1.000 1.000 -0.03 0.38 0.18 0.62 0.59 0.39 0.78 0.42 0.20 0.67 0.55 0.36 0.74

1.000 PCI 11 11 14 12 1.000 1.000 0.03 0.48 0.26 0.70 0.56 0.35 0.75 0.45 0.24 0.68 0.58 0.37 0.77

Kingston, Frontenac and Lennox and 1.000 SS 13 9 15 11 0.385 1.000 0.16 0.57 0.34 0.78 0.59 0.39 0.78 0.52 0.31 0.73 0.64 0.43 0.82

Kingston, Frontenac and Lennox and 2 FOS 10 16 14 8 1.000 1.000 0.01 0.36 0.18 0.57 0.65 0.43 0.84 0.53 0.28 0.77 0.48 0.30 0.67

Kingston, Frontenac and Lennox and 2.000 PCI 16 10 14 8 0.147 1.000 0.25 0.60 0.39 0.79 0.65 0.43 0.84 0.65 0.43 0.84 0.60 0.39 0.79


Sudbury and District Health Unit 1 FOS 14 9 14 11 0.265 1.000 0.17 0.59 0.36 0.79 0.58 0.37 0.77 0.54 0.33 0.74 0.63 0.41 0.81

Sudbury and District Health Unit 1.000 PCI 11 12 12 13 1.000 1.000 -0.05 0.45 0.24 0.68 0.50 0.30 0.70 0.43 0.23 0.66 0.52 0.31 0.72

Sudbury and District Health Unit 1.000 SS 12 11 15 10 0.563 1.000 0.12 0.50 0.28 0.72 0.62 0.41 0.80 0.52 0.30 0.74 0.59 0.39 0.78


Sudbury and District Health Unit 2.000 PCI 16 8 15 9 0.082 1.000 0.29 0.65 0.43 0.84 0.64 0.43 0.82 0.63 0.41 0.81 0.67 0.45 0.84


Kingston, Frontenac and Lennox and

Addington Health Unit

Health Unit Weeks

Ahead

Forecasting

Method

Confusion Matrix

for Flagging

Increases

Fisher's

Exact Test

p-Value

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)


136



TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Huron County Health Unit 1 FOS 13 11 14 10 0.564 1.000 0.12 0.52 0.31 0.73 0.60 0.39 0.79 0.55 0.32 0.76 0.58 0.37 0.77

Huron County Health Unit 1.000 PCI 9 15 8 16 0.082 1.000 -0.25 0.39 0.20 0.61 0.36 0.18 0.57 0.36 0.18 0.57 0.39 0.20 0.61

Huron County Health Unit 1.000 SS 11 13 14 10 1.000 1.000 0.04 0.43 0.23 0.66 0.60 0.39 0.79 0.50 0.27 0.73 0.54 0.34 0.72


Huron County Health Unit 2.000 PCI 13 9 16 10 0.246 1.000 0.20 0.57 0.34 0.78 0.63 0.42 0.81 0.55 0.32 0.76 0.65 0.44 0.83


Wellington-Dufferin-Guelph Health Unit 1 FOS 11 13 12 12 1.000 1.000 -0.05 0.43 0.23 0.66 0.52 0.31 0.72 0.45 0.24 0.68 0.50 0.30 0.70

Wellington-Dufferin-Guelph Health 1.000 PCI 12 12 13 11 1.000 1.000 0.08 0.52 0.31 0.73 0.56 0.35 0.76 0.52 0.31 0.73 0.56 0.35 0.76

Wellington-Dufferin-Guelph Health 1.000 SS 11 13 10 14 0.564 1.000 -0.13 0.43 0.23 0.66 0.44 0.24 0.65 0.42 0.22 0.63 0.46 0.26 0.67

Wellington-Dufferin-Guelph Health 2 FOS 13 11 15 9 0.385 1.000 0.16 0.52 0.31 0.73 0.64 0.43 0.82 0.57 0.34 0.78 0.59 0.39 0.78

Wellington-Dufferin-Guelph Health 2.000 PCI 11 13 13 11 1.000 1.000 -0.01 0.43 0.23 0.66 0.56 0.35 0.76 0.48 0.26 0.70 0.52 0.32 0.71

Wellington-Dufferin-Guelph Health 2.000 SS 13 11 15 9 0.385 1.000 0.16 0.52 0.31 0.73 0.64 0.43 0.82 0.57 0.34 0.78 0.59 0.39 0.78

Perth District Health Unit 1 FOS 12 13 11 12 1.000 1.000 0.00 0.50 0.29 0.71 0.50 0.29 0.71 0.50 0.29 0.71 0.50 0.29 0.71

Perth District Health Unit 1.000 PCI 10 15 13 10 1.000 1.000 0.00 0.42 0.22 0.63 0.58 0.37 0.78 0.50 0.27 0.73 0.50 0.31 0.69

Perth District Health Unit 1.000 SS 12 13 10 13 0.578 1.000 -0.08 0.46 0.26 0.67 0.46 0.26 0.67 0.46 0.26 0.67 0.46 0.26 0.67


Perth District Health Unit 2.000 PCI 7 13 16 12 0.766 1.000 -0.05 0.37 0.16 0.62 0.59 0.39 0.76 0.37 0.16 0.62 0.59 0.39 0.76

Perth District Health Unit 2.000 SS 8 12 16 12 1.000 1.000 0.01 0.42 0.20 0.67 0.59 0.39 0.76 0.40 0.19 0.64 0.61 0.41 0.78

Brant County Health Unit 1 FOS 13 11 15 9 0.385 1.000 0.16 0.52 0.31 0.73 0.64 0.43 0.82 0.57 0.34 0.78 0.59 0.39 0.78

Brant County Health Unit 1.000 PCI 14 10 11 13 1.000 1.000 0.05 0.57 0.34 0.77 0.48 0.28 0.69 0.50 0.30 0.70 0.55 0.32 0.76

Brant County Health Unit 1.000 SS 13 11 14 10 0.564 1.000 0.12 0.52 0.31 0.73 0.60 0.39 0.79 0.55 0.32 0.76 0.58 0.37 0.77




Timiskaming Health Unit 1 FOS 9 9 16 14 1.000 1.000 0.03 0.50 0.26 0.74 0.53 0.34 0.72 0.39 0.20 0.61 0.64 0.43 0.82

Timiskaming Health Unit 1.000 PCI 12 6 15 15 0.369 1.000 0.16 0.67 0.41 0.87 0.50 0.31 0.69 0.44 0.25 0.65 0.71 0.48 0.89

Timiskaming Health Unit 1.000 SS 12 6 17 13 0.145 1.000 0.26 0.67 0.41 0.87 0.60 0.41 0.77 0.50 0.29 0.71 0.75 0.53 0.90




Elgin-St. Thomas Health Unit 1 FOS 16 10 12 10 0.384 1.000 0.17 0.60 0.39 0.79 0.57 0.34 0.77 0.60 0.39 0.79 0.57 0.34 0.77

Elgin-St. Thomas Health Unit 1.000 PCI 14 12 13 9 0.401 1.000 0.13 0.52 0.31 0.72 0.61 0.39 0.80 0.59 0.36 0.79 0.54 0.33 0.73

Elgin-St. Thomas Health Unit 1.000 SS 14 12 11 11 1.000 1.000 0.04 0.52 0.31 0.72 0.52 0.31 0.73 0.54 0.33 0.74 0.50 0.29 0.71




Positive Predictive Value Negative Predictive ValueAdjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)

Sensitivity SpecificityHealth Unit Weeks

Ahead

Forecasting

Method

Confusion Matrix

for Flagging

Increases

Fisher's

Exact Test

p-Value

137

Table B.2: Ability of Telehealth Ontario Calls to Directly Discriminate between Increases and Decreases in

the Weekly Aggregate Number of Hospital Emergency Department Visits One or Two Weeks in Advance

Health Unit Weeks

Ahead

Area

Under

ROC

Lower

95% CI

Upper

95% CI

City of Toronto Health Unit 1 0.67 0.50 0.83

2 0.60 0.43 0.76

Grey Bruce Health Unit 1 0.59 0.43 0.76

2 0.51 0.34 0.68

Simcoe Muskoka District Health Unit 1 0.47 0.30 0.64

2 0.58 0.41 0.75

Niagara Regional Area Health Unit 1 0.41 0.24 0.57

2 0.67 0.51 0.82

Peel Regional Health Unit 1 0.45 0.29 0.62

2 0.56 0.38 0.73

City of Ottawa Health Unit 1 0.69 0.53 0.84

2 0.60 0.43 0.77

City of Hamilton Health Unit 1 0.48 0.31 0.65

2 0.54 0.37 0.71

York Regional Health Unit 1 0.61 0.45 0.77

2 0.61 0.44 0.77

Leeds, Grenville and Lanark District Health Unit 1 0.50 0.33 0.67

2 0.47 0.31 0.64

Middlesex-London Health Unit 1 0.68 0.53 0.83

2 0.59 0.42 0.75

Durham Regional Health Unit 1 0.67 0.50 0.83

2 0.54 0.37 0.71

The Eastern Ontario Health Unit 1 0.37 0.20 0.53

2 0.56 0.38 0.73

Peterborough County-City Health Unit 1 0.60 0.44 0.77

2 0.67 0.51 0.82

Hastings and Prince Edward Counties Health Unit 1 0.64 0.48 0.81

2 0.56 0.39 0.72

Waterloo Health Unit 1 0.50 0.33 0.66

2 0.56 0.40 0.73

The District of Algoma Health Unit 1 0.58 0.41 0.75

2 0.53 0.36 0.70

Renfrew County and District Health Unit 1 0.56 0.39 0.72

2 0.51 0.34 0.69

Thunder Bay District Health Unit 1 0.57 0.40 0.73

2 0.54 0.37 0.72

138

Table B.2 (Continued): Ability of Telehealth Ontario Calls to Directly Discriminate between Increases and

Decreases in the Weekly Aggregate Number of Hospital Emergency Department Visits One or Two Weeks

in Advance Health Unit Weeks

Ahead

Area

Under

ROC

Lower

95% CI

Upper

95% CI

Porcupine Health Unit 1 0.48 0.31 0.65

2 0.46 0.29 0.63

Haliburton, Kawartha, Pine Ridge District Health Unit 1 0.55 0.38 0.72

2 0.56 0.39 0.73

North Bay Parry Sound District Health Unit 1 0.49 0.32 0.66

2 0.55 0.37 0.72

Oxford County Health Unit 1 0.57 0.40 0.73

2 0.47 0.30 0.63

Lambton Health Unit 1 0.52 0.35 0.69

2 0.55 0.38 0.72

Chatham-Kent Health Unit 1 0.52 0.35 0.69

2 0.62 0.45 0.78

Haldimand-Norfolk Health Unit 1 0.50 0.32 0.67

2 0.49 0.32 0.66

Halton Regional Health Unit 1 0.55 0.38 0.71

2 0.59 0.43 0.76

Windsor-Essex County Health Unit 1 0.48 0.31 0.66

2 0.49 0.33 0.66

Northwestern Health Unit 1 0.33 0.17 0.49

2 0.46 0.29 0.63

Kingston, Frontenac and Lennox and Addington Health Unit 1 0.54 0.37 0.71

2 0.53 0.36 0.70

Sudbury and District Health Unit 1 0.52 0.36 0.69

2 0.44 0.26 0.61

Huron County Health Unit 1 0.50 0.33 0.67

2 0.40 0.23 0.56

Wellington-Dufferin-Guelph Health Unit 1 0.47 0.30 0.64

2 0.62 0.46 0.78

Perth District Health Unit 1 0.45 0.28 0.62

2 0.50 0.32 0.67

Brant County Health Unit 1 0.48 0.32 0.65

2 0.64 0.48 0.80

Timiskaming Health Unit 1 0.50 0.33 0.67

2 0.60 0.44 0.77

Elgin-St. Thomas Health Unit 1 0.55 0.37 0.72

2 0.61 0.44 0.79

139

Table B.3: Ability to Predict Increases in the Weekly Aggregate Number of Hospital Emergency Department Visits Above a Nominal 10% Increase One

Week Ahead or Nominal 15% Increase Two Weeks Ahead

TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI







Grey Bruce Health Unit 1 FOS 0 8 40 0 1.000 1.000 0.00 0.00 0.37 1.00 0.91 1.00 0.83 0.70 0.93



Grey Bruce Health Unit 2 FOS* 2 10 36 0 0.059 1.000 0.36 0.17 0.02 0.48 1.00 0.90 1.00 1.00 0.16 1.00 0.78 0.64 0.89



Simcoe Muskoka District Health Unit 1 FOS 1 7 37 3 0.530 1.000 0.07 0.13 0.00 0.53 0.93 0.80 0.98 0.25 0.01 0.81 0.84 0.70 0.93



Simcoe Muskoka District Health Uni 2 FOS* 0 13 32 3 0.553 1.000 -0.16 0.00 0.00 0.25 0.91 0.77 0.98 0.00 0.00 0.71 0.71 0.56 0.84











Peel Regional Health Unit 1.000 SS 4 13 23 8 1.000 1.000 -0.03 0.24 0.07 0.50 0.74 0.55 0.88 0.33 0.10 0.65 0.64 0.46 0.79










Health Unit Weeks

Ahead

Forecasting

Method

Fisher's

Exact Test

p-Value


for Flagging

Increases

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)

140

Table B.3 (Continued): Ability to Predict Increases in the Weekly Aggregate Number of Hospital Emergency Department Visits Above a Nominal 10%

Increase One Week Ahead or Nominal 15% Increase Two Weeks Ahead

TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI







York Regional Health Unit 1 FOS 1 14 29 4 1.000 1.000 -0.08 0.07 0.00 0.32 0.88 0.72 0.97 0.20 0.01 0.72 0.67 0.51 0.81






Leeds, Grenville and Lanark District Health Unit1 FOS* 0 9 39 0 1.000 0.00 0.00 0.34 1.00 0.91 1.00 0.81 0.67 0.91

Leeds, Grenville and Lanark Distri 1.000 PCI 4 5 28 11 0.432 1.000 0.14 0.44 0.14 0.79 0.72 0.55 0.85 0.27 0.08 0.55 0.85 0.68 0.95


Leeds, Grenville and Lanark Distri 2 FOS* 0 16 31 1 1.000 1.000 -0.10 0.00 0.00 0.21 0.97 0.84 1.00 0.00 0.00 0.98 0.66 0.51 0.79

Leeds, Grenville and Lanark Distri 2.000 PCI 4 12 21 11 0.742 1.000 -0.10 0.25 0.07 0.52 0.66 0.47 0.81 0.27 0.08 0.55 0.64 0.45 0.80










Durham Regional Health Unit 1.000 SS 5 10 28 5 0.249 1.000 0.21 0.33 0.12 0.62 0.85 0.68 0.95 0.50 0.19 0.81 0.74 0.57 0.87



Durham Regional Health Unit 2.000 SS 5 10 21 12 1.000 1.000 -0.03 0.33 0.12 0.62 0.64 0.45 0.80 0.29 0.10 0.56 0.68 0.49 0.83


The Eastern Ontario Health Unit 1.000 PCI 3 11 25 9 1.000 1.000 -0.05 0.21 0.05 0.51 0.74 0.56 0.87 0.25 0.05 0.57 0.69 0.52 0.84

The Eastern Ontario Health Unit 1.000 SS 4 10 25 9 1.000 1.000 0.02 0.29 0.08 0.58 0.74 0.56 0.87 0.31 0.09 0.61 0.71 0.54 0.85


The Eastern Ontario Health Unit 2.000 PCI 6 8 28 6 0.139 1.000 0.26 0.43 0.18 0.71 0.82 0.65 0.93 0.50 0.21 0.79 0.78 0.61 0.90

The Eastern Ontario Health Unit 2.000 SS 3 11 26 8 1.000 1.000 -0.02 0.21 0.05 0.51 0.76 0.59 0.89 0.27 0.06 0.61 0.70 0.53 0.84

Positive Predictive Value Negative Predictive ValueHealth Unit Weeks

Ahead

Forecasting

Method

Confusion Matrix Fisher's

Exact Test

p-Value

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)


141



TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Peterborough County-City Health Unit 1 FOS 5 13 27 3 0.132 1.000 0.23 0.28 0.10 0.53 0.90 0.73 0.98 0.63 0.24 0.91 0.68 0.51 0.81



Peterborough County-City Health Un 2 FOS 5 8 28 7 0.263 1.000 0.19 0.38 0.14 0.68 0.80 0.63 0.92 0.42 0.15 0.72 0.78 0.61 0.90



Hastings and Prince Edward Counties Health Unit1 FOS 4 7 37 0 0.002 1.000 0.55 0.36 0.11 0.69 1.00 0.91 1.00 1.00 0.40 1.00 0.84 0.70 0.93



Hastings and Prince Edward Countie 2 FOS* 2 10 30 6 1.000 1.000 0.00 0.17 0.02 0.48 0.83 0.67 0.94 0.25 0.03 0.65 0.75 0.59 0.87


Hastings and Prince Edward Countie 2.000 SS 2 10 24 12 0.465 1.000 -0.16 0.17 0.02 0.48 0.67 0.49 0.81 0.14 0.02 0.43 0.71 0.53 0.85

Waterloo Health Unit 1 FOS* 0 17 31 0 1.000 0.00 0.00 0.20 1.00 0.89 1.00 0.65 0.49 0.78






The District of Algoma Health Unit 1 FOS 0 14 33 1 1.000 1.000 -0.09 0.00 0.00 0.23 0.97 0.85 1.00 0.00 0.00 0.98 0.70 0.55 0.83




The District of Algoma Health Unit 2.000 PCI 3 9 19 17 0.311 1.000 -0.20 0.25 0.05 0.57 0.53 0.35 0.70 0.15 0.03 0.38 0.68 0.48 0.84

The District of Algoma Health Unit 2.000 SS 3 9 24 12 0.728 1.000 -0.08 0.25 0.05 0.57 0.67 0.49 0.81 0.20 0.04 0.48 0.73 0.54 0.87

Renfrew County and District Health Unit 1 FOS 1 13 34 0 0.292 1.000 0.23 0.07 0.00 0.34 1.00 0.90 1.00 1.00 0.03 1.00 0.72 0.57 0.84



Renfrew County and District Health 2 FOS 1 15 32 0 0.333 1.000 0.21 0.06 0.00 0.30 1.00 0.89 1.00 1.00 0.03 1.00 0.68 0.53 0.81




Thunder Bay District Health Unit 1.000 PCI 5 11 20 12 0.757 1.000 -0.06 0.31 0.11 0.59 0.63 0.44 0.79 0.29 0.10 0.56 0.65 0.45 0.81





Specificity Positive Predictive Value Negative Predictive ValueHealth Unit Weeks

Ahead

Forecasting

Method


Exact Test

p-Value

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)

Sensitivity

142



TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI







Haliburton, Kawartha, Pine Ridge District Health Unit1 FOS 2 11 33 2 0.294 1.000 0.16 0.15 0.02 0.45 0.94 0.81 0.99 0.50 0.07 0.93 0.75 0.60 0.87



Haliburton, Kawartha, Pine Ridge D 2 FOS 4 13 22 9 0.747 1.000 -0.06 0.24 0.07 0.50 0.71 0.52 0.86 0.31 0.09 0.61 0.63 0.45 0.79



North Bay Parry Sound District Health Unit 1 FOS 1 15 32 0 0.333 1.000 0.21 0.06 0.00 0.30 1.00 0.89 1.00 1.00 0.03 1.00 0.68 0.53 0.81



North Bay Parry Sound District Hea 2 FOS* 12 4 19 13 0.034 1.000 0.32 0.75 0.48 0.93 0.59 0.41 0.76 0.48 0.28 0.69 0.83 0.61 0.95



Oxford County Health Unit 1 FOS 1 15 32 0 0.333 1.000 0.21 0.06 0.00 0.30 1.00 0.89 1.00 1.00 0.03 1.00 0.68 0.53 0.81



Oxford County Health Unit 2 FOS* 1 13 34 0 0.292 1.000 0.23 0.07 0.00 0.34 1.00 0.90 1.00 1.00 0.03 1.00 0.72 0.57 0.84









Chatham-Kent Health Unit 1 FOS 2 15 31 0 0.121 1.000 0.28 0.12 0.01 0.36 1.00 0.89 1.00 1.00 0.16 1.00 0.67 0.52 0.80



Chatham-Kent Health Unit 2 FOS 2 12 29 5 1.000 1.000 -0.01 0.14 0.02 0.43 0.85 0.69 0.95 0.29 0.04 0.71 0.71 0.54 0.84

Chatham-Kent Health Unit 2.000 PCI 5 9 22 12 1.000 1.000 0.00 0.36 0.13 0.65 0.65 0.46 0.80 0.29 0.10 0.56 0.71 0.52 0.86


Specificity Positive Predictive Value Negative Predictive ValueFisher's

Exact Test

p-Value

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)

SensitivityHealth Unit Weeks

Ahead

Forecasting

Method

Confusion Matrix

143



TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Haldimand-Norfolk Health Unit 1 FOS 1 13 27 7 0.407 1.000 -0.16 0.07 0.00 0.34 0.79 0.62 0.91 0.13 0.00 0.53 0.68 0.51 0.81

Haldimand-Norfolk Health Unit 1.000 PCI 3 11 23 11 0.510 1.000 -0.11 0.21 0.05 0.51 0.68 0.49 0.83 0.21 0.05 0.51 0.68 0.49 0.83

Haldimand-Norfolk Health Unit 1.000 SS 4 10 21 13 0.741 1.000 -0.09 0.29 0.08 0.58 0.62 0.44 0.78 0.24 0.07 0.50 0.68 0.49 0.83






Halton Regional Health Unit 1.000 SS 3 14 25 6 1.000 1.000 -0.02 0.18 0.04 0.43 0.81 0.63 0.93 0.33 0.07 0.70 0.64 0.47 0.79




Windsor-Essex County Health Unit 1 FOS 1 11 32 4 1.000 1.000 -0.04 0.08 0.00 0.38 0.89 0.74 0.97 0.20 0.01 0.72 0.74 0.59 0.86

Windsor-Essex County Health Unit 1.000 PCI 3 9 30 6 0.671 1.000 0.09 0.25 0.05 0.57 0.83 0.67 0.94 0.33 0.07 0.70 0.77 0.61 0.89

Windsor-Essex County Health Unit 1.000 SS 5 7 29 7 0.143 1.000 0.22 0.42 0.15 0.72 0.81 0.64 0.92 0.42 0.15 0.72 0.81 0.64 0.92

Windsor-Essex County Health Unit 2 FOS 1 8 37 2 0.472 1.000 0.10 0.11 0.00 0.48 0.95 0.83 0.99 0.33 0.01 0.91 0.82 0.68 0.92

Windsor-Essex County Health Unit 2.000 PCI 1 8 26 13 0.250 1.000 -0.19 0.11 0.00 0.48 0.67 0.50 0.81 0.07 0.00 0.34 0.76 0.59 0.89

Windsor-Essex County Health Unit 2.000 SS 2 7 27 12 1.000 1.000 -0.07 0.22 0.03 0.60 0.69 0.52 0.83 0.14 0.02 0.43 0.79 0.62 0.91

Northwestern Health Unit 1 FOS 1 11 32 4 1.000 1.000 -0.04 0.08 0.00 0.38 0.89 0.74 0.97 0.20 0.01 0.72 0.74 0.59 0.86



Northwestern Health Unit 2 FOS* 1 20 26 1 1.000 1.000 0.03 0.05 0.00 0.24 0.96 0.81 1.00 0.50 0.01 0.99 0.57 0.41 0.71



Kingston, Frontenac and Lennox and Addington Health Unit1 FOS 4 11 31 2 0.067 1.000 0.29 0.27 0.08 0.55 0.94 0.80 0.99 0.67 0.22 0.96 0.74 0.58 0.86

Kingston, Frontenac and Lennox and 1.000 PCI 5 10 23 10 1.000 1.000 0.03 0.33 0.12 0.62 0.70 0.51 0.84 0.33 0.12 0.62 0.70 0.51 0.84


Kingston, Frontenac and Lennox and 2 FOS 2 15 27 4 1.000 1.000 -0.02 0.12 0.01 0.36 0.87 0.70 0.96 0.33 0.04 0.78 0.64 0.48 0.78

Kingston, Frontenac and Lennox and 2.000 PCI 5 12 19 12 0.753 1.000 -0.09 0.29 0.10 0.56 0.61 0.42 0.78 0.29 0.10 0.56 0.61 0.42 0.78

Kingston, Frontenac and Lennox and 2.000 SS 6 11 17 14 0.555 1.000 -0.10 0.35 0.14 0.62 0.55 0.36 0.73 0.30 0.12 0.54 0.61 0.41 0.78




Sudbury and District Health Unit 2 FOS 0 18 30 0 1.000 0.00 0.00 0.19 1.00 0.88 1.00 0.63 0.47 0.76



Health Unit Weeks

Ahead

Forecasting

Method


Exact Test

p-Value

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)


144



TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI







Wellington-Dufferin-Guelph Health Unit 1 FOS 4 15 27 2 0.197 1.000 0.21 0.21 0.06 0.46 0.93 0.77 0.99 0.67 0.22 0.96 0.64 0.48 0.78



Wellington-Dufferin-Guelph Health 2 FOS 5 13 27 3 0.132 1.000 0.23 0.28 0.10 0.53 0.90 0.73 0.98 0.63 0.24 0.91 0.68 0.51 0.81



Perth District Health Unit 1 FOS 2 17 22 7 0.286 1.000 -0.17 0.11 0.01 0.33 0.76 0.56 0.90 0.22 0.03 0.60 0.56 0.40 0.72

Perth District Health Unit 1.000 PCI 6 13 21 8 1.000 1.000 0.04 0.32 0.13 0.57 0.72 0.53 0.87 0.43 0.18 0.71 0.62 0.44 0.78



Perth District Health Unit 2.000 PCI 3 14 20 11 0.320 1.000 -0.19 0.18 0.04 0.43 0.65 0.45 0.81 0.21 0.05 0.51 0.59 0.41 0.75





Brant County Health Unit 2 FOS 0 14 30 4 0.307 1.000 -0.19 0.00 0.00 0.23 0.88 0.73 0.97 0.00 0.00 0.60 0.68 0.52 0.81




Timiskaming Health Unit 1.000 PCI 4 11 21 12 0.742 1.000 -0.10 0.27 0.08 0.55 0.64 0.45 0.80 0.25 0.07 0.52 0.66 0.47 0.81


Timiskaming Health Unit 2 FOS 2 15 27 4 1.000 1.000 -0.02 0.12 0.01 0.36 0.87 0.70 0.96 0.33 0.04 0.78 0.64 0.48 0.78



Elgin-St. Thomas Health Unit 1 FOS 2 18 24 4 1.000 1.000 -0.06 0.10 0.01 0.32 0.86 0.67 0.96 0.33 0.04 0.78 0.57 0.41 0.72


Elgin-St. Thomas Health Unit 1.000 SS 7 13 18 10 1.000 1.000 -0.01 0.35 0.15 0.59 0.64 0.44 0.81 0.41 0.18 0.67 0.58 0.39 0.75




Positive Predictive Value Negative Predictive ValueAdjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)

Sensitivity SpecificityHealth Unit Weeks

Ahead

Forecasting

Method


Exact Test

p-Value

145

Table B.4: Ability of Telehealth Ontario Calls to Directly Predict Increases the Weekly Aggregate

Number of Hospital Emergency Department Visits Above a Nominal 10% Increase One Week Ahead

or Nominal 15% Increase Two Weeks Ahead Health Unit Weeks

Ahead

Area

Under

ROC

Lower

95% CI

Upper

95% CI

City of Toronto Health Unit 1 0.70 0.55 0.86

2 0.67 0.52 0.83

Grey Bruce Health Unit 1 0.74 0.57 0.91

2 0.62 0.43 0.80

Simcoe Muskoka District Health Unit 1 0.63 0.39 0.88

2 0.52 0.31 0.73

Niagara Regional Area Health Unit 1 0.52 0.34 0.70

2 0.65 0.49 0.81

Peel Regional Health Unit 1 0.50 0.33 0.67

2 0.68 0.52 0.83

City of Ottawa Health Unit 1 0.66 0.46 0.86

2 0.69 0.51 0.87

City of Hamilton Health Unit 1 0.68 0.49 0.86

2 0.55 0.37 0.72

York Regional Health Unit 1 0.57 0.41 0.74

2 0.61 0.45 0.78

Leeds, Grenville and Lanark District Health Unit 1 0.64 0.47 0.81

2 0.57 0.40 0.74

Middlesex-London Health Unit 1 0.69 0.53 0.84

2 0.61 0.43 0.79

Durham Regional Health Unit 1 0.69 0.54 0.85

2 0.59 0.40 0.77

The Eastern Ontario Health Unit 1 0.42 0.23 0.61

2 0.51 0.31 0.70

Peterborough County-City Health Unit 1 0.56 0.39 0.73

2 0.56 0.39 0.72

Hastings and Prince Edward Counties Health Unit 1 0.60 0.39 0.81

2 0.56 0.36 0.75

Waterloo Health Unit 1 0.53 0.36 0.70

2 0.63 0.47 0.79

The District of Algoma Health Unit 1 0.55 0.33 0.76

2 0.56 0.35 0.77

Renfrew County and District Health Unit 1 0.55 0.36 0.75

2 0.44 0.25 0.63

Thunder Bay District Health Unit 1 0.47 0.29 0.64

2 0.57 0.40 0.74

146

Table B.4 (Continued): Ability of Telehealth Ontario Calls to Directly Predict Increases the Weekly

Aggregate Number of Hospital Emergency Department Visits Above a Nominal 10% Increase One

Week Ahead or Nominal 15% Increase Two Weeks Ahead Health Unit Weeks

Ahead

Area

Under

ROC

Lower

95% CI

Upper

95% CI

Porcupine Health Unit 1 0.52 0.32 0.72

2 0.51 0.33 0.69

Haliburton, Kawartha, Pine Ridge District Health Unit 1 0.65 0.46 0.84

2 0.58 0.41 0.75

North Bay Parry Sound District Health Unit 1 0.65 0.48 0.82

2 0.57 0.40 0.75

Oxford County Health Unit 1 0.43 0.25 0.61

2 0.52 0.32 0.72

Lambton Health Unit 1 0.53 0.35 0.70

2 0.49 0.31 0.66

Chatham-Kent Health Unit 1 0.62 0.46 0.78

2 0.57 0.39 0.75

Haldimand-Norfolk Health Unit 1 0.43 0.24 0.63

2 0.55 0.37 0.72

Halton Regional Health Unit 1 0.48 0.29 0.66

2 0.66 0.49 0.84

Windsor-Essex County Health Unit 1 0.61 0.41 0.80

2 0.59 0.41 0.77

Northwestern Health Unit 1 0.41 0.24 0.58

2 0.39 0.22 0.56

Kingston, Frontenac and Lennox and Addington Health Unit 1 0.59 0.41 0.77

2 0.50 0.31 0.69

Sudbury and District Health Unit 1 0.54 0.37 0.72

2 0.54 0.37 0.72

Huron County Health Unit 1 0.62 0.42 0.81

2 0.54 0.35 0.73

Wellington-Dufferin-Guelph Health Unit 1 0.53 0.37 0.70

2 0.58 0.41 0.75

Perth District Health Unit 1 0.61 0.44 0.78

2 0.58 0.40 0.76

Brant County Health Unit 1 0.61 0.44 0.78

2 0.65 0.49 0.81

Timiskaming Health Unit 1 0.44 0.27 0.62

2 0.57 0.40 0.74

Elgin-St. Thomas Health Unit 1 0.62 0.45 0.78

2 0.64 0.48 0.79

147

Table B.5: Ability to Discriminate Between Increases and Decreases in the Aggregate Number of Hospital Emergency Department Visits over the Next

Four-Days

TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

City of Toronto Health Unit FOS 40 7 35 7 0.000 0.000 0.68 0.85 0.72 0.94 0.83 0.69 0.93 0.85 0.72 0.94 0.83 0.69 0.93

City of Toronto Health Unit PCI 39 8 35 7 0.000 0.000 0.66 0.83 0.69 0.92 0.83 0.69 0.93 0.85 0.71 0.94 0.81 0.67 0.92

City of Toronto Health Unit SS 37 10 33 9 0.000 0.000 0.57 0.79 0.64 0.89 0.79 0.63 0.90 0.80 0.66 0.91 0.77 0.61 0.88

Grey Bruce Health Unit FOS 23 18 32 16 0.035 1.000 0.23 0.56 0.40 0.72 0.67 0.52 0.80 0.59 0.42 0.74 0.64 0.49 0.77

Grey Bruce Health Unit PCI 27 14 28 20 0.033 1.000 0.24 0.66 0.49 0.80 0.58 0.43 0.72 0.57 0.42 0.72 0.67 0.50 0.80

Grey Bruce Health Unit SS 25 16 29 19 0.057 1.000 0.21 0.61 0.45 0.76 0.60 0.45 0.74 0.57 0.41 0.72 0.64 0.49 0.78

Simcoe Muskoka District Health Unit FOS 21 24 24 20 1.000 1.000 0.01 0.47 0.32 0.62 0.55 0.39 0.70 0.51 0.35 0.67 0.50 0.35 0.65

Simcoe Muskoka District Health Uni PCI 23 22 23 21 0.833 1.000 0.03 0.51 0.36 0.66 0.52 0.37 0.68 0.52 0.37 0.68 0.51 0.36 0.66

Simcoe Muskoka District Health Uni SS 23 22 22 22 1.000 1.000 0.01 0.51 0.36 0.66 0.50 0.35 0.65 0.51 0.36 0.66 0.50 0.35 0.65

Niagara Regional Area Health Unit FOS 26 14 33 16 0.003 0.319 0.32 0.65 0.48 0.79 0.67 0.52 0.80 0.62 0.46 0.76 0.70 0.55 0.83

Niagara Regional Area Health Unit PCI 28 12 29 20 0.010 1.000 0.29 0.70 0.53 0.83 0.59 0.44 0.73 0.58 0.43 0.72 0.71 0.54 0.84

Niagara Regional Area Health Unit SS 19 21 26 23 1.000 1.000 0.01 0.48 0.32 0.64 0.53 0.38 0.67 0.45 0.30 0.61 0.55 0.40 0.70

Peel Regional Health Unit FOS 32 12 34 11 0.000 0.001 0.48 0.73 0.57 0.85 0.76 0.60 0.87 0.74 0.59 0.86 0.74 0.59 0.86

Peel Regional Health Unit PCI 34 10 32 13 0.000 0.001 0.48 0.77 0.62 0.89 0.71 0.56 0.84 0.72 0.57 0.84 0.76 0.61 0.88

Peel Regional Health Unit SS 32 12 32 13 0.000 0.005 0.44 0.73 0.57 0.85 0.71 0.56 0.84 0.71 0.56 0.84 0.73 0.57 0.85

City of Ottawa Health Unit FOS 27 14 32 16 0.003 0.316 0.32 0.66 0.49 0.80 0.67 0.52 0.80 0.63 0.47 0.77 0.70 0.54 0.82

City of Ottawa Health Unit PCI 27 14 29 19 0.019 1.000 0.26 0.66 0.49 0.80 0.60 0.45 0.74 0.59 0.43 0.73 0.67 0.51 0.81

City of Ottawa Health Unit SS 23 18 28 20 0.205 1.000 0.14 0.56 0.40 0.72 0.58 0.43 0.72 0.53 0.38 0.69 0.61 0.45 0.75

City of Hamilton Health Unit FOS 25 17 30 17 0.035 1.000 0.23 0.60 0.43 0.74 0.64 0.49 0.77 0.60 0.43 0.74 0.64 0.49 0.77

City of Hamilton Health Unit PCI 26 16 28 19 0.057 1.000 0.21 0.62 0.46 0.76 0.60 0.44 0.74 0.58 0.42 0.72 0.64 0.48 0.78

City of Hamilton Health Unit SS 26 16 31 16 0.011 1.000 0.28 0.62 0.46 0.76 0.66 0.51 0.79 0.62 0.46 0.76 0.66 0.51 0.79

York Regional Health Unit FOS 34 9 36 10 0.000 0.000 0.57 0.79 0.64 0.90 0.78 0.64 0.89 0.77 0.62 0.89 0.80 0.65 0.90

York Regional Health Unit PCI 35 8 36 10 0.000 0.000 0.60 0.81 0.67 0.92 0.78 0.64 0.89 0.78 0.63 0.89 0.82 0.67 0.92

York Regional Health Unit SS 36 7 35 11 0.000 0.000 0.60 0.84 0.69 0.93 0.76 0.61 0.87 0.77 0.62 0.88 0.83 0.69 0.93

FOS 22 25 25 17 0.669 1.000 0.06 0.47 0.32 0.62 0.60 0.43 0.74 0.56 0.40 0.72 0.50 0.36 0.64

PCI 25 22 21 21 0.833 1.000 0.03 0.53 0.38 0.68 0.50 0.34 0.66 0.54 0.39 0.69 0.49 0.33 0.65

Leeds, Grenville and Lanark Distri SS 21 26 25 17 0.830 1.000 0.04 0.45 0.30 0.60 0.60 0.43 0.74 0.55 0.38 0.71 0.49 0.35 0.63

Middlesex-London Health Unit FOS 30 18 25 16 0.034 1.000 0.23 0.63 0.47 0.76 0.61 0.45 0.76 0.65 0.50 0.79 0.58 0.42 0.73

Middlesex-London Health Unit PCI 27 21 26 15 0.088 1.000 0.20 0.56 0.41 0.71 0.63 0.47 0.78 0.64 0.48 0.78 0.55 0.40 0.70

Middlesex-London Health Unit SS 31 17 25 16 0.020 1.000 0.26 0.65 0.49 0.78 0.61 0.45 0.76 0.66 0.51 0.79 0.60 0.43 0.74

Durham Regional Health Unit FOS 30 13 34 12 0.000 0.005 0.44 0.70 0.54 0.83 0.74 0.59 0.86 0.71 0.55 0.84 0.72 0.57 0.84

Durham Regional Health Unit PCI 32 11 33 13 0.000 0.002 0.46 0.74 0.59 0.86 0.72 0.57 0.84 0.71 0.56 0.84 0.75 0.60 0.87

Durham Regional Health Unit SS 30 13 31 15 0.001 0.070 0.37 0.70 0.54 0.83 0.67 0.52 0.80 0.67 0.51 0.80 0.70 0.55 0.83

The Eastern Ontario Health Unit FOS 28 18 29 14 0.011 1.000 0.28 0.61 0.45 0.75 0.67 0.51 0.81 0.67 0.50 0.80 0.62 0.46 0.75

The Eastern Ontario Health Unit PCI 28 18 31 12 0.003 0.291 0.33 0.61 0.45 0.75 0.72 0.56 0.85 0.70 0.53 0.83 0.63 0.48 0.77

The Eastern Ontario Health Unit SS 28 18 27 16 0.034 1.000 0.24 0.61 0.45 0.75 0.63 0.47 0.77 0.64 0.48 0.78 0.60 0.44 0.74


Health Unit

Health Unit Forecasting

Method

Fisher's

Exact Test

p-Value


for Flagging

Increases

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)

148

Table B.5 (Continued): Ability to Discriminate Between Increases and Decreases in the Aggregate Number of Hospital Emergency Department Visits

over the Next Four-Days

TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Peterborough County-City Health Unit FOS 26 17 26 20 0.139 1.000 0.17 0.60 0.44 0.75 0.57 0.41 0.71 0.57 0.41 0.71 0.60 0.44 0.75

Peterborough County-City Health Un PCI 21 22 23 23 1.000 1.000 -0.01 0.49 0.33 0.65 0.50 0.35 0.65 0.48 0.32 0.63 0.51 0.36 0.66

Peterborough County-City Health Un SS 25 18 24 22 0.397 1.000 0.10 0.58 0.42 0.73 0.52 0.37 0.67 0.53 0.38 0.68 0.57 0.41 0.72

FOS 20 22 18 29 0.206 1.000 -0.14 0.48 0.32 0.64 0.38 0.25 0.54 0.41 0.27 0.56 0.45 0.29 0.62

PCI 22 20 25 22 0.673 1.000 0.06 0.52 0.36 0.68 0.53 0.38 0.68 0.50 0.35 0.65 0.56 0.40 0.70

Hastings and Prince Edward Countie SS 22 20 21 26 0.833 1.000 -0.03 0.52 0.36 0.68 0.45 0.30 0.60 0.46 0.31 0.61 0.51 0.35 0.67

Waterloo Health Unit FOS 33 14 31 11 0.000 0.005 0.44 0.70 0.55 0.83 0.74 0.58 0.86 0.75 0.60 0.87 0.69 0.53 0.82

Waterloo Health Unit PCI 28 19 24 18 0.140 1.000 0.17 0.60 0.44 0.74 0.57 0.41 0.72 0.61 0.45 0.75 0.56 0.40 0.71

Waterloo Health Unit SS 26 21 27 15 0.088 1.000 0.20 0.55 0.40 0.70 0.64 0.48 0.78 0.63 0.47 0.78 0.56 0.41 0.71

The District of Algoma Health Unit FOS 17 25 29 18 1.000 1.000 0.02 0.40 0.26 0.57 0.62 0.46 0.75 0.49 0.31 0.66 0.54 0.40 0.67

The District of Algoma Health Unit PCI 21 21 19 28 0.399 1.000 -0.10 0.50 0.34 0.66 0.40 0.26 0.56 0.43 0.29 0.58 0.48 0.32 0.64

The District of Algoma Health Unit SS 21 21 25 22 0.833 1.000 0.03 0.50 0.34 0.66 0.53 0.38 0.68 0.49 0.33 0.65 0.54 0.39 0.69

FOS 31 13 32 13 0.000 0.013 0.42 0.70 0.55 0.83 0.71 0.56 0.84 0.70 0.55 0.83 0.71 0.56 0.84

PCI 25 19 27 18 0.140 1.000 0.17 0.57 0.41 0.72 0.60 0.44 0.74 0.58 0.42 0.73 0.59 0.43 0.73

Renfrew County and District Health SS 26 18 22 23 0.525 1.000 0.08 0.59 0.43 0.74 0.49 0.34 0.64 0.53 0.38 0.67 0.55 0.38 0.71

Thunder Bay District Health Unit FOS* 25 14 30 20 0.033 1.000 0.24 0.64 0.47 0.79 0.60 0.45 0.74 0.56 0.40 0.70 0.68 0.52 0.81

Thunder Bay District Health Unit PCI 24 15 32 18 0.020 1.000 0.25 0.62 0.45 0.77 0.64 0.49 0.77 0.57 0.41 0.72 0.68 0.53 0.81

Thunder Bay District Health Unit SS 21 18 26 24 0.671 1.000 0.06 0.54 0.37 0.70 0.52 0.37 0.66 0.47 0.32 0.62 0.59 0.43 0.74

Porcupine Health Unit FOS 18 22 30 19 0.666 1.000 0.06 0.45 0.29 0.62 0.61 0.46 0.75 0.49 0.32 0.66 0.58 0.43 0.71

Porcupine Health Unit PCI 21 19 27 22 0.527 1.000 0.08 0.53 0.36 0.68 0.55 0.40 0.69 0.49 0.33 0.65 0.59 0.43 0.73

Porcupine Health Unit SS 20 20 27 22 0.674 1.000 0.05 0.50 0.34 0.66 0.55 0.40 0.69 0.48 0.32 0.64 0.57 0.42 0.72

FOS 27 23 24 15 0.199 1.000 0.15 0.54 0.39 0.68 0.62 0.45 0.77 0.64 0.48 0.78 0.51 0.36 0.66

PCI 32 18 23 16 0.035 1.000 0.23 0.64 0.49 0.77 0.59 0.42 0.74 0.67 0.52 0.80 0.56 0.40 0.72

Haliburton, Kawartha, Pine Ridge D SS 28 22 19 20 0.675 1.000 0.05 0.56 0.41 0.70 0.49 0.32 0.65 0.58 0.43 0.72 0.46 0.31 0.63

FOS 26 19 26 18 0.140 1.000 0.17 0.58 0.42 0.72 0.59 0.43 0.74 0.59 0.43 0.74 0.58 0.42 0.72

PCI 23 22 21 23 1.000 1.000 -0.01 0.51 0.36 0.66 0.48 0.32 0.63 0.50 0.35 0.65 0.49 0.33 0.65

North Bay Parry Sound District Hea SS 21 24 25 19 0.832 1.000 0.04 0.47 0.32 0.62 0.57 0.41 0.72 0.53 0.36 0.68 0.51 0.36 0.66

Oxford County Health Unit FOS 28 13 30 18 0.006 0.596 0.31 0.68 0.52 0.82 0.63 0.47 0.76 0.61 0.45 0.75 0.70 0.54 0.83

Oxford County Health Unit PCI 25 16 25 23 0.287 1.000 0.13 0.61 0.45 0.76 0.52 0.37 0.67 0.52 0.37 0.67 0.61 0.45 0.76

Oxford County Health Unit SS 28 13 31 17 0.003 0.305 0.33 0.68 0.52 0.82 0.65 0.49 0.78 0.62 0.47 0.76 0.70 0.55 0.83

Lambton Health Unit FOS 26 23 27 13 0.058 1.000 0.21 0.53 0.38 0.67 0.68 0.51 0.81 0.67 0.50 0.81 0.54 0.39 0.68

Lambton Health Unit PCI 31 18 22 18 0.093 1.000 0.18 0.63 0.48 0.77 0.55 0.38 0.71 0.63 0.48 0.77 0.55 0.38 0.71

Lambton Health Unit SS 27 22 23 17 0.289 1.000 0.13 0.55 0.40 0.69 0.58 0.41 0.73 0.61 0.45 0.76 0.51 0.36 0.66

Chatham-Kent Health Unit FOS 21 21 31 16 0.139 1.000 0.16 0.50 0.34 0.66 0.66 0.51 0.79 0.57 0.39 0.73 0.60 0.45 0.73

Chatham-Kent Health Unit PCI 20 22 20 27 0.399 1.000 -0.10 0.48 0.32 0.64 0.43 0.28 0.58 0.43 0.28 0.58 0.48 0.32 0.64

Chatham-Kent Health Unit SS 21 21 26 21 0.674 1.000 0.05 0.50 0.34 0.66 0.55 0.40 0.70 0.50 0.34 0.66 0.55 0.40 0.70


Health Unit

Renfrew County and District Health

Unit




Unit

Positive Predictive Value Negative Predictive ValueHealth Unit Forecasting

Method

Confusion Matrix

for Flagging

Increases

Fisher's

Exact Test

p-Value

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)


149

Table B.5 (Continued): Ability to Discriminate Between Increases and Decreases in the Aggregate Number of Hospital Emergency Department Visits

over the Next Four-Days

TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Haldimand-Norfolk Health Unit FOS 20 18 22 29 0.830 1.000 -0.04 0.53 0.36 0.69 0.43 0.29 0.58 0.41 0.27 0.56 0.55 0.38 0.71

Haldimand-Norfolk Health Unit PCI 26 12 26 25 0.085 1.000 0.19 0.68 0.51 0.82 0.51 0.37 0.65 0.51 0.37 0.65 0.68 0.51 0.82

Haldimand-Norfolk Health Unit SS 19 19 26 25 1.000 1.000 0.01 0.50 0.33 0.67 0.51 0.37 0.65 0.43 0.28 0.59 0.58 0.42 0.72

Halton Regional Health Unit FOS 31 8 36 14 0.000 0.000 0.51 0.79 0.64 0.91 0.72 0.58 0.84 0.69 0.53 0.82 0.82 0.67 0.92

Halton Regional Health Unit PCI 29 10 35 15 0.000 0.005 0.44 0.74 0.58 0.87 0.70 0.55 0.82 0.66 0.50 0.80 0.78 0.63 0.89

Halton Regional Health Unit SS 28 11 34 16 0.000 0.028 0.39 0.72 0.55 0.85 0.68 0.53 0.80 0.64 0.48 0.78 0.76 0.60 0.87

Windsor-Essex County Health Unit FOS 33 10 34 12 0.000 0.000 0.51 0.77 0.61 0.88 0.74 0.59 0.86 0.73 0.58 0.85 0.77 0.62 0.89

Windsor-Essex County Health Unit PCI 34 9 33 13 0.000 0.000 0.51 0.79 0.64 0.90 0.72 0.57 0.84 0.72 0.57 0.84 0.79 0.63 0.90

Windsor-Essex County Health Unit SS 31 12 33 13 0.000 0.005 0.44 0.72 0.56 0.85 0.72 0.57 0.84 0.70 0.55 0.83 0.73 0.58 0.85

Northwestern Health Unit FOS* 24 17 24 24 0.523 1.000 0.09 0.59 0.42 0.74 0.50 0.35 0.65 0.50 0.35 0.65 0.59 0.42 0.74

Northwestern Health Unit PCI 21 20 19 29 0.401 1.000 -0.09 0.51 0.35 0.67 0.40 0.26 0.55 0.42 0.28 0.57 0.49 0.32 0.65

Northwestern Health Unit SS 19 22 23 25 0.672 1.000 -0.06 0.46 0.31 0.63 0.48 0.33 0.63 0.43 0.28 0.59 0.51 0.36 0.66

FOS 26 17 29 17 0.034 1.000 0.24 0.60 0.44 0.75 0.63 0.48 0.77 0.60 0.44 0.75 0.63 0.48 0.77

PCI 24 19 26 20 0.292 1.000 0.12 0.56 0.40 0.71 0.57 0.41 0.71 0.55 0.39 0.70 0.58 0.42 0.72

Kingston, Frontenac and Lennox and SS 23 20 26 20 0.399 1.000 0.10 0.53 0.38 0.69 0.57 0.41 0.71 0.53 0.38 0.69 0.57 0.41 0.71

Sudbury and District Health Unit FOS 25 21 24 19 0.399 1.000 0.10 0.54 0.39 0.69 0.56 0.40 0.71 0.57 0.41 0.72 0.53 0.38 0.68

Sudbury and District Health Unit PCI 27 19 24 19 0.206 1.000 0.15 0.59 0.43 0.73 0.56 0.40 0.71 0.59 0.43 0.73 0.56 0.40 0.71

Sudbury and District Health Unit SS 19 27 18 25 0.140 1.000 -0.17 0.41 0.27 0.57 0.42 0.27 0.58 0.43 0.28 0.59 0.40 0.26 0.56

Huron County Health Unit FOS 24 20 27 18 0.205 1.000 0.15 0.55 0.39 0.70 0.60 0.44 0.74 0.57 0.41 0.72 0.57 0.42 0.72

Huron County Health Unit PCI 24 20 20 25 1.000 1.000 -0.01 0.55 0.39 0.70 0.44 0.30 0.60 0.49 0.34 0.64 0.50 0.34 0.66

Huron County Health Unit SS 23 21 26 19 0.399 1.000 0.10 0.52 0.37 0.68 0.58 0.42 0.72 0.55 0.39 0.70 0.55 0.40 0.70

Wellington-Dufferin-Guelph Health Unit FOS 24 20 26 19 0.292 1.000 0.12 0.55 0.39 0.70 0.58 0.42 0.72 0.56 0.40 0.71 0.57 0.41 0.71

Wellington-Dufferin-Guelph Health PCI 23 21 26 19 0.399 1.000 0.10 0.52 0.37 0.68 0.58 0.42 0.72 0.55 0.39 0.70 0.55 0.40 0.70

Wellington-Dufferin-Guelph Health SS 23 21 24 21 0.674 1.000 0.06 0.52 0.37 0.68 0.53 0.38 0.68 0.52 0.37 0.68 0.53 0.38 0.68

Perth District Health Unit FOS 30 15 30 14 0.001 0.153 0.35 0.67 0.51 0.80 0.68 0.52 0.81 0.68 0.52 0.81 0.67 0.51 0.80

Perth District Health Unit PCI 25 20 27 17 0.139 1.000 0.17 0.56 0.40 0.70 0.61 0.45 0.76 0.60 0.43 0.74 0.57 0.42 0.72

Perth District Health Unit SS 19 26 24 20 0.832 1.000 -0.03 0.42 0.28 0.58 0.55 0.39 0.70 0.49 0.32 0.65 0.48 0.34 0.63

Brant County Health Unit FOS 21 19 26 23 0.672 1.000 0.06 0.53 0.36 0.68 0.53 0.38 0.67 0.48 0.32 0.63 0.58 0.42 0.72

Brant County Health Unit PCI 18 22 19 30 0.141 1.000 -0.16 0.45 0.29 0.62 0.39 0.25 0.54 0.38 0.24 0.53 0.46 0.31 0.63

Brant County Health Unit SS 20 20 23 26 0.833 1.000 -0.03 0.50 0.34 0.66 0.47 0.33 0.62 0.43 0.29 0.59 0.53 0.38 0.69

Timiskaming Health Unit FOS 20 22 25 22 1.000 1.000 0.01 0.48 0.32 0.64 0.53 0.38 0.68 0.48 0.32 0.64 0.53 0.38 0.68

Timiskaming Health Unit PCI 20 22 21 26 0.527 1.000 -0.08 0.48 0.32 0.64 0.45 0.30 0.60 0.43 0.29 0.59 0.49 0.33 0.65

Timiskaming Health Unit SS 20 22 21 26 0.527 1.000 -0.08 0.48 0.32 0.64 0.45 0.30 0.60 0.43 0.29 0.59 0.49 0.33 0.65

Elgin-St. Thomas Health Unit FOS 29 14 24 22 0.086 1.000 0.20 0.67 0.51 0.81 0.52 0.37 0.67 0.57 0.42 0.71 0.63 0.46 0.78

Elgin-St. Thomas Health Unit PCI 24 19 27 19 0.206 1.000 0.15 0.56 0.40 0.71 0.59 0.43 0.73 0.56 0.40 0.71 0.59 0.43 0.73

Elgin-St. Thomas Health Unit SS 27 16 28 18 0.034 1.000 0.24 0.63 0.47 0.77 0.61 0.45 0.75 0.60 0.44 0.74 0.64 0.48 0.78




Method

Confusion Matrix

for Flagging

Increases

Fisher's

Exact Test

p-Value

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)


150

Table B.6: Ability of Telehealth Ontario Calls to Directly Discriminate Between Increases and

Decreases in the Aggregate Number of Hospital Emergency Department Visits Over the Next Four

Days

Health Unit AUROC Lower

95% CI

Upper 95%

CI

City of Toronto Health Unit 0.30 0.19 0.42

Grey Bruce Health Unit 0.58 0.46 0.70

Simcoe Muskoka District Health Unit 0.46 0.34 0.58

Niagara Regional Area Health Unit 0.48 0.36 0.61

Peel Regional Health Unit 0.56 0.44 0.68

City of Ottawa Health Unit 0.46 0.34 0.58

City of Hamilton Health Unit 0.48 0.36 0.61

York Regional Health Unit 0.30 0.19 0.41

Leeds, Grenville and Lanark District Health Unit 0.50 0.38 0.62

Middlesex-London Health Unit 0.45 0.33 0.58

Durham Regional Health Unit 0.34 0.23 0.46

The Eastern Ontario Health Unit 0.62 0.51 0.74

Peterborough County-City Health Unit 0.42 0.30 0.54

Hastings and Prince Edward Counties Health Unit 0.58 0.46 0.70

Waterloo Health Unit 0.35 0.23 0.46

The District of Algoma Health Unit 0.60 0.48 0.72

Renfrew County and District Health Unit 0.59 0.48 0.71

Thunder Bay District Health Unit 0.46 0.33 0.58

Porcupine Health Unit 0.44 0.32 0.57

Haliburton, Kawartha, Pine Ridge District Health Unit 0.47 0.35 0.59

North Bay Parry Sound District Health Unit 0.43 0.31 0.55

Oxford County Health Unit 0.50 0.38 0.62

Lambton Health Unit 0.57 0.45 0.69

Chatham-Kent Health Unit 0.57 0.45 0.69

Haldimand-Norfolk Health Unit 0.49 0.37 0.61

Halton Regional Health Unit 0.46 0.34 0.59

Windsor-Essex County Health Unit 0.46 0.33 0.58

Northwestern Health Unit 0.51 0.38 0.63

Kingston, Frontenac and Lennox and Addington Health Unit 0.52 0.40 0.64

Sudbury and District Health Unit 0.62 0.50 0.73

Huron County Health Unit 0.67 0.55 0.78

Wellington-Dufferin-Guelph Health Unit 0.57 0.44 0.69

Perth District Health Unit 0.40 0.28 0.52

Brant County Health Unit 0.50 0.38 0.62

Timiskaming Health Unit 0.48 0.36 0.61

Elgin-St. Thomas Health Unit 0.57 0.44 0.69

151

Table B.7: Ability to Predict a Nominal 10% Increase in the Aggregate Number of Hospital Emergency Department Visits over the Next Four Days

TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

City of Toronto Health Unit FOS 15 13 52 9 0.000 0.027 0.41 0.54 0.34 0.72 0.85 0.74 0.93 0.63 0.41 0.81 0.80 0.68 0.89

City of Toronto Health Unit PCI 16 12 46 15 0.004 0.436 0.32 0.57 0.37 0.76 0.75 0.63 0.86 0.52 0.33 0.70 0.79 0.67 0.89

City of Toronto Health Unit SS 18 10 45 16 0.001 0.100 0.36 0.64 0.44 0.81 0.74 0.61 0.84 0.53 0.35 0.70 0.82 0.69 0.91

Grey Bruce Health Unit FOS 4 19 62 4 0.197 1.000 0.17 0.17 0.05 0.39 0.94 0.85 0.98 0.50 0.16 0.84 0.77 0.66 0.85

Grey Bruce Health Unit PCI 8 15 46 20 0.795 1.000 0.04 0.35 0.16 0.57 0.70 0.57 0.80 0.29 0.13 0.49 0.75 0.63 0.86

Grey Bruce Health Unit SS 5 18 49 17 0.786 1.000 -0.04 0.22 0.07 0.44 0.74 0.62 0.84 0.23 0.08 0.45 0.73 0.61 0.83

Simcoe Muskoka District Health Unit FOS 5 13 60 11 0.301 1.000 0.13 0.28 0.10 0.53 0.85 0.74 0.92 0.31 0.11 0.59 0.82 0.71 0.90

Simcoe Muskoka District Health Uni PCI 5 13 57 14 0.522 1.000 0.08 0.28 0.10 0.53 0.80 0.69 0.89 0.26 0.09 0.51 0.81 0.70 0.90

Simcoe Muskoka District Health Uni SS 3 15 49 22 0.378 1.000 -0.13 0.17 0.04 0.41 0.69 0.57 0.79 0.12 0.03 0.31 0.77 0.64 0.86

Niagara Regional Area Health Unit FOS 10 13 57 9 0.006 0.654 0.32 0.43 0.23 0.66 0.86 0.76 0.94 0.53 0.29 0.76 0.81 0.70 0.90

Niagara Regional Area Health Unit PCI 8 15 52 14 0.261 1.000 0.14 0.35 0.16 0.57 0.79 0.67 0.88 0.36 0.17 0.59 0.78 0.66 0.87

Niagara Regional Area Health Unit SS 7 16 44 22 1.000 1.000 -0.03 0.30 0.13 0.53 0.67 0.54 0.78 0.24 0.10 0.44 0.73 0.60 0.84

Peel Regional Health Unit FOS 19 8 48 14 0.000 0.003 0.45 0.70 0.50 0.86 0.77 0.65 0.87 0.58 0.39 0.75 0.86 0.74 0.94

Peel Regional Health Unit PCI 20 7 48 14 0.000 0.001 0.49 0.74 0.54 0.89 0.77 0.65 0.87 0.59 0.41 0.75 0.87 0.76 0.95

Peel Regional Health Unit SS 15 12 45 17 0.016 1.000 0.27 0.56 0.35 0.75 0.73 0.60 0.83 0.47 0.29 0.65 0.79 0.66 0.89

City of Ottawa Health Unit FOS 12 11 53 13 0.006 0.640 0.32 0.52 0.31 0.73 0.80 0.69 0.89 0.48 0.28 0.69 0.83 0.71 0.91

City of Ottawa Health Unit PCI 11 12 53 13 0.014 1.000 0.28 0.48 0.27 0.69 0.80 0.69 0.89 0.46 0.26 0.67 0.82 0.70 0.90

City of Ottawa Health Unit SS 10 13 48 18 0.194 1.000 0.15 0.43 0.23 0.66 0.73 0.60 0.83 0.36 0.19 0.56 0.79 0.66 0.88

City of Hamilton Health Unit FOS 6 20 52 11 0.562 1.000 0.06 0.23 0.09 0.44 0.83 0.71 0.91 0.35 0.14 0.62 0.72 0.60 0.82

City of Hamilton Health Unit PCI 13 13 50 13 0.010 1.000 0.29 0.50 0.30 0.70 0.79 0.67 0.89 0.50 0.30 0.70 0.79 0.67 0.89

City of Hamilton Health Unit SS 14 12 47 16 0.014 1.000 0.27 0.54 0.33 0.73 0.75 0.62 0.85 0.47 0.28 0.66 0.80 0.67 0.89

York Regional Health Unit FOS 20 14 45 10 0.000 0.017 0.42 0.59 0.41 0.75 0.82 0.69 0.91 0.67 0.47 0.83 0.76 0.63 0.86

York Regional Health Unit PCI 20 14 43 12 0.001 0.066 0.37 0.59 0.41 0.75 0.78 0.65 0.88 0.63 0.44 0.79 0.75 0.62 0.86

York Regional Health Unit SS 18 16 42 13 0.006 0.683 0.30 0.53 0.35 0.70 0.76 0.63 0.87 0.58 0.39 0.75 0.72 0.59 0.83

FOS 0 27 58 4 0.310 1.000 -0.14 0.00 0.00 0.13 0.94 0.84 0.98 0.00 0.00 0.60 0.68 0.57 0.78

PCI 6 21 46 16 0.795 1.000 -0.04 0.22 0.09 0.42 0.74 0.62 0.84 0.27 0.11 0.50 0.69 0.56 0.79

Leeds, Grenville and Lanark Distri SS 6 21 45 17 0.793 1.000 -0.05 0.22 0.09 0.42 0.73 0.60 0.83 0.26 0.10 0.48 0.68 0.56 0.79

Middlesex-London Health Unit FOS 7 20 56 6 0.057 1.000 0.21 0.26 0.11 0.46 0.90 0.80 0.96 0.54 0.25 0.81 0.74 0.62 0.83

Middlesex-London Health Unit PCI 12 15 49 13 0.039 1.000 0.24 0.44 0.25 0.65 0.79 0.67 0.88 0.48 0.28 0.69 0.77 0.64 0.86

Middlesex-London Health Unit SS 12 15 47 15 0.079 1.000 0.20 0.44 0.25 0.65 0.76 0.63 0.86 0.44 0.25 0.65 0.76 0.63 0.86

Durham Regional Health Unit FOS 12 20 50 7 0.008 0.811 0.30 0.38 0.21 0.56 0.88 0.76 0.95 0.63 0.38 0.84 0.71 0.59 0.82

Durham Regional Health Unit PCI 21 11 46 11 0.000 0.002 0.46 0.66 0.47 0.81 0.81 0.68 0.90 0.66 0.47 0.81 0.81 0.68 0.90

Durham Regional Health Unit SS 22 10 43 14 0.000 0.010 0.43 0.69 0.50 0.84 0.75 0.62 0.86 0.61 0.43 0.77 0.81 0.68 0.91

The Eastern Ontario Health Unit FOS 11 13 55 10 0.005 0.504 0.32 0.46 0.26 0.67 0.85 0.74 0.92 0.52 0.30 0.74 0.81 0.70 0.89

The Eastern Ontario Health Unit PCI 12 12 48 17 0.043 1.000 0.23 0.50 0.29 0.71 0.74 0.61 0.84 0.41 0.24 0.61 0.80 0.68 0.89

The Eastern Ontario Health Unit SS 12 12 47 18 0.076 1.000 0.21 0.50 0.29 0.71 0.72 0.60 0.83 0.40 0.23 0.59 0.80 0.67 0.89

Positive Predictive Value Negative Predictive ValueSensitivityConfusion Matrix

for Flagging

Increases

Adjusted p-

Value

Matthew's

Correlation

Coefficient

(MCC)


Method

Fisher's

Exact Test

p-Value

Specificity


Health Unit

152

Table B.7 (Continued): Ability to Predict a Nominal 10% Increase in the Aggregate Number of Hospital Emergency Department Visits over the Next

Four Days

TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Peterborough County-City Health Unit FOS 7 26 49 7 0.367 1.000 0.12 0.21 0.09 0.39 0.88 0.76 0.95 0.50 0.23 0.77 0.65 0.53 0.76

Peterborough County-City Health Un PCI 6 27 35 21 0.061 1.000 -0.20 0.18 0.07 0.35 0.63 0.49 0.75 0.22 0.09 0.42 0.56 0.43 0.69

Peterborough County-City Health Un SS 16 17 37 19 0.187 1.000 0.14 0.48 0.31 0.66 0.66 0.52 0.78 0.46 0.29 0.63 0.69 0.54 0.80

FOS 5 21 55 8 0.512 1.000 0.08 0.19 0.07 0.39 0.87 0.77 0.94 0.38 0.14 0.68 0.72 0.61 0.82

PCI 7 19 47 16 1.000 1.000 0.02 0.27 0.12 0.48 0.75 0.62 0.85 0.30 0.13 0.53 0.71 0.59 0.82

Hastings and Prince Edward Countie SS 9 17 39 24 0.813 1.000 -0.03 0.35 0.17 0.56 0.62 0.49 0.74 0.27 0.13 0.46 0.70 0.56 0.81

Waterloo Health Unit FOS 12 21 51 5 0.002 0.241 0.34 0.36 0.20 0.55 0.91 0.80 0.97 0.71 0.44 0.90 0.71 0.59 0.81

Waterloo Health Unit PCI 16 17 44 12 0.010 1.000 0.28 0.48 0.31 0.66 0.79 0.66 0.88 0.57 0.37 0.76 0.72 0.59 0.83

Waterloo Health Unit SS 13 20 43 13 0.147 1.000 0.17 0.39 0.23 0.58 0.77 0.64 0.87 0.50 0.30 0.70 0.68 0.55 0.79

The District of Algoma Health Unit FOS 1 27 57 4 1.000 1.000 -0.06 0.04 0.00 0.18 0.93 0.84 0.98 0.20 0.01 0.72 0.68 0.57 0.78

The District of Algoma Health Unit PCI 4 24 43 18 0.185 1.000 -0.16 0.14 0.04 0.33 0.70 0.57 0.81 0.18 0.05 0.40 0.64 0.52 0.76

The District of Algoma Health Unit SS 6 22 44 17 0.608 1.000 -0.07 0.21 0.08 0.41 0.72 0.59 0.83 0.26 0.10 0.48 0.67 0.54 0.78

Renfrew County and District Health UnitFOS 7 24 55 3 0.029 1.000 0.26 0.23 0.10 0.41 0.95 0.86 0.99 0.70 0.35 0.93 0.70 0.58 0.79

Renfrew County and District Health PCI 8 23 44 14 1.000 1.000 0.02 0.26 0.12 0.45 0.76 0.63 0.86 0.36 0.17 0.59 0.66 0.53 0.77

Renfrew County and District Health SS 7 24 44 14 1.000 1.000 -0.02 0.23 0.10 0.41 0.76 0.63 0.86 0.33 0.15 0.57 0.65 0.52 0.76

Thunder Bay District Health Unit FOS* 11 19 48 11 0.074 1.000 0.20 0.37 0.20 0.56 0.81 0.69 0.90 0.50 0.28 0.72 0.72 0.59 0.82

Thunder Bay District Health Unit PCI 16 14 43 16 0.020 1.000 0.26 0.53 0.34 0.72 0.73 0.60 0.84 0.50 0.32 0.68 0.75 0.62 0.86

Thunder Bay District Health Unit SS 15 15 40 19 0.113 1.000 0.17 0.50 0.31 0.69 0.68 0.54 0.79 0.44 0.27 0.62 0.73 0.59 0.84

Porcupine Health Unit FOS 9 23 50 7 0.085 1.000 0.20 0.28 0.14 0.47 0.88 0.76 0.95 0.56 0.30 0.80 0.68 0.57 0.79

Porcupine Health Unit PCI 13 19 44 13 0.092 1.000 0.19 0.41 0.24 0.59 0.77 0.64 0.87 0.50 0.30 0.70 0.70 0.57 0.81

Porcupine Health Unit SS 12 20 37 20 0.822 1.000 0.02 0.38 0.21 0.56 0.65 0.51 0.77 0.38 0.21 0.56 0.65 0.51 0.77

FOS 4 32 47 6 1.000 1.000 0.00 0.11 0.03 0.26 0.89 0.77 0.96 0.40 0.12 0.74 0.59 0.48 0.70

PCI 13 23 42 11 0.145 1.000 0.17 0.36 0.21 0.54 0.79 0.66 0.89 0.54 0.33 0.74 0.65 0.52 0.76

Haliburton, Kawartha, Pine Ridge D SS 8 28 48 5 0.128 1.000 0.18 0.22 0.10 0.39 0.91 0.79 0.97 0.62 0.32 0.86 0.63 0.51 0.74

FOS 6 26 55 2 0.023 1.000 0.26 0.19 0.07 0.36 0.96 0.88 1.00 0.75 0.35 0.97 0.68 0.57 0.78

PCI 7 25 38 19 0.333 1.000 -0.12 0.22 0.09 0.40 0.67 0.53 0.79 0.27 0.12 0.48 0.60 0.47 0.72

North Bay Parry Sound District Hea SS 8 24 45 12 0.792 1.000 0.05 0.25 0.11 0.43 0.79 0.66 0.89 0.40 0.19 0.64 0.65 0.53 0.76

Oxford County Health Unit FOS 12 20 51 6 0.005 0.521 0.32 0.38 0.21 0.56 0.89 0.78 0.96 0.67 0.41 0.87 0.72 0.60 0.82

Oxford County Health Unit PCI 13 19 38 19 0.500 1.000 0.07 0.41 0.24 0.59 0.67 0.53 0.79 0.41 0.24 0.59 0.67 0.53 0.79

Oxford County Health Unit SS 13 19 41 16 0.247 1.000 0.13 0.41 0.24 0.59 0.72 0.58 0.83 0.45 0.26 0.64 0.68 0.55 0.80

Lambton Health Unit FOS 2 31 51 5 1.000 1.000 -0.05 0.06 0.01 0.20 0.91 0.80 0.97 0.29 0.04 0.71 0.62 0.51 0.73

Lambton Health Unit PCI 13 20 42 14 0.163 1.000 0.15 0.39 0.23 0.58 0.75 0.62 0.86 0.48 0.29 0.68 0.68 0.55 0.79

Lambton Health Unit SS 11 22 40 16 0.642 1.000 0.05 0.33 0.18 0.52 0.71 0.58 0.83 0.41 0.22 0.61 0.65 0.51 0.76

Chatham-Kent Health Unit FOS 4 23 56 6 0.484 1.000 0.07 0.15 0.04 0.34 0.90 0.80 0.96 0.40 0.12 0.74 0.71 0.60 0.81

Chatham-Kent Health Unit PCI 7 20 43 19 0.801 1.000 -0.05 0.26 0.11 0.46 0.69 0.56 0.80 0.27 0.12 0.48 0.68 0.55 0.79

Chatham-Kent Health Unit SS 10 17 50 12 0.108 1.000 0.19 0.37 0.19 0.58 0.81 0.69 0.90 0.45 0.24 0.68 0.75 0.63 0.84

Forecasting

Method

Confusion Matrix

for Flagging

Increases

Fisher's

Exact Test

p-Value

Adjusted p-

Value

Positive Predictive Value Negative Predictive ValueMatthew's

Correlation

Coefficient

(MCC)





Unit


Health Unit

Health Unit

153

Table B.7 (Continued): Ability to Predict a Nominal 10% Increase in the Aggregate Number of Hospital Emergency Department Visits over the Next

Four Days

TP FN TN FP Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Point

Estimate

Lower

95% CI

Upper

95% CI

Haldimand-Norfolk Health Unit FOS 7 24 44 14 1.000 1.000 -0.02 0.23 0.10 0.41 0.76 0.63 0.86 0.33 0.15 0.57 0.65 0.52 0.76

Haldimand-Norfolk Health Unit PCI 13 18 38 20 0.500 1.000 0.07 0.42 0.25 0.61 0.66 0.52 0.78 0.39 0.23 0.58 0.68 0.54 0.80

Haldimand-Norfolk Health Unit SS 8 23 42 16 1.000 1.000 -0.02 0.26 0.12 0.45 0.72 0.59 0.83 0.33 0.16 0.55 0.65 0.52 0.76

Halton Regional Health Unit FOS 16 10 49 14 0.001 0.062 0.38 0.62 0.41 0.80 0.78 0.66 0.87 0.53 0.34 0.72 0.83 0.71 0.92

Halton Regional Health Unit PCI 18 8 47 16 0.000 0.024 0.41 0.69 0.48 0.86 0.75 0.62 0.85 0.53 0.35 0.70 0.85 0.73 0.94

Halton Regional Health Unit SS 15 11 49 14 0.002 0.255 0.34 0.58 0.37 0.77 0.78 0.66 0.87 0.52 0.33 0.71 0.82 0.70 0.90

Windsor-Essex County Health Unit FOS 21 16 40 12 0.002 0.188 0.34 0.57 0.39 0.73 0.77 0.63 0.87 0.64 0.45 0.80 0.71 0.58 0.83

Windsor-Essex County Health Unit PCI 25 12 38 14 0.000 0.022 0.40 0.68 0.50 0.82 0.73 0.59 0.84 0.64 0.47 0.79 0.76 0.62 0.87

Windsor-Essex County Health Unit SS 21 16 39 13 0.004 0.408 0.32 0.57 0.39 0.73 0.75 0.61 0.86 0.62 0.44 0.78 0.71 0.57 0.82

Northwestern Health Unit FOS* 3 24 57 5 0.694 1.000 0.05 0.11 0.02 0.29 0.92 0.82 0.97 0.38 0.09 0.76 0.70 0.59 0.80

Northwestern Health Unit PCI 5 22 39 23 0.135 1.000 -0.18 0.19 0.06 0.38 0.63 0.50 0.75 0.18 0.06 0.37 0.64 0.51 0.76

Northwestern Health Unit SS 5 22 49 13 1.000 1.000 -0.03 0.19 0.06 0.38 0.79 0.67 0.88 0.28 0.10 0.53 0.69 0.57 0.79

FOS 12 19 51 7 0.006 0.639 0.31 0.39 0.22 0.58 0.88 0.77 0.95 0.63 0.38 0.84 0.73 0.61 0.83

PCI 12 19 42 16 0.340 1.000 0.11 0.39 0.22 0.58 0.72 0.59 0.83 0.43 0.24 0.63 0.69 0.56 0.80

Kingston, Frontenac and Lennox and SS 10 21 40 18 1.000 1.000 0.01 0.32 0.17 0.51 0.69 0.55 0.80 0.36 0.19 0.56 0.66 0.52 0.77

Sudbury and District Health Unit FOS 6 29 43 11 0.788 1.000 -0.04 0.17 0.07 0.34 0.80 0.66 0.89 0.35 0.14 0.62 0.60 0.47 0.71

Sudbury and District Health Unit PCI 8 27 43 11 0.796 1.000 0.03 0.23 0.10 0.40 0.80 0.66 0.89 0.42 0.20 0.67 0.61 0.49 0.73

Sudbury and District Health Unit SS 10 25 32 22 0.267 1.000 -0.12 0.29 0.15 0.46 0.59 0.45 0.72 0.31 0.16 0.50 0.56 0.42 0.69

Huron County Health Unit FOS 4 26 52 7 1.000 1.000 0.02 0.13 0.04 0.31 0.88 0.77 0.95 0.36 0.11 0.69 0.67 0.55 0.77

Huron County Health Unit PCI 9 21 39 20 0.813 1.000 -0.04 0.30 0.15 0.49 0.66 0.53 0.78 0.31 0.15 0.51 0.65 0.52 0.77

Huron County Health Unit SS 9 21 41 18 1.000 1.000 -0.01 0.30 0.15 0.49 0.69 0.56 0.81 0.33 0.17 0.54 0.66 0.53 0.78

Wellington-Dufferin-Guelph Health Unit FOS 9 26 42 12 0.800 1.000 0.04 0.26 0.12 0.43 0.78 0.64 0.88 0.43 0.22 0.66 0.62 0.49 0.73

Wellington-Dufferin-Guelph Health PCI 10 25 39 15 1.000 1.000 0.01 0.29 0.15 0.46 0.72 0.58 0.84 0.40 0.21 0.61 0.61 0.48 0.73

Wellington-Dufferin-Guelph Health SS 9 26 40 14 1.000 1.000 0.00 0.26 0.12 0.43 0.74 0.60 0.85 0.39 0.20 0.61 0.61 0.48 0.72

Perth District Health Unit FOS 18 20 39 12 0.024 1.000 0.25 0.47 0.31 0.64 0.76 0.63 0.87 0.60 0.41 0.77 0.66 0.53 0.78

Perth District Health Unit PCI 18 20 33 18 0.281 1.000 0.12 0.47 0.31 0.64 0.65 0.50 0.78 0.50 0.33 0.67 0.62 0.48 0.75

Perth District Health Unit SS 12 26 32 19 0.656 1.000 -0.06 0.32 0.18 0.49 0.63 0.48 0.76 0.39 0.22 0.58 0.55 0.42 0.68

Brant County Health Unit FOS 9 20 47 13 0.432 1.000 0.10 0.31 0.15 0.51 0.78 0.66 0.88 0.41 0.21 0.64 0.70 0.58 0.81

Brant County Health Unit PCI 5 24 53 7 0.516 1.000 0.08 0.17 0.06 0.36 0.88 0.77 0.95 0.42 0.15 0.72 0.69 0.57 0.79

Brant County Health Unit SS 11 18 45 15 0.224 1.000 0.13 0.38 0.21 0.58 0.75 0.62 0.85 0.42 0.23 0.63 0.71 0.59 0.82

Timiskaming Health Unit FOS 7 26 41 15 0.619 1.000 -0.06 0.21 0.09 0.39 0.73 0.60 0.84 0.32 0.14 0.55 0.61 0.49 0.73

Timiskaming Health Unit PCI 8 25 33 23 0.166 1.000 -0.17 0.24 0.11 0.42 0.59 0.45 0.72 0.26 0.12 0.45 0.57 0.43 0.70

Timiskaming Health Unit SS 7 26 48 8 0.399 1.000 0.09 0.21 0.09 0.39 0.86 0.74 0.94 0.47 0.21 0.73 0.65 0.53 0.76

Elgin-St. Thomas Health Unit FOS 8 28 40 13 1.000 1.000 -0.03 0.22 0.10 0.39 0.75 0.62 0.86 0.38 0.18 0.62 0.59 0.46 0.71

Elgin-St. Thomas Health Unit PCI 15 21 35 18 0.507 1.000 0.08 0.42 0.26 0.59 0.66 0.52 0.78 0.45 0.28 0.64 0.63 0.49 0.75

Elgin-St. Thomas Health Unit SS 17 19 36 17 0.185 1.000 0.15 0.47 0.30 0.65 0.68 0.54 0.80 0.50 0.32 0.68 0.65 0.51 0.78

Matthew's

Correlation

Coefficient

(MCC)

Sensitivity Specificity Positive Predictive Value Negative Predictive ValueForecasting

Method

Confusion Matrix

for Flagging

Increases

Fisher's

Exact Test

p-Value

Adjusted p-

Value



Health Unit

154

Table B.8: Ability of Telehealth Ontario Calls to Directly Discriminate Between Increases in the

Aggregate Number of Hospital Emergency Department Visits Above a Nominal 10% Increase One

Week Ahead Health Unit AUROC Lower

95% CI

Upper 95%

CI

City of Toronto Health Unit 0.36 0.23 0.49

Grey Bruce Health Unit 0.49 0.34 0.64

Simcoe Muskoka District Health Unit 0.40 0.25 0.54

Niagara Regional Area Health Unit 0.44 0.31 0.57

Peel Regional Health Unit 0.49 0.36 0.62

City of Ottawa Health Unit 0.46 0.31 0.60

City of Hamilton Health Unit 0.41 0.28 0.54

York Regional Health Unit 0.34 0.23 0.46

Leeds, Grenville and Lanark District Health Unit 0.46 0.33 0.59

Middlesex-London Health Unit 0.41 0.29 0.54

Durham Regional Health Unit 0.34 0.22 0.47

The Eastern Ontario Health Unit 0.52 0.38 0.66

Peterborough County-City Health Unit 0.38 0.25 0.50

Hastings and Prince Edward Counties Health Unit 0.45 0.31 0.59

Waterloo Health Unit 0.39 0.27 0.51

The District of Algoma Health Unit 0.66 0.54 0.78

Renfrew County and District Health Unit 0.52 0.40 0.65

Thunder Bay District Health Unit 0.47 0.34 0.60

Porcupine Health Unit 0.47 0.34 0.59

Haliburton, Kawartha, Pine Ridge District Health Unit 0.40 0.27 0.52

North Bay Parry Sound District Health Unit 0.39 0.27 0.51

Oxford County Health Unit 0.45 0.33 0.58

Lambton Health Unit 0.50 0.37 0.62

Chatham-Kent Health Unit 0.56 0.42 0.70

Haldimand-Norfolk Health Unit 0.52 0.40 0.64

Halton Regional Health Unit 0.38 0.24 0.51

Windsor-Essex County Health Unit 0.48 0.35 0.60

Northwestern Health Unit 0.51 0.38 0.63

Kingston, Frontenac and Lennox and Addington Health Unit 0.53 0.40 0.66

Sudbury and District Health Unit 0.64 0.51 0.76

Huron County Health Unit 0.58 0.45 0.71

Wellington-Dufferin-Guelph Health Unit 0.56 0.44 0.69

Perth District Health Unit 0.44 0.32 0.57

Brant County Health Unit 0.46 0.33 0.58

Timiskaming Health Unit 0.49 0.37 0.62

Elgin-St. Thomas Health Unit 0.59 0.47 0.72

155

APPENDIX C: One-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency Visits for All Ontario Health Units

156

Figure C.1: One-Week-Ahead Forecasts of the Weekly Aggregate Number of Hospital Emergency



157




158



Actual Visits for the (Approximate) Simcoe Muskoka District Health Unit

159



Actual Visits for the (Approximate) Niagara Regional Health Unit

160



Actual Visits for the (Approximate) Peel Regional Health Unit

161



Actual Visits for the (Approximate) City of Ottawa Health Unit

162



Actual Visits for the (Approximate) City of Hamilton Health Unit

163



Actual Visits for the (Approximate) York Regional Health Unit

164



Actual Visits for the (Approximate) Leeds, Grenville and Lanark District Health Unit

165



Actual Visits for the (Approximate) Middlesex-London Health Unit

166



Actual Visits for the (Approximate) Durham Regional Health Unit

167



Actual Visits for the (Approximate) Eastern Ontario Health Unit

168



Actual Visits for the (Approximate) Peterborough County-City Health Unit

169



Actual Visits for the (Approximate) Hastings and Prince Edward Counties Health Unit

170



Actual Visits for the (Approximate) Waterloo Health Unit

171



Actual Visits for the (Approximate) District of Algoma Health Unit

172



Actual Visits for the (Approximate) Renfrew County and District Health Unit

173



Actual Visits for the (Approximate) Thunder Bay Health Unit

174



Actual Visits for the (Approximate) Porcupine Health Unit

175



Actual Visits for the (Approximate) Haliburton, Kawartha, Pine Ridge District Health Unit

176



Actual Visits for the (Approximate) North Bay Parry Sound District Health Unit

177



Actual Visits for the (Approximate) Oxford County Health Unit

178



Actual Visits for the (Approximate) Lambton District Health Unit

179



Actual Visits for the (Approximate) Chatham-Kent Health Unit

180



Actual Visits for the (Approximate) Haldimand-Norfolk Health Unit

181



Actual Visits for the (Approximate) Halton Regional Unit

182



Actual Visits for the (Approximate) Windsor-Essex County Health Unit

183



Actual Visits for the (Approximate) Northwestern Health Unit

184



Actual Visits for the (Approximate) Kingston, Frontenac and Lennox and Addington Health Unit

185



Actual Visits for the (Approximate) Sudbury and District Health Unit

186



Actual Visits for the (Approximate) Huron County Health Unit

187



Actual Visits for the (Approximate) Wellington-Dufferin-Guelph Health Unit

188



Actual Visits for the (Approximate) Perth District Health Unit

189



Actual Visits for the (Approximate) Brant County Health Unit

190



Actual Visits for the (Approximate) Timiskaming Health Unit

191



Actual Visits for the (Approximate) Elgin-St. Thomas Health Unit

FORECASTING HOSPITAL EMERGENCY DEPARTMENT ...

Documents

Transcript of FORECASTING HOSPITAL EMERGENCY DEPARTMENT ...