Distributed, intelligent, interactive visualization and exploration of time-oriented clinical data...

Artificial Intelligence in Medicine (2006) 38, 115—135

http://www.intl.elsevierhealth.com/journals/aiim

Distributed, intelligent, interactive visualizationand exploration of time-oriented clinical dataand their abstractions

Yuval Shahar, Dina Goren-Bar *, David Boaz, Gil Tahan

Department of Information Systems Engineering, Faculty of Engineering Sciences,Ben-Gurion University of the Negev, P.O. Box 653, Beer-Sheva 84105, Israel

Received 31 March 2004; received in revised form 13 March 2005; accepted 29 March 2005

KEYWORDSIntelligentvisualization;Clinical systems;Temporal reasoning;Temporal abstraction;Knowledge-basedsystems;Human-computerinteraction;Medical informatics

Summary

Objectives: We present KNAVE-II, an intelligent interface to a distributed architec-ture specific to the tasks of query, knowledge-based interpretation, summarization,visualization, interactive exploration of large numbers of distributed time-orientedclinical data, and dynamic sensitivity analysis of these data. KNAVE-II main contribu-tions to the fields of temporal reasoning and intelligent user interfaces are: (1) thecapability for interactive computation and visualization of domain specific temporalabstractions, supported by ALMA — a computational engine that applies the domainknowledge base to the clinical time-oriented database. (2) Semantic (ontology-based)navigation and exploration of the data, knowledge, and temporal abstractions,supported by the IDAN mediator, a distributed architecture that enables runtimeaccess to domain-specific knowledge bases that are maintained by expert physicians.Methods and materials: KNAVE-II was designed according to 12 requirements thatwere defined through iterative cycles of design and user-centered evaluation. Thecomplete architecture has been implemented and evaluated in a cross-over studydesign that compared the KNAVE-II module versus two existing methods: paper chartsand an Excel electronic spreadsheet. A small group of clinicians answered the samequeries, using the domain of oncology and a set of 1000 patients followed after bone-marrow transplantation.Results: The results show that users are able to perform medium to hard difficultylevel queries faster and more accurately by using KNAVE-II than paper charts andExcel. Moreover, KNAVE-II was ranked first in preference by all users, along allusability dimensions.

* Corresponding author. Tel.: +972 8 6472789; fax: +972 8 6477527.E-mail addresses: [email protected] (Y. Shahar), [email protected] (D. Goren-Bar), [email protected]

(D. Boaz), [email protected] (G. Tahan).

0933-3657/$ — see front matter # 2005 Elsevier B.V. All rights reserved.doi:10.1016/j.artmed.2005.03.001

1. Background: the need forabstraction, visualization, andexploration of time-oriented clinicaldata and concepts

Management of patients, especially chronic patients,requires collection, interpretationandexplorationoflarge amounts of time oriented data. The task ofcreating interval-based concepts (abstractions) fromtime-stamped raw data is called temporal abstrac-tion [1]. Anexample is abstractionof periods of bone-marrow toxicity (as defined in a particular context,e.g., following Bone-Marrow Transplantation andusing a particular therapy protocol) from raw indivi-dual hematological data (Fig. 1).

Reasoning about propositions interpreted overtime points and time intervals has a long historyin philosophy, linguistics, computer science, and inparticular, artificial intelligence. Examples includeMcDermott’s point-based logic of time and actions[2], intended to support automated planning;Allen’s interval-based logic of time [3], intendedto support planning and natural-language under-standing, and Shoham’s temporal logic and analysisof temporal-semantic properties [4], which pro-vided also a certain common ground to Allen’sand McDermott’s frameworks.

The ability to automatically create interval-based abstractions of time-stamped data, in parti-cular in the medical domain, has multiple implica-

Figure 1 Temporal abstraction of a patient’s data following aplotted over time at the bottom. External events and the absabove the data. ( ) An external event (medical interventiocontext interval; ( ) an abstraction (derived concept) intetherapy protocol for treating chronic graft vs. host disease (CGgrade n, a (temporal) abstraction of platelet and granulocyt

tions. Data summaries of time-oriented electronicdata have an immediate value to a human user, suchas to a physician scanning a long patient record formeaningful trends.

Several approaches were proposed to the tem-poral-abstraction task, typically focusing on clinicaldomains involving chronic patients, such as patientswho have oncological or immunological diseases,insulin-dependent diabetes, or growth and devel-opment problems [5—10], or on more hectic clinicaldomains, in which data arrive rapidly, such as occursin intensive neonatal or adult care units [11—13].

A consideration orthogonal to the rate of arrivalof the information (although it is more common in‘‘slow’’ domains), which is highly relevant to theissue of optimal visualization and exploration oftime-oriented data and derived concepts, is howto treat the multiple temporal granularities (e.g.,hours, days, months) inherent in any discussion of alongitudinal course. Several syntactic and semanticapproaches were proposed and implemented[14,15].

Temporal abstractions also support recommenda-tion and explanation of recommended actions byintelligent decision-support systems, as well asmonitoring of clinical guidelines during application[16], and are a useful representation for intentionsof clinical guidelines [17]. Meaningful time-orientedcontexts enable generation of context-specificabstractions, maintenance of several interpreta-tions of the same data within different contexts,

116 Y. Shahar et al.

Conclusions: Initial evaluation of KNAVE-II and its supporting knowledge basedtemporal-mediation architecture, by applying it to a large data base of patientsmonitored several years after bone marrow transplantation (BMT), has producedhighly encouraging results.# 2005 Elsevier B.V. All rights reserved.

bone-marrow transplantation intervention. Raw data aretractions computed from the data are plotted as intervalsn); (*) platelet counts; (~) granulocyte counts; ( ) arval; BMT, bone-marrow transplantation (an event); PAZ, aVHD), a complication of BMT; B(n), bone-marrow toxicity

e counts that holds over a particular interval.

Time-oriented clinical data and their abstractions 117

and a certain amount of hindsight and foresight [18].Domain-specific, meaningful, interval-based char-acterizations of time-oriented data are a prerequi-site for effective visualization and exploration ofthese data [19—21].

A problem solver implementing a formal pro-blem-solving method, the knowledge-based tem-poral abstraction (KBTA) method [1], calledRESUME, has been previously implemented andevaluated in several clinical domains [22]. The KBTAmethod requires explicit representation of fourtypes of domain-specific knowledge: structural(e.g., part-of relations), functional (e.g., classifica-tion functions and definitions of complex patterns),logical (e.g., what concepts can be concatenatedover time), and probabilistic (e.g., interpolationfunctions that bridge temporal gaps between similarconcepts and fill in missing data), organized within adomain-specific temporal-abstraction ontology [1].The knowledge required by the KBTA method under-lying the RESUME system is acquired from medicaldomain experts [23]. The knowledge base and com-putational mechanisms of the RESUME system havebeen extended so as to enable representation oflinear and repeating patterns using the CAPSULtemporal-pattern language and methodology [24].The methodology was implemented and tested withrespect to both acquisition and representation ofpatterns and detection of the patterns within a largelongitudinal clinical database [25].

A stand-alone temporal-abstraction module isuseless without the capability for effectively inte-grating it with the required clinical data and med-ical knowledge, and for effectively mediatingtemporal queries, when relevant, from various med-ical decision-support applications. An early tem-poral-mediation architecture, TZOLKIN [26] hasbeen previously developed to integrate the data-bases, knowledge bases, and the computationaltemporal-abstraction module. Such architecturesupports mediation of queries to time-oriented clin-ical databases. In our work, we have extended thisearly architecture to fully answer the requirementsof care providers.

Supporting automated applications is not suffi-cient, however, to fully assist clinical care providersand researchers. An additional aspect of supportingclinicians performing tasks such as diagnosis, ther-apy, and research, is by supplying them with thetechnology for on-the-fly visualization, interpreta-tion and exploration of the clinical data and of thehigher level, knowledge-based, concepts that canbe derived from these data.

Visualization and exploration of information ingeneral, and of large amounts of time-oriented datain particular, is essential for effective decision mak-

ing. Larkin and Simon [27] have demonstrated thatthe usefulness of visual representation is mainly dueto (1) the reduction of logical computation throughthe use of direct perceptual inference, and (2) thereduction of necessary search for informationthrough the use of efficient graphical representa-tion.

Researchers in the areas of visualization of clin-ical time-oriented data [21,28], have developeduseful visualization techniques for static presenta-tion of raw time-oriented data and for browsinginformation. An excellent treatise on visualizationis the series of books by Edward Tufte on methods todisplay information [29,30]. However, the methodspresented in the above-mentioned papers and booksare typically not interactive and relate to the rawdata and not to abstractions of it. The Lifelinesproject [31,32] developed an intuitive visualizationof historical events and data, demonstrated also in amedical domain, including an easy-to-use zoom-inand zoom-out interface. However, it does not buildon any domain-specific abstraction knowledge andno underlying computational abstraction mechan-isms. InfoZoom [33] uses a novel technique to dis-play data sets as highly compressed tables whichalways fit completely onto the screen. The goal oftheir approach is not a completely automatic algo-rithm that searches for interesting results. Rather,they intend to enable the user to interactivelyexplore and get a feeling of the data, detect inter-esting knowledge, and gain a deep understanding ofthe data set. Queries are performed by selectingparts of the displayed data. Derived attributes canbe defined as in a spreadsheet program and areautomatically updated when necessary. InfoZoomsupports the medical expert in understanding thedata and detecting the hidden knowledge. Ourapproach also enables interactive exploration oflarge amounts of clinical data; however, as will bedemonstrated, it includes continuous access to anunderlying architecture that performs the abstrac-tions and to a domain-specific knowledge base; theknowledge base is used by he abstraction mechan-isms as well as for navigating among various domain-specific concepts (e.g., from a pattern to its com-ponents) during the exploration process.

Several techniques specifically address the visua-lization of multidimensional data regarding clinicalexaminations [34]. In the MedView project [35], twoinformation visualization tools were developed forvisualizing clinical experience derived from largeamounts of clinical data. The first tool (The Cube)implements dynamic three-dimensional parallel dia-grams to enable the clinician to intelligibly analyzeexisting patient material and to allow for patternrecognition and statistical analysis. The system


displays a 3D cube containing a number of parallelplanes: each plane is devoted to one of the chosenattributes, and points on that plane display the indi-vidual values of the attribute for specific patients.The second tool, SimVis, was designed to help clin-icians to classify and cluster clinical examinationdata. User interaction was supported by 3D visualiza-tion of clusters and similarity measures. The tools donot focus on temporal data or on abstractions of it.

Spotfire, a well known application for visual datamining and information visualization has been usedin the pharmaceutical drug discovery task. Imple-menting dynamic queries, Spotfire supports easyimport/export of data, rapid change of axes, coloror size coding, and collaboration support [36—38].Spotfire became a leader in dynamic data visualiza-tion with a very nice direct manipulation user inter-face; however, it lacks a time dimension (the defaultis that all data were gathered at the same time orare timeless), and has no knowledge base, or com-putational mechanisms that could use such a knowl-edge base to generate high-level abstractions.Another visualization project in the medicaldomain, the MeSHBrowse system, allows users tobrowse semantically associated links in the MedicalSubjects Headings (MeSH) [39]. Although it displaysa two-dimensional tree representation of associatedcategories to a selected concept, MeSHBrowse is notconnected to the individual patient’s data.

In recent years, researches have investigatedvarious techniques for information visualization,including conceptual maps, radar maps, tree maps(conetree/camtree maps, hyperbolic tree), Koho-nen maps, fish eye views and dynamic queries inter-faces and more [40—44], as well as using well knownstatistical and graphical methodologies, such as 3Drepresentations, scattergrams, pie charts, barcharts, and their derivative techniques. These dis-play methods, however, typically do not focus onvisualization of domain-specific temporal abstrac-tions and on the issue of interactive manipulationand exploration of the data and multiple levels of itsabstractions, using domain-specific knowledge.Typically, these capabilities have been omitted,because they require a formal, domain-independentrepresentation of the domain-specific temporal-abstraction knowledge, considerable effort in mod-eling the visualized domain, and availability ofcomputational mechanisms for creation of theabstractions.

The information visualization literature definesseveral techniques that enable interactive manip-ulation and exploration of data [42]. Brushing andlinking enable dynamic connection of two ormore views of the same data, such that achange to the representation in one view affects

the representation in the other views as well [45].Zooming is used to move ‘‘closer’’ to a particularitem [46]. Zooming can help us to detect a specificdata item within a set of closely-related items.Focus-plus-context makes a portion of the view —the focus of attention — larger, while simultaneouslyshrinking the surrounding objects [47]. Shneiderman[48] defines the central principle of the visual designguidelines as the visual-information-seeking-mantrathat states: ‘‘Overview first, zoom and filter, thendetails on demand.’’ In his data Type by Task Tax-onomy (TTT) he adds three more actions: (1) relateby visualizing relationships between items, (2) keepa history of actions to support undo, replay andprogressive refinement, and (3) allow extractionof sub-collections and of the query parameters.

We have implemented several of the above-men-tioned functionality (mainly with respect to visua-lization of knowledge-based temporal abstractions)in a previous version of a system called Knowledge-based Navigation of Abstractions for Visualizationand Explanation (KNAVE), as a stand-alone proto-type module. Preliminary assessments of the KNAVEsystem, in the oncology domain, were highlyencouraging, and demonstrated the feasibility ofthe whole architecture, and in particular of theknowledge-based exploration concept [19,20].

In order to provide a useful service for interactiveexploration of time-oriented clinical data, addi-tional requirements have been identified. Theresults of our preliminary studies have led to thedefinition of an enhanced version of the overalltemporal-mediation architecture and, in particular,an enhanced interactive visualization and explora-tion tool: KNAVE-II.

Interactive visualization and exploration of clin-ical data and abstracted knowledge is supported bytwo mechanisms directly connected to the KNAVE-IIinterface, providing two main contributions to thefield of intelligent user interfaces:

1. A
dding the capability of interactive computationand access of domain specific temporal abstrac-tions, a capability supported by the ALMA system[49], a computational engine that applies thedomain KB through the clinical DB.
2. E
nabling navigation and exploration of the clin-ical data and abstractions derivable from them,by moving through the links of a semantic net-work that is embodied in a series of domain-specific knowledge bases; the knowledge basesare maintained by expert physicians. The inter-active mediation of the queries to the clinicaldatabases, the relevant medical knowledgebase, and the clinically meaningful abstractionsthat can be derived from the data using the


knowledge is supported by the IDAN mediator[50], a distributed architecture that enablesonline integration of clinical time-oriented datawith relevant domain-specific knowledge, toproduce the requested clinically meaningfultemporal abstraction.

We implemented the user centered designapproach to define a set of requirements that shouldbe included in KNAVE-II. User Centered-Design (UCD)is a process that places the person at the centerfocusing on cognitive factors (such as perception,memory, learning, problem-solving, etc.) as theycome into play during peoples’ interactions withthe interface [51]. UCD seeks to answer questionsabout users and their tasks and goals and use thefindings to drive development and design. Adopting auser-centered design process leads to more usablesystems and products dealing with both usefulnessand usability [52]. The active involvement of usersfrom the very early stages of concept developmentenables the incorporation of user-derived feedbackinto system design whereby a prototype is designed,tested and modified iteratively [53].

An independent group of three clinicians in inter-nal medicine, pediatrics, and oncology from Stan-ford University’s Medical Informatics Program andthe Palo Alto Veterans Administration Health CareSystem, as well as three Ob/Gyn and other expertsfrom the Soroka Medical Center, associated withBen-Gurion University’s School of Medicine, helpedus to define the requirements. A multidisciplinaryteam composed of a user interface designer, amedical informatics expert, a software engineerand a knowledge engineer conducted with eachexpert at least three face to face interviews, fiveconference calls and sent and received several doz-ens of emails. All the diverse communications withthe experts aimed at the design and evaluation ofmockup and working prototypes. This user-centereddesign iterative process has led to the definition oftwelve requirements for the provision of a compre-hensive service for interactive exploration of timeoriented clinical data, and its implementation as theKNAVE-II system (see Section 3).

2. Desiderata for interactiveexploration of time-oriented clinicaldata

Our decade-long research into the subject of auto-mated interpretation and exploration of time-oriented clinical data together with the user cen-tered design approach have uncovered a set ofdesiderata that must be answered to truly assist

clinical care providers, administrators and research-ers who need to understand and explore a mass oftime-oriented clinical data. The desiderata for pro-vision of a service for interactive exploration of timeoriented clinical data include the following require-ments, which we will exemplify within the clinicalscenario of management of oncology patients:

2.1. D-1: distributed and modulararchitecture

Amodular, scalable architecture to enable the appli-cation of diverse (and preferable distributed) typesof knowledge to the same clinical data, or the appli-cation of the same knowledge to different clinicaldata bases; the architecture should support access toboth the data and the abstractions derivable from it.

The need for abstraction of meaningful domain-specific concepts and patterns in clinical tasks hasbeen already noted [22]. In order to support decisionsupport, clinicians should access diverse clinicaldata and apply different types of knowledge to thatdata. While applying the knowledge to the clinicaldata they make abstractions of that data to arrive tomeaningful decisions. This task is very complex andtime consuming. Thus, an architecture that enablesaccess at the point of care (or quality assessment orresearch) to abstractions automatically extractedfrom the large mass of raw clinical data can saveprecious time and prevent errors.

Moreover, the distributed architecture is a keyrequirement, because domain-specific knowledgebases are often maintained by expert physicianswho are geographically dispersed. Thus, a knowl-edge base regarding liver functions during the laterecuperation phase following a bone-marrow trans-plantation, created at one site, can be automati-cally applied to data of patients who are beingtreated by a family practitioner at another site. Acapability for displaying interactively the results ofthe integration of a certain set of clinical data with acertain set of medical knowledge instances willenable physicians who not necessarily as expert asthe editor of the knowledge base to automaticallyapply expert knowledge to their own patients clin-ical data while being remotely located, thus provid-ing a higher level of clinical decision support.

2.2. D-2: visualization of both raw dataand its abstractions

Effective visualization and exploration shouldinclude both the raw clinical data and the abstractconcepts derived from those data. These conceptsshould be derived in a context-sensitive fashion(e.g., by using the bone-marrow toxicity definition


that is most specific to the particular combination ofage, gender, therapy protocol, period before orafter the bone-marrow transplantation interven-tion, or type of chemotherapy medication; and,for economic reasons, only that knowledge).

Interactive visualization of clinical data has beenalready supported in many previous works. It isalready recognized that visualization and explora-tion of information of large amounts of time-oriented data is essential for effective decisionmaking [27,31,32]. Although several applicationsprovide the visualization of multidimensional clin-ical data [34,35,38], none of them supports anydomain-specific abstraction knowledge or abstrac-tion mechanisms. At most, InfoZoom [33] supportsthe medical expert in understanding the data anddetecting the hidden knowledge. Our approach alsoenables interactive exploration of large amounts ofclinical data, includes online computation of therelevant abstractions, and is supported by a con-tinuous link to a knowledge base. It enables theclinician to visualize not only the raw clinical data ofher patients (e.g., Hemoglobin values at particulartime points), but also abstract, clinically meaningfulconcepts (e.g., ‘‘3 weeks of moderate anemia’’)that may save her time and support her clinicaldecision making.

2.3. D-3: temporal granularity

The visualization should support interactiveexploration of time oriented data at different tem-poral granularities (e.g., hours, days, months).

Representation and visualization of various tem-poral granularities and their suitability for variousclinical tasks have been widely studied [14,54].

This requirement seems to be very important forproviding thephysicianwith a capability forexploringspecific known patterns at various time intervals(from few hours to several years) or even for disco-vering newpatterns by looking at the raw data and itsabstractions while viewing them within periods oftime that are not necessarily previously defined.Thus, we need to enable the clinician or clinicalresearchers who are browsing a month’s worth ofdata to zoom into the data or the derived concepts ofeach day in that month, or into any particular period(e.g., 6 particular days where the white blood-cellcounts have rapidly decreased) within that month.

2.4. D-4: absolute and relative time lines

The system should support both a calendar-basedtime line (e.g., answer queries regarding hemoglo-bin values during August 1994) and a relative timeline, which refers only to clinically significant events

(e.g., periods of decreasing liver function within 6weeks from the start of chemotherapy).

This capability has been already been supportedin previous studies. Kahn [9] showed in the TOPAZarchitecture that physicians were interested inmanipulating time lines in the context of a che-motherapy treatment and not in the absolute datethat it occurred. Also Lifelines [31,32] enabledsimilar capability. An interactive visualization inter-face should enable to visualize how the applicationof a specific treatment protocol has affected thepatient a few days or weeks after its provision at anylevel of granularity required by that specific clinicaltask.

2.5. D-5: intelligent exploration of rawdata and abstractions

Effective exploration of both the raw data and theirabstractions, using meaningful domain-specificsemantic relations (e.g., derived-from, part-of).Thus, motion should be supported along the seman-tic network implicitly defined by the various types ofdomain knowledge required for computing theabstract concepts (e.g., bone-marrow toxicitylevels) from the raw data of measurements andinterventions (e.g., white blood-cell counts, hemo-globin values, and platelet counts; and the bone-marrow transplantation or various types of transfu-sions).

In the field of information visualization severaltechniques have been implemented to abstracthigher level concepts from raw free-text data.Scatter/gather [55], scatter plots [56] and Kohonenmaps [57] show clusters of words or documentsorganized by automatically extracted categoriesat several levels. They enable free exploration ofthe categories at different levels of abstraction tillthe level of the individual object (terms or docu-ments). However, this requirement has not beenimplemented in the field of quantitative timerelated data supported by abstracted knowledge.

2.6. D-6: explanation

The system should enable an explanation of theabstractions by providing on demand the most rele-vant data and knowledge used to derive the abstrac-tion currently considered (e.g., ‘‘Which particulardata underlie the current bone-marrow toxicityabstraction? How was it derived and what classifica-tion-definitions and other types of knowledge wereused?’’). These explanations should be availablerecursively for all intermediate abstractions.

Since the late seventies, rule-based systemsattempted to provide explanations in order to let


the user understand and rely on the way the systemarrived to a conclusion. Explanations — or even thepotential of asking for them, whether that potentialis exploited or not — increase the confidence ofphysicians using a medical decision support system[58]. Our requirement demands that the systemshould provide full details of the data that lead toan abstracted concept as well as the functional-classification, temporal-interpolation functions, ortime and value constraints that define each abstrac-tion.

2.7. D-7: statistics

The system should provide statistics (customized foreach data type, whether numeric or symbolic)regarding both the raw data and the conceptsabstracted from it. Providing various statistics hasusually been one of the primary tasks supported byvisualization interfaces (Card, 2003). They should bedynamic, recomputed each time any new dataappears in the data base. However, this service hasnot been provided for abstracted knowledge that isbasedonquantitative time relateddata.The require-ment demands that statistics should be provided forboth types of data, raw and abstracted knowledgewith the relevant matching statistical analysis anddisplays (i.e., not just for the hemoglobin values butalso for the bone-marrow toxicity periods).

2.8. D-8: search and retrieval

The system should support easy and fast search andretrieval of clinically significant concepts. Searchfunctionality has become one of the most popularfeatures in any user interface. The search function-ality should reduce the user’s memory burden, pro-vide feedback, let the user a sense of control overthe search results, and easy reversal of actions [42].The existence of multiple medical vocabularies andthe lack of a standard terminology impose the needfor a search and retrieval facility that enables tobrowse amongst the retrieved concepts and selectthose the user would like to visualize. Thus, the usershould be able to quickly search for a concept suchas multi-organ toxicity, either through a semanti-cally organized concept hierarchy, through a fasttext-string search, or through other means.

2.9. D-9: dynamic sensitivity analysis

The system should include capabilities for interac-tive exploration of the effects of simulated hypothe-tical modifications of raw data on the derivedconcepts, to increase the clinician’s confidence inthe robustness of the interpretation. Thus, one

should be able to determine whether changing aparticular suspicious platelet-count value on a cer-tain day affects in any way an overall pattern ofmulti-organ toxicity during that month. SensitivityAnalysis (sometimes referred to asWhat-if Analysis)is a well known feature already integrated in elec-tronic spreadsheets. It determines which values in aspreadsheet model have the greatest impact on theresults. Input values are varied (over a range) andthe amount of change on the results is recorded. TheMicrosoft Excel spreadsheet, for example, includesan application named TopRank to perform a multi-way what-if analysis on a decision tree. The user,however, needs to define the links among variousconcepts herself-they are not part of the domain’sontology — and the algorithms using those links; theconcept of time is not a first class citizen within thecomputations; and the output concepts are notnecessarily of the same type (and a part of the sametimeline) as the input data, which would be highlydesirable in order to continue and apply to theseconcepts all of the computational and display opera-tors already familiar to the user.

2.10. D-10: clinical-task support

The system should be customizable for a specificclinical task (e.g., monitoring of diabetic or oncol-ogy patients), which might require repeatedly pro-ducing for each patient or a group of patients acluster of closely related multiple abstractions(e.g., fasting, lunch and dinner blood-glucoseabstractions together with insulin administrationsand renal function abstractions; or various hemato-logical abstractions together with various renal andliver functions). Clinical-task support is a require-ment that refers to the ability of the clinician tocustomize the interface for specific clinical tasks. Aresearch community who has been dealing withcustomization issues for a long time is the userinterface community, which has often emphasizedthe notion of adaptive user interfaces [59]. Adaptiveuser interfaces are designed to tailor a system’sinteractive behavior considering both individualneeds of human users and changing conditionswithin an application environment. Kappel et al.[60] propose to add customization as a new model-ing dimension into their requirements framework.The scope of customization comprises all the mod-eling dimensions; however, it is especially crucial forthe content, presentation and hypertext levels.Chen et al. [56] have also shown the potentialusefulness of concept-based clustering, albeit inthe case of searching the internet to find a pageuseful to the user. As will be seen when discussingthe KNAVE-II system, we have defined two services


for customization. One relates to the content to bevisualized, enabling an expert physician to save in aspecial set of data elements and derived conceptscommonly related to a specific clinical task (such asmonitoring of diabetic patients) that can be appliedto any patient that a physician wants to monitor onthat respect. The other customization servicerelates to the graphical objects of the presentationbymeans of ‘‘plug-in’’ modules that are applied onlyin particular domains or contexts to provide a ‘‘lookand feel’’ of a task-related interface. For example,in the case of the domain of monitoring the variousclinical parameters of women who come to BenGurion University’s Medical Center fertility clinic,the need has emerged for having a context-sensitivetool tip that will show a baby’s smiling face and themultiple quantitative values regarding the embryo’sdevelopment, different from the usual one used tovisualize the woman’s physiological data.

2.11. D-11: collaboration

The system should support collaboration betweendifferent clinicians and researchers who need toaccess the same patient data and concepts deriva-ble from these data. The need for collaborationbetween medical staff or different clinical roleshas already been established. Groz [61] defined acollaborative enterprise as a cognitive entity inwhich the processes of information acquisition,planning, decision-making, and learning have prop-erties that in turn shape behaviors and determineoutcomes, that applies to many patient manage-ment clinical tasks. In the medical domain, moststudies focus on developingmodels and technologiesto support clinical discussions, clinical decisionmak-ing and common medical terminology [62]. How-ever, the provision for collaboration based on theactual clinical data and concepts abstracted fromthem using a particular domain-specific knowledgebase is still lacking. Such a capability might improvesignificantly the support for tasks such as clinical-trial management, in which clinicians and biostatis-ticians need to access the same data, but typicallyat different levels of abstraction. Similarly, it wouldbe useful if the capability for sending to a domainexpert or pointing to her a particular surprisingpattern discovered within an oncology patient’srecord was supported as part of the same interfaceenabling exploration of the patient’s record.

2.12. D-12: documentation

The system should support documentation of theexploration process for multiple future uses, suchas the clinician’s own review of patient data, colla-

boration, external auditing, and quality assessmentat various administrative levels. This requirementcan be supported at a simple level by talking a snap-shot of the data (for example, using the standardprint screen function). However, an advanced serviceshould support documentation of the whole abstrac-tion and exploration process, to better enable back-ward exploration from the current documentedpatient state and to assess the clinical decision-mak-ing process. Thus, the documentation should includenot just a static graphical snapshot of a top-levelhematological pattern, but also all the related valuesof the raw data, the knowledge relevant to thesedata, and the abstracted concepts that wereexplored and displayed at any selected time span.

The above-mentioned desiderata emerged fromour extensive experience and from studies per-formed by multiple researchers. However, thesedesiderata have not been applied to the task ofexploring qualitatively and quantitatively largemasses of longitudinal clinical data, through theuse of a context-sensitive domain-specific knowl-edge base and dedicated computational mechan-isms, to support actual clinical tasks such as patientmanagement.

3. The KNAVE-II architecture andoperators

We have recently developed an advanced prototypemodule, KNAVE-II, a new intelligent interfacedesigned to fulfill all the above-mentioned desider-ata. In this section, we explain in detail how theKNAVE-II architecture supports all desiderata (D-1 toD-12).

3.1. D-1: distributed and modulararchitecture

In order to support the modularity and accessibilityrequirements (D-1) (in the sense used above, such asat the point of care), a new knowledge-based dis-tributed temporal-abstraction mediation architec-ture, IDAN [50], was designed and implemented.IDAN uses a modern version of the RESUME problemsolver,ALMA [49]. Themodular architecture includesautomated acquisition of domain-specific temporal-abstraction knowledge, a computational temporal-abstractionmechanismusing that knowledge, adata-access service that accesses time-oriented data-bases, and controlled-vocabulary servers. The mod-ular architecture includes multiple knowledge basesand time-oriented databases (Fig. 2).

Each use of a KNAVE-II operator issues implicitly acommand to the application interface of the IDAN


Figure 2 The distributed architecture of the IDAN knowledge-based temporal-abstraction mediator and its relationshipto the KNAVE-II system. End users interact with KNAVE-II to submit time-oriented queries. The temporal-abstractionmediator answers these queries using data from the appropriate local data-source, and temporal-abstraction knowledgefrom the appropriate domain-specific knowledge base. The integration of the data and knowledge is performed by thetemporal-abstraction service, a Web service containing the ALMA temporal-abstraction computational module. KNAVE-IIthen enables users to visually and dynamically explore the resultant abstractions, using a specialized graphical displayand direct access to the domain-specific knowledge. Arrows indicates a ‘‘uses’’ relation.

temporal-abstraction controller (see Fig. 2), usuallyas an XML message. For example, clicking on anabstract concept in the ontology browser issues agoal-directed query to compute that concept, whichleads to querying the knowledge base for the defini-tion of that concept and for the raw data needed tocompute it, and then to querying the clinical data-base for the necessary data. The data and knowl-edge are then sent to the temporal-abstraction(Web) service, containing the ALMA module, whichintegrates the data and knowledge and returns theanswers set to the controller, which in turn returns itto the querying application, in this case KNAVE-II.

To make the knowledge reusable and facilitate itsapplication to multiple clinical databases, we areusing, to represent the clinical raw-data conceptsfound at the leaves of the knowledge-base’s abstractconcepts, only terms (and measurement units) takenfrom a set of controlled medical terminologies. Wehave createda standardmedical vocabularies service(see Fig. 2) that serves as a search engine for a set ofdistributed, web-based standard medical vocabularyservers that we had implemented. The main voca-bularies used by the IDAN architecture include theInternational Classification of Diseases, Ninth Revi-sion, Clinical Modification (ICD-9-CM) [63] and theSystematized Nomenclature of Medicine, ClinicalTerminology (SNOMED-CT), a comprehensive diag-nostic terminology [64], in the case of clinical diag-noses; Current Procedural Terminology (CPT) in thecase of diagnostic and therapeutic procedures [65];the Logical Observation Identifiers Names and Codes

(LOINC) standard, in the case of laboratory tests andphysical signs and symptoms [66]; and the AmericanNational Drug File (NDF) ontology in the case ofmedications.

The use of standard vocabularies is a key conceptin our framework; it enables us to share temporal-abstraction knowledge sources that are not specificto a particular set of data-source terms, but can beapplied to any clinical database that stores similardomain-specific data types. The vocabulary server isused by local data-source owners to associate localdata-source concepts with standard medical con-cepts, and also by medical experts to associateclinical terms in the temporal-abstraction knowl-edge base, with standardized medical terms.

A full scalable distributed architecture requiresthe capability of remote connectivity to diversedatabases, knowledge bases, vocabularies and algo-rithms to enable the application of types of knowl-edge to the same data and different databases to thesame knowledge. KNAVE-II is a client to the web-based distributed architecture described above, andis implemented as a downloadable client applicationfrom our internet website. After installation at theuser’s site, the system recognizes the current KNAVE-II version and updates it automatically through itsinternet connection.

A configuration service enables users of a KNAVE-IIclient to select, at the beginning of an explorationsession, the desired database, the desired temporal-abstraction service (a Web service encapsulating theALMA temporal-abstraction computational module),


and an appropriate knowledge base (of the applica-tion, needed by the selected temporal abstractionservice). The selection of a temporal-abstractionservice constrains the choice of relevant knowl-edge-base sources.

3.2. D-2: visualization of both raw dataand its abstractions

To solve the visualization desiderata, we use anintelligent user interface. Intelligent user interfacesare knowledge-based interfaces that mediatebetween person and machine to increase the easeand effectiveness of user interactions [67,68].KNAVE-II is a knowledge-based interactive visualiza-tion and exploration intelligent user interface. Theinterface is used to explore a single patient record ora set of such records. Fig. 3 shows the main interfaceof a KNAVE-II visualization and exploration client, asused for exploration of an oncology patient moni-tored after bone-marrow transplantation. KNAVE-IIenables interactive, dynamic exploration by the userof raw data and their abstractions in the domain ofoncology. Fig. 3 demonstrates our current design forthe integration of the knowledge browser (whichreflects the contents of the domain’s temporal-abstraction knowledge base, that is, its temporal-abstraction ontology) and data-browsing panels(which show the contents of the database or of theresults of a temporal abstraction process applied onsuch contents). Each panel represents a differentconcept (raw data, such as white blood-cell counts;abstractions, such as white blood-cell levels andbone-marrow toxicity grades; or even higher-level,more complex patterns, such as multi-organ toxicityor decreasing platelet half-life). One could imagineseveral otherwaysofdisplaying the resulting abstrac-tions, such as displaying both raw data and conceptsderived from it within the same panel. However, oneof our main design choices was to separate conceptsinto different panels, so as to enable performingoperations such as computation of statistics, zoomingin and out, various exploration, explanation, anddynamic sensitivity analysis, on each concept andeach abstraction of that concept separately (thisdesign choice can be overridden only through theuse of a domain specific plug-in module, mentionedwhen discussing Desideratum D-10). However, panelscan bemoved around freely, and even overlaid on topof each other. Zooming into and out of temporalgranularities or particular time periods is appliedto all panels simultaneously, thus preserving tem-poral alignment among all data and derived con-cepts, unless a panel is zoomed into and magnified,or unless a panel is asynchronized from the otherpanels, by using the time-synchronization function

(pin-shaped) icon, which toggles between synchro-nizedpanningof all panels or individualmotion of anypanel in which the pin is removed (see Fig. 3).

3.3. D-3: temporal granularity

KNAVE-II implements five operators (zoom-in func-tions) for manipulating temporal granularity: (a) auser defined granularity zoom enables specificationof any desired temporal granularity (e.g., year,month); (b) a calendric-range zoom uses a standardcalendar function to enable the user to specify thestart and end time points to zoom-in into a specificabsolute time range; (c) a single-panel zoom opensand magnifies a particular panel displaying raw dataor a derived concept in a separate sub-window; (d) atime-granule-sensitive zoom enables users to selecta specific predefined period of time of a predefinedgranularity, within the timeline of a particular panel(e.g., ‘‘August 1995’’) by clicking on that granule(e.g., click on ‘‘August’’ within the ‘‘1995’’ time line)and (e) a content-based zoom to mark-up specificcontents in the panel (e.g., a group of bone-marrowtoxicity abstractions of grades 2 and higher) and thenzoom into the temporal range implicitly determinedby the selected time range, whether that rangedefines any predefined complete temporal-granular-ity unit or not. Fig. 4 demonstrates the random-granularity, calendric-range, and time-granule-sen-sitive zooms, and Fig. 5 presents an example of thecontent-based zoom.

3.4. D-4: absolute and relative time lines

Changing dynamically the point of view from anabsolute (calendar-based) time line into a relativetime line is another KNAVE-II innovative capability.The relative time line is set by identifying clinicallysignificant events in the domain’s temporal-abstrac-tion ontology (e.g., bone-marrow transplantationtime, start of chemotherapy, birth of the child)which serve as a date of reference (time zero) toall the other displays. Once the relative time-line hasbeen selected the time display will change to� timeunits starting from that event, based on the timegranularity selected (hours, days, months, years).The user can interactively select the event to beused as the zero-time reference, through access tothe predefined list of potential reference points(Fig. 6).

3.5. D-5: intelligent exploration of rawdata and abstractions

Exploration of raw data and abstract conceptsincludes navigation along semantic links in the


Figure 3 A view of an individual oncology patient’s data in the KNAVE-II system. On the left hand side, a browser to theclinical domain’s ontology, coming from the ontology knowledge base, is shown. The user selects a rawdata type (third, fifthand sixth panels from the top) or an abstract concept (first, second and fourth panels) by clicking on a node in the browser’sontologytree,forexample,theWBCStateappears inthe4thpanel fromthetop.Theusercanalsosearchforaconceptusingastandard string-search function, as shown in the left hand bottompanel; the concept is then retrieved an, if the user doubleclickson it, computedontheflyanddisplayedasapanelonthe righthandside.Explorationoperators, representedas icons ineachpanel, enable the user to performactions such as: (a) re-align the display of the panels, using the time-synchronizationfunction (pin-shaped) icon, which toggles between synchronized panning of all panels or individual motion of any panel inwhich the pin is removed; (b) query the knowledge used to derive the concept through the ‘‘kb’’ icon; (c) add statisticsregardingrawandderivedconcepts,byclickingonthestatistics (graph) iconbelowthekb icon(seethestatisticsdisplayedonfirst, third and fifth panels from the top, respectively); (d) semantically explore the concept and the semantic network ofrelations around it, by clicking on the semantic-explorer (cross) iconbelow the statistic icon; (e) skip to thenearest period inthe past or future in which data can be found, using the left and right arrows, respectively. A set of top-level (menu) globalwidgets above the top panel controls all panels: (f) the random granularity zoom enables slide-bar zooming to any desiredtemporal granularity; (g) the calendric-range zoom enables zooming into a time range, by specifying the start and end timepoints, using a standard calendar; (h) the patient-selection box enables the user to select the current patient-record toexplore; (i) theglobal-statisticsbuttonaddsorremoves statistical informationtoorfromallpanels; (j) theabsolute/relativetime line functions enable the user to set a specific event (such as a particular type of medical intervention) as the date ofreference (time zero) for all the other displays, by selection from a list of predefined reference points; (k) The search andretrieval service enables lexicographic search, by typing a string in the input window on the bottom left-hand part of theinterface. The search retrieves all the related concepts from the domain’s temporal-abstraction ontology. Clicking on theFind Similar checkbox triggers retrieval of similar-sounding concepts without requiring the user to have prior knowledgeabout theexact formaconceptappears in theontology knowledgebase.The retrievedconceptscanbeorderedaccording totheir type and related context (e.g., post-BMT), and then opened in either the semantic explorer, to explore their semanticrelations or properties, or (after computing the concept for the current patient) as a separate panel on the right hand side ofthe screen, for visualization and further exploration.

domain’s temporal-abstraction ontology, such asabstracted-from relations, using the semanticexplorer (Fig. 7). For any domain, the semanticsof the query, visualization, and exploration pro-

cesses are the same, since these processes usethe terms of the domain-independent knowledge-based temporal-abstraction ontology [1]. All con-cepts in the temporal-abstraction ontology have an


Figure 4 Three of the operators for changing temporal granularities: (a) the random granularity zoom uses apredefined scale of temporal granularities; (b) the calendric-range zoom opens a calendar and enables specificationof start and end time points to zoom into; (c) the time-granule-sensitive zoom enables users to zoom into a specificpredefined period of time, by clicking on any specific temporal granularity button within the timeline of a particularpanel, e.g., zooming into September 1995 by clicking on the ‘‘Sep’’ button in the monthly granules above the ‘‘1995’’ bar.Zooming back into the whole of 1995 is then accomplished by clicking on the ‘‘Sep 95’’ bar. The ‘‘1995’’ bar will thenreappear, above which would be the monthly granules, as shown in the upper panel.

is-a relation, which enables the user to explore theclass the concept belongs to and its other siblingswithin that class, or the sub-concepts the concept infocus subsumes, by going up or down, respectively,in the type hierarchy of the semantic explorer’slocal knowledge browser (the full version of whichis also displayed on the left side panel of the mainscreen, see Fig. 3). In Fig. 7, we can see thatPLATELET_STATE_BMT is-a BONE-MARROW_STATE,and by further climbing in the knowledge browserhierarchy we could have found that it is-a STATE.However, for any domain, the main semantic-exploration operators use the domain-specific con-tents of the relations of the temporal-abstractionontology, which depend on the type of entityexplored (these relations are referred to in Fig. 8

Figure 5 The content-based zoom enables users to mark sptemporal-granularity unit or not (see shadowed area), and thethese contents. The user indicates the temporal region of intarea in (a)), which then is expanded to fill the whole panel (

as ‘‘special relations’’). The result is a uniform-behavior, but context-sensitive (with respect tothe knowledge) visualization and exploration inter-face in all time-oriented domains. We have definedfour types of special semantic relationships of aconcept in the knowledge-based temporal ontology:a ‘‘meta-children’’ relation, such as abstracted-from in the case of parameters, or parts in the caseof events; a ‘‘meta-parents’’ relation, such asabstracted-into in the case of parameters, orpart-of in the case of events; a ‘‘meta-siblings’’relation, such as the other arguments in the functiondefining the parameter, or other components on thepattern into which the current entity is abstracted,or other parts of the event which the current eventin focus is a part of; and a ‘‘context relation’’, such

ecific contents in the panel, whether within a completen zoom into the temporal range implicitly determined byerest by marking-up the range of interest (see shadowedb).


Figure 6 Absolute/relative time lines in KNAVE-II. Once the relative time-line was set by selecting the Allogenic Bone-Marrow transplant (BMT-Al) as the reference point the time display will change to � units starting from that event. Theselection of the time reference event can be done by direct manipulation or by selecting a predefined reference eventfrom the knowledge-base, in which case KNAVE-II will show the nearest event enabling direct browsing between events(in the case that there were more than one such points in the patient’s record). Here, both the Hemoglobin (HGB) rawdata and its abstraction into the State of Hemoglobin (HGB-State-BMT) is displayed in granularity of months relative tothe BMT-Al event. Clicking on the ‘‘Open’’ button opens a panel, in which the reference event is displayed (see bottompanel). As in the case of all raw data and concepts, moving the mouse over the event opens a tool tip baloon withinformation about the contents of the displayed entity.

Figure 7 Exploration of data and knowledge in KNAVE-II. The semantic explorer is evoked by clicking on the explorationbutton of the panel (see Fig. 3). The user uses the semantic relationships of a concept, which depend on its type (e.g.,abstracted-from, abstracted-into, siblings, created context, and is-a, in the case of a raw or abstract data type) tonavigate to other concepts semantically related to the original concept. The ‘‘special relations’’ are emphasized here bya dashed-lines boxed area.


Figure 8 Knowledge-based explanation, evoked by clicking on the knowledge-base (kb) button in a panel (see Fig. 3).The user thus examines the temporal-abstraction knowledge that was used to derive a specific displayed concept. Notethat the user can click on the ‘‘Persistence Function’’ tab to explore the interpolation function used to join separate datapoints or intervals into a longer interval. In the case of a raw-data type, the explanation would consist on only thestandard vocabulary used (e.g., LOINC), the key in that vocabulary, and the measurement units.

as the generated-context relation for most entities,or the generated-from relation in the case of con-texts themselves. These special relations depend onthe type of concept in the current focus and enablethe user to navigate to other concepts semanticallyrelated to the focused concept. Table 1 summarizesand briefly describes the types of special semanticrelationships.

3.6. D-6: explanation

We added the capability for browsing the contentsof the domain-specific knowledge base relevant tothe derivation of each panel in a context-sensitivemanner. During exploration, the user is able toobtain context-sensitive explanations to questionssuch as ‘‘From which data is this conceptabstracted?’’, by using the semantic explorer, andmoving from a derived concept into its components;

Table 1 Types of special semantic relationships used for in

Type Relation

‘‘Meta-children’’relation

‘‘Meta-parents’’relation

Parameter Abstracted-from Abstracted-into

Event Parts Parts-of

Pattern Components Components-of

Context Sub-context Super-context

and to questions such as ‘‘What classification func-tion defines this abstraction?’’ by clicking on theknowledge-base icon in each panel and then focus-ing on the type of knowledge of interest to the user,such as the table that maps raw data into thebrowsed concept (Fig. 8). In the case of a raw-datatype, such as a Hemoglobin value, the ‘‘knowledgebase’’ explanation would consist of only the stan-dard term used to map that concept into the specificvocabulary (e.g., LOINC), the value of the key inthat vocabulary (e.g., LOINC code), and the mea-surement units (e.g. grams per 100 cc).

3.7. D-7: statistics

To support clinical research, it is imperative toprovide several types of descriptive statistics aspart of the interactive visualization and explora-tion. Statistics in KNAVE-II can be computed and

telligent exploration of concepts

‘‘Meta-siblings’’ relation Context relation

Other parameters abstractedinto the ‘‘parent’’ parameter

Generated contexts

Other parts belonging to the‘‘parent’’ event

Generated contexts

Other components definingthe ‘‘parent’’ pattern

Generated contexts

Other sub-contexts ofthe super-context

Generated from


displayed for either raw data or abstracted para-meters. The computation of statistics is sensitive tothe particular time window displayed in each panel,and thus changes dynamically when the contents ofthe panel are changed. Default statistics for rawdata types include descriptive statistics such asmean, maximum, minimum, �standard deviation,etc. (see third panel from the top in Fig. 3). In thecase of abstract data types, the default statisticsdisplayed are a detailed distribution (in absolutetime periods and relative proportion to all displayedvalues) of the total duration of the intervals overwhich each value of the abstraction (e.g., GRADE-IIbone-marrow toxicity) held within the particulartime window dynamically selected or, the temporalgranularity (e.g., month of September) zoomed into(see first panel from the top in Fig. 3). (In addition,the absolute and relative start and end time andduration of each interval are displayed within thetool tip when passing over the interval.)

3.8. D-8: search and retrieval

KNAVE-II supports easy and fast search and retrievalof ontology-based clinically-significant concepts.Besides the possibility for actively looking withinthe knowledge browser (see left hand frame inFig. 3), which directly exploits the access to thedomain’s ontology, or using the semantic explorer tonavigate within that ontology, by starting from arelated term (see Fig. 7), it is possible to use a text-based search to find any arbitrary concept. Thesearch and retrieval service (at the left bottom sideof the main interface, see Fig. 3), enables a text-string lexicographic search. The search retrieves allthe related concepts from the domain’s temporal-abstraction ontology. The Find Similar function trig-gers retrieval of similar-sounding concepts withoutrequiring the user to have prior knowledge about theexact form a concept appears in the ontology knowl-edge base. The retrieved concepts can be orderedaccording to their type and the related context(such as BMT), and then opened in either the seman-tic explorer to explore their properties and relatedconcepts, or displayed (after computing the conceptfor the current patient whose data is examined),within a separate panel for visualization and furtherexploration.

3.9. D-9: dynamic sensitivity analysis

The exploration functionality offered in KNAVE-IIsupports, among other features, dynamic simulationof hypothetical modifications to raw data. By usingthe tight link to the underlying computational cap-abilities of the IDAN temporal-abstraction mediator,

which can propagate the effects of any data mod-ifications to the resultant abstractions, the user isable to simulate the effect of modifying the data,thus adding a dynamic sensitivity analysis (‘‘What-ifdynamic simulation’’) capability by modifying,deleting, or adding values within a specific panel.Exploiting the direct access to the domain’s tem-poral-abstraction ontology and to the temporalabstraction server enables the simulation-basedexplanations (see Fig. 9a—c). The display reflectsthe computational implications of these modifica-tions in all the raw data and abstracted conceptsrelated to the modified data. When the dynamicsensitivity analysis is enabled the system accesses,through the IDAN mediator, a simulation state inwhich the modified values are kept in the cache anddo not affect the real patients’ data in the database, thus enabling easy reversibility of the compu-tations and their effects in the cache, and a returnto the state before any simulated modification wasdone. An indication for modification of the conceptis displayed (see Fig. 9b) and navigation throughviews with varying degrees of modification of theunderlying data can be performed using operatorsthat control the underlying computational propaga-tion process, such as add, delete, and modify, aswell as apply, restore, undo and redo (Fig. 9b). It isalso possible to open another copy of the patient’srecord and compare the hypothetically modifiedconcepts with the actual ones.

3.10. D-10: clinical-task support

D-10: clinical-task support is achieved by enablingthe physicians and medical researchers to easilybrowse several raw-data and abstract-conceptpanels that are all related to a specific clinical task(e.g., monitoring of diabetic patients). KNAVE-IIenables customization of the displays for each spe-cific clinical task, by enabling the user to save, atany point during the exploration, all the currentpanels as a new profile (e.g., a diabetes profile, or ahematological profile, a useful abstraction in thecase of the oncology domain). The selection of aspecific profile applies it to the current selectedpatient record. In addition we have added function-ality for customizing the interface itself for parti-cular clinical domains or tasks, by adding a domain-specific plug-in module associated with each knowl-edge base. For example, the tool-tip can displayadditional data types, but still within the sameoverall framework and while still supporting all ofthe usual exploration operators, For example, it wasoften found useful to display multiple attributes of aconcept and not just one value, as in the case ofdisplaying medications, so as to show as one value


Figure 9 (a) The Dynamic sensitivity analysis capability of KNAVE-II, supported by the underlying IDAN architecture,enables the user to add hypothetical raw data, modify the existing data by changing their values and time-stamps ordelete selected data. By clicking on the Enter Simulation Mode button the menu frame turns into pink to indicate to theuser that simulationmodewas selected (b). (b) Performing a dynamic sensitivity analysis within the simulationmode. Theuser added three platelet data points between 10 and 12 of August 95. When the user modifies data in the simulation


the dose, the mode, the administration frequency,etc.; or in the case of various attributes of embryosfollowed in the fertility clinic.

3.11. D-11: collaboration

Clinicians and researchers usually like to consult orshare the result of the exploration of data andabstractions with colleagues. KNAVE-II enables col-laboration by saving the selected data and abstrac-tions of a particular exploration to a specialexploration file format. The exploration files encap-sulate the relevant data, concepts, knowledgeclasses, and display format appearing within a com-plete set of panels (e.g., all the hematological dataand abstractions viewed, the relative timeline thatwas used to explore them, and the knowledge usedto create the concepts). An exploration file can besaved to a shared directory or sent by email to acollaborating colleague. The collaborator can openthe file and explore the same data and knowledge,starting from the same point in which the image ofthe exploration was saved. She also can add otherraw-data and abstract-concept panels that seem tobe relevant to the case in discussion, and continuethe visualization and exploration session (and evensend it back to the original clinician or researcher).An exploration file can be visualized and exploredoff-line by another KNAVE-II module, even withoutaccess to the mediator, the KB and the DB.

3.12. D-12: documentation

Standard clinical, research and administrative pro-cedures require documentation of the patient’sclinical data to the patient’s file, thus showingthe exploration that supported a clinical decision.KNAVE-II enables taking a snapshot of the explora-tion process, saving it as a documentation file thatcannot be further manipulated. In addition, weadded the capability of saving the current explora-tion as an exploration file, as explained above,which supports documentation of the explorationprocess while preserving the option of furtherexploration in the future to better understand thereason that a particular interpretation was derived.This dynamic snapshot is enabled by saving thecurrent state of the application that includes theselected displayed panels but also all the related

mode a Modified status appears in the upper right hand sidecomputation of abstractions derived from the modified raw dtheir effects. (c) Assessing the effects of the dynamic sensitivitthe computation of the platelet_state_BMT abstract conceptdata (as an interval on the same dates). By clicking on the Exdata and their effects in the cache, and returns to the state

values of the raw data and the abstracted conceptsactive at any selected time span.

4. Evaluation

A preliminary evaluation of the overall distributedarchitecture, and in particular of the currentexploration interface, had been carried out by ourcollaborators in the Palo Alto, CA, USA, VeteransAdministration Health Care Center, and is describedin detail elsewhere [69].

The temporal-abstraction service was running onone of the Ben-Gurion University (BGU), Israel, BeerSheva, servers. For the purpose of the evaluation,the knowledge service and data service were run-ning on other BGU servers.

The evaluation used an online retrospective data-base of more than 1000 unidentifiable laboratory-test records of bone-marrow transplantationpatients who were followed for 2—4 years at theRush medical center, Chicago.

The knowledge source used for the evaluationwas an oncology knowledge base specific to thebone-marrow transplantation domain, previouslyelicited from one of our colleagues at StanfordUniversity [23].

The subjects included eight clinicians with vary-ing medical background and experience in the use ofcomputers. Each user was given a brief (10—15 min)demonstration of the KNAVE-II interface. Each userwas asked to answer 10 queries common in oncology,about individual patients, at increasing difficultylevels; queries were inspired by typical oncologyprotocols. For example, from a protocol sentencesuch as ‘‘if after the bone-marrow transplantationthere was a period of bone-marrow toxicity of gradetwo or more within the past 2 months, attenuate thedose of the drug by 20%’’ the following query can beproduced, a query that appeared in our study withinthe set of queries of hard, but not the hardest levelof difficulty: ‘‘Give the starting and ending dates ofthe last period of grade 3 myelotoxicity followingthe bone-marrow transplantation’’ [69].

A cross-over study design compared the KNAVE-IImodule versus two existing methods: paper chartsand an Excel electronic spreadsheet. Each useranswered equivalent questions using all three meth-ods in randomized order.

of the pannels involved. The Apply button triggers theata, while the Undo button cancels the modifications andy analysis. By clicking on the Apply button the user triggers, which is dynamically abstracted from the modified rawit Simulation Mode button, KNAVE-II deletes all simulatedbefore any simulated modification was done (a).


The measures used for the evaluation includedquantitative measures such as the time to answerand accuracy of responses, and qualitativemeasuressuch as the Standard Usability Score (SUS) [70] andcomparative ranking of all tools by usability.

The results were quite encouraging. In the direct-ranking comparison, KNAVE-II was ranked first inpreference by all users, along all dimensions (suchas ease of use), except, of course, for familiarity inusing the tool (an aspect introduced for validation).

The SUS mean scores (the overall SUS range is 0—100) were 69 for KNAVE-II, 48 for Excel, 46 for Paper(P = 0.006) (over 50 is usually considered as afriendly interface).

With respect to quantitative measures of thetime to answer: users were significantly faster usingKNAVE-II as the level of difficulty increased, up to amean of 93 s per query faster versus paper and 49 sfaster versus Excel, for the hardest query (time toanswer queries ranged from 5 to 300 s; P = 0.0006).

Regarding the correctness of the answers, usingKNAVE-II significantly enhanced correctness versususing paper, especially as level of difficulty increased(P = 0.01); the comparison with Excel showed a simi-lar trend.

5. Discussion

We have introduced KNAVE-II, a fully implementedintelligent visualization interface, which is able touse the advanced features of a distributed archi-tecture for intelligent visualization and explorationof time-oriented data. The computational architec-ture underlying KNAVE-II supports the generationand exploration of context-sensitive interpreta-tions (abstractions) of the time-stamped data interms of domain-specific concepts and temporalpatterns.

Thus, two of the major innovations in the KNAVE-II architecture are:

(1) D
irect access to and continuous use of a set oftemporal-abstraction computational mechan-isms (based on the KBTA method), which sup-ports real-time generation (and, thus,visualization) of domain-specific meaningfulconcepts and patterns, by applying domain spe-cific knowledge to the data. Unlike previoussystems, which focused mainly on visualizationof raw data and various statistical functions ofit, we focus mainly on visual display of inter-pretations (temporal-abstractions) of the time-oriented data. We consider that capability to bemore useful for clinical decision support,although we do support the application, in uni-
form fashion, of all exploration operators forboth raw and abstract concepts.

(2) T
he capability for direct access to one or moreof the domain-specific knowledge-bases usedfor the abstraction, which enables the user ofthe KNAVE-II system to interactively explore theabstractions and the raw data alike, by naviga-tion along the semantic links implicitly definedby the knowledge roles of the semantic networkthat is derived from the domain’s temporal-abstraction ontology.
Moreover, the KNAVE-II general architecture istask-specific (in contrast to domain specific) andhighly scalable and modular, and can be appliedto time-oriented databases in any clinical domain(or even in a non-clinical domain; the underlyingcomputational temporal-abstraction methodologyhad been also applied to the task of traffic control,for example [71]). Using KNAVE-II requires adesigner to link the IDAN controller to the relevantdata base(s) and to the domain specific knowledgebase, thus enabling the application of domain-spe-cific temporal abstraction knowledge to each rele-vant database.

We have started by listing multiple desiderata forsupporting the needs of care providers, quality-assessment professionals, and clinical researcherswho need to browse large amounts of time-orientedclinical data and reduce the cognitive burdeninvolved in that task. Our key insight is the additionof an interactive intelligent interface supported byan architecture that automatically integratesknowledge and data. One of the core needs of aframework for intelligent data analysis is the capa-city to explore dynamically both raw data and theinterpretations derivable from these data (usingdomain-specific knowledge); and perform this taskin a scalable fashion.

Initial applications of KNAVE-II and its supportingknowledge-based distributed temporal-mediationarchitecture, to a large data base of patients mon-itored several years after bone marrow transplanta-tion (BMT), have provided highly encouragingresults. KNAVE-II was ranked first in preference byall users, when compared with a computerized Excelelectronic spreadsheet and with printed versions ofExcel, without having been familiar with this toolprior to the experiment. Usability evaluation indi-cated a significantly better support for the explora-tion and visualization of time oriented clinical data.The usability scores were significantly higher forKNAVE-II, compared to Excel and to paper charts.Users also performed better using KNAVE-II thanwith the other two tools. Users of KNAVE-II weresignificantly faster, compared their use of standard


tools, as the level of difficulty increased; note that amean difference of 93 s per query versus paper and49 s versus Excel, for the hardest query, translatesinto many hours that might potentially be saved perphysician or per clinical researcher each week, forexample. Using KNAVE-II significantly enhanced cor-rectness versus using paper, especially as the levelof difficulty increased; the comparison with Excelshowed a similar trend, which we expect to behighly significant as we continue to compare thetools across increasingly difficult queries. Theenhanced accuracy provided by the IDAN/KNAVEarchitecture translates into better support for run-time quality assurance and retrospective qualityassessment.

6. Future work

Our current plans for future work involve enhance-ment of the architecture in several ways:

1. D
eveloping the computational and graphical cap-abilities for visualization and exploration ofaggregations of patients selected according todynamic criteria. For instance, exploration ofthe distribution of the duration and severitylevels of bone-marrow toxicity episodes, for allpatients who received a certain chemotherapyprotocol. This functionality will extend the sup-port that currently we provide for D-2 and D-5 forvisualizing and exploring raw data andabstracted knowledge from one patient to entirepopulations.
2. E
nhancing the generic capability for addition ofcustomized plug-in modules that extend the cap-abilities of KNAVE-II. For example, wementionedthe need for display of multi-attribute para-meters required for specific clinical tasks (D-10). Examples include listing the various attri-butes involved in the administration of medica-tions (dose, mode, frequency, etc.), and thepotential need for overriding the current defaultof displaying each concept within a separatepanel, which might sometimes be modified, aswhen displaying systolic and diastolic blood pres-sures together. Such modifications are typicallydomain or task-specific, and can be achievedthrough a plug-in module that capitalizes onthe significant powerful operators that arealready a part of KNAVE-II. We would like tocreate a library of such plug-ins and facilitatenew creation or modification by local knowledgeengineers.
3. S
upporting on-the-fly specification (at explora-tion time) of new temporal patterns, possibly
saving them as part of the user’s profile in addi-tion to the exploration of the data using prede-fined concepts and patterns from the domain’stemporal-abstraction ontology combining thesupport currently provided for requirementsexploration of raw data and abstracted concepts(D-5) and dynamic sensitivity analysis (D-9).

4. A
dding the capability for simulation of theeffects of dynamic modifications of thedomain-specific knowledge, not only of the data,as part of the dynamic sensitivity analysis (D-9).
5. E
xperimenting with a natural language inter-face, and even a speech recognition interface,both capitalizing on the underlying rich ontologyand semantic network embodied in eachdomain’s temporal-abstraction ontology. Weexpect this capability to facilitate the interac-tion with the multiple functionalities of theKNAVE-II user interface (D-1 to D-12) by enablingthe manipulation of all the GUI actions by simplyissuing voice commands.
We are in the process of performing additionalevaluations of the overall architecture, using a morediverse set of users and a more complex set ofqueries; we also intend to evaluate the cognitiveand task-specific effects of the more advanced fea-tures of the KNAVE-II system,which aremainlymean-ingful for complex tasks such as clinical research,quality assessment, or repetitive management oflarge numbers of patients. These include featuressuch as dynamic sensitivity analysis and the creationand application of clinical task-related profiles.

Acknowledgements

This research was supported in part by NIH awardno. LM-06806. We thank Drs. Mary Goldstein, SusanaMartins, Lawrence Basso, Herbert Kaizer, AneelAdvani, and Eitan Lunenfeld, for their useful com-ments regarding the KNAVE-II interface and for theirassistance in its evaluation. Dr. Martins was espe-cially helpful in conducting the KNAVE-II evaluation.We would also like to thank many of the researchstudents at the Ben Gurion Medical InformaticsResearch Center for their efforts regarding theimplementation of multiple aspects of the IDAN,ALMA, and KNAVE-II architectures.

References

[1] Shahar Y. A framework for knowledge-based temporalabstraction. Artif Intell 1997;90(1—2):79—133.


[2] McDermott D. A temporal logic for reasoning about processesand plans. Cogn Sci 1982;6(2):101—55.

[3] Allen J. Towards a general theory of action and time. ArtifIntell 1984;23:123—54.

[4] Shoham Y. Temporal logics in AI: semantical and ontologicalconsiderations. Artif Intell 1987;33(1):89—104.

[5] Downs S,Walker M, Blum R. Automated summarization of on-line medical records. In: MEDINFO’86: the Fifth Conferenceon Medical Informatics; 1986.p. 8000—804.

[6] De Zegher-Geets, I. IDEFIX: intelligent summarization of atime-oriented medical database. M.S. dissertation. Programin Medical Information Sciences. Stanford University Schoolof Medicine; 1987. also Knowledge Systems Laboratory Tech-nical Report KSL-88-34. Department of Computer Science:Stanford University, Stanford, CA; 1988.

[7] Kohane I. Temporal reasoning in medical expert systems. In:Technical report. Cambridge, MA: Laboratory of ComputerScience, Massachusetts Institute of Technology, 1987.

[8] Russ, T. Using hindsight in medical decision making. In: TheThirteenth Annual Symposium on Computer Applications inMedical Care. Washington, DC: IEEE Computer Society Press;1989. p. 38—44.

[9] Kahn M. Combining physiologic models and symbolic meth-ods to interpret time-varying patient data. Methods InformMed 1991;30:167—78.

[10] Haimowitz I, Kohane I. Automated trend detection withalternate temporal hypotheses. In: The Thirteenth Interna-tional Joint Conference on Artificial Intelligence; 1993.p.1460—151.

[11] Fagan, L. VM: representing time-dependent relations in amedical setting. Ph.D. dissertation. Department of Compu-ter Science: Stanford University, Stanford, CA; 1980.

[12] Miksch S, Horn W, Popow C, Paky F. Time-oriented analysis ofhigh-frequency data in ICU monitoring. In: Lavrac N, Ker-avnou E, Zupan B, editors. Intelligent data analysis in med-icine and pharmacology. Kluwer; 1997. p. 17—36.

[13] Salatian A, Hunter J. Deriving trends in historical and real-time continuously sampled medical data. J Intell Inform Syst1999;13:47—71.

[14] Combi C, Pinciroli F, Pozzi G. Managing different time gran-ularities of clinical information by an interval-based tem-poral data model. Methods Inform Med 1995;34:458—74.

[15] Keravnou E. A multidimensional and multigranular model oftime for medical knowledge-based systems. J Intell InformSyst 1999;13(1—2):73—120.

[16] Musen M, Tu SW, Das AK, Shahar Y. EON: a component-basedapproach to automation of protocol-directed therapy. JAmer Med Assoc 1996;3(6):367—88.

[17] Shahar Y, Miksch S, Johnson P. The Asgaard project: a task-specific framework for the application and critiquing oftime-oriented clinical guidelines. Artif Intell Med1998;14:29—51.

[18] Shahar Y. Dynamic temporal interpretation contexts fortemporal abstraction. Ann Math Artif Intell 1998;22(1—2):159—92.

[19] Shahar Y, Cheng C. Intelligent visualization and explorationof time-oriented clinical data. Topics Health Inform Manage1999;20(2):15—31.

[20] Shahar Y, Cheng C. Model-based visualization of temporalabstractions. Computat Intell 2000;16(2):279—306.

[21] Cousins S, Kahn M. The visual display of temporal informa-tion. Artif Intell Med 1991;3:341—57.

[22] Shahar Y, Musen M. Knowledge-based temporal abstractionin clinical domains. Artif Intell Med 1996;8(3):267—98.

[23] Shahar Y, et al. Semiautomated acquisition of clinical tem-poral-abstraction knowledge. J Amer Med Inform Assoc1999;6:494—511.

[24] Chakravarty S, Shahar Y. CAPSUL: a constraint-based speci-fication of repeating patterns in time-oriented data. AnnMath Artif Intell 2000;30:3—22.

[25] Chakravarty S, Shahar Y. Specification and detection ofperiodicity in clinical data. Methods Inform Med2001;40(5):410—20.

[26] Nguyen JH, Shahar Y, Tu SW, Das AK, Musen MA. Integrationof temporal reasoning and temporal-data maintenance intoa reusable database mediator to answer abstract, time-oriented queries: the Tzolkin system. J Intell Inform Syst1999;13(1—2):121—45.

[27] Larkin J, Simon H. Why a diagram is (sometimes) worth tenthousand words. Cogn Sci 1987;11:65—99.

[28] Powsner S, Tufte E. Graphical summary of patient status.Lancet 1994;344:386—9.

[29] Tufte E. Envisioning information. Graphics Press; 1990.[30] Tufte E. Visual Explanations. Graphics Press; 1997.[31] Plaisant C, Milash B, Rose A, Widoff S, Shneiderman B.

Lifelines: visualizing personal histories. In: Proceedings ofthe CHI’96. Vancouver BC: ACM Press; 1996. p. 221—7.

[32] Plaisant C, Mushlin R, Snyder A, Li J, Heller D, ShneidermanB. LifeLines: using visualization to enhance navigation andanalysis of patient records. In: American Medical InformaticsAssociation Annual Fall Symposium; 1998.p. 760—80.

[33] Spenke M. Visualization and interactive analysis of bloodparameters with InfoZoom. Artif Intell Med 2001;22(2):159—72.

[34] Chittaro L. Information visualization and its application tomedicine. Artif Intell Med 2001;22(2):81—8.

[35] Falkman G. Information visualisation in clinical odontology:multidimensional analysis and interactive data exploration.Artif Intell Med 2001;22(2):133—58.

[36] Ahlberg C, Shneiderman B. Visual information seeking: tightcoupling of dynamic query filters with starfield displays. In:ACM CHI’94; 1994. p. 313—7.

[37] Ahlberg C, Wistrand E. IVEE: an information visualizationand exploration environment. In: IEEE information visuali-zation’95. IEEE Computer Press; 1995. p. 66—73.

[38] Shneiderman B. Dynamic queries, starfield displays, and thepath to spotfire. In: Research summary. University of Mary-land: Human-Computer Interaction Lab, 1999.

[39] Korn F, Shneiderman B. Navigating terminology hierarchiesto access a digital library of medical images. In: Technicalreport. University of Maryland; 1995.

[40] Card S, Mackinlay J, Shneiderman B. Information visualiza-tion: using vision to think. San Francisco, California: Morgan-Kaufmann, 1999.

[41] Chi E. A framework for visualizing information. The Nether-lands: Kluwer Academic, 2002.

[42] Hearst M. User interfaces and visualization. In: YR B,Ribeiro-Neto B, editors. Modern information retrieval. NY:ACM Press; 2000. p. 257—324 [chapter 10].

[43] Shneiderman B. Dynamic queries for visual informationseeking. IEEE Software 1994;11(6):70—7.

[44] Shneiderman B, Feldman D, Rose A, Ferre’ Grau X. Visualiz-ing digital library search results with categorical and hier-archical axes. In: ACM digital libraries. ACM Press; 2000. p.57—66.

[45] Eick S, Wills G. High interaction graphics. Eur J Oper Res1995;81(3):445—59.

[46] Bederson BB, Hollan JD, Perlin K, Meyer J, Bacon D, FurnasGW. Pad++: a zoomable graphical sketchpad for exploringalternate interface physics. J Visual Languages Comput1996;7:3—31.

[47] Leung Y, Apperley MD. A review and taxonomy of distortion-oriented presentation techniques. ACM Trans Comput-Human Interact 1994;1(2):126—60.


[48] Shneiderman B. Designing the user interface: strategies foreffective human-computer interaction, 3rd ed, Addison-Wesley; 1998. p. 509—49.

[49] Balaban M, Boaz D, Shahar Y. Applying temporal abstractionin medical information systems. Ann Math Comput Telein-form 2004;1:56—64.

[50] Boaz D, Shahar Y. A framework for distributed mediation oftemporal-abstraction queries to clinical databases. ArtifIntell Med 2005;34(1):3—24.

[51] Norman D. The invisible computer why good products canfail the personal computer is so complex and informationappliances are the solution, MIT Press; 1998. p. 23—50, 185—202.

[52] Nielsen J. Usability engineering; 1995 [AP professional].[53] Vredenburg K, Isensee S, Righi C. User-centered design: an

integrated approach. Upper Saddle River, NJ: Prentice HallPTR, 2001.

[54] Chittaro L, Combi C. Representation of temporal intervalsand relations: information visualization aspects and theirevaluation. In: TIME-01: Eight International Symposium onTemporal Representation and Reasoning. Los Alamitos, CA:IEEE Press; 2001. p. 13—20.

[55] Hearst M, Pedersen J. Reexamining the cluster hypothesis:scatter/gather on retrieval results. In: The 19th AnnualInternational ACM SIGIR Conference; 1996.p. 760—84.

[56] Chen H, Houston AL, Sewel RR, Schatz BR. Internet browsingand searching: user evaluations of category map and con-cept space techniques. J Amer Soc Inform Sci1998;49(7):582—608.

[57] Kohonen T. Self-organizing maps, 2nd ed., Berlin: Springer-Verlag, 1997.

[58] Teach R, Shortliffe E. An analysis of physician attitudesregarding computer-based clinical consultation systems.Comput Biomed Res 1981;542—58.

[59] De Bra P. Design issues in adaptiveweb-site development. In:The Second Workshop on Adaptive Systems and User Model-

ing on the WWWof the Eightth International Word Wide WebConference. Toronto, Canada; 1999.

[60] Kappel G, Retschitzegger W, Schwinger W. Modeling custo-mizable web applications — a requirement’s perspective. In:ICDL2000-11th International Conference on Digital Libraries;2000.

[61] Grosz B. Collaborative systems. AI Magazine 1996;17:67—85.[62] Shortliffe E, Patel VL, Cimino JJ, Barnett GO, Greenes RA. A

study of aollaboration among medical informatics researchlaboratories. Artif Intell Med 1998;12(2):97—123.

[63] Association AM. The international classification of diseases,9th Revision, Clinical Modification (ICD-9-CM); 2002. p.1977—2002.

[64] Pathologists CoA. The systematized nomenclature of med-icine clinical terminology. SNOMED CT 2002.

[65] Association AM: current procedural terminology, vol. 2002.[66] Regenstrief Institute, I.a.t.L.O.I.N.a.C.L.C.: logical obser-

vation identifiers names and codes. vol. 2002.[67] Goren-Bar D. Designing intelligent user interfaces: the IUIM

model. In: Mastorakis. Software and hardware engineeringfor the 21th century. World Scientific and EngineeringSociety Press; 1999. p. 79—85.

[68] Goren-Bar D. Designing model-based intelligent dialoguesystems. In: Rossi M, Siau K, editors. Information modelingin the next millennium. Idea Group Publishing; 2001 . p.271—87.

[69] Martins SB, Shahar Y, Galperin M, Goren-Bar D, Boaz D, TahanG, et al. Evaluation of KNAVE-II: a tool for intelligent queryand exploration of patient data. In: Medinfo 2004. SanFrancisco, CA; 2004. p. 648—52.

[70] Brooke J. SUS: A ‘quick and dirty’ usability scale. In: JordanPW, Thomas B, Weerdmeester BA, McClelland IL, editors.Usability evaluation in industry; London, UK: Taylor & Fran-cis, 1996. p. 189—94.

[71] Shahar Y, Molina M. Knowledge-based spatiotemporal linearabstraction. Pattern Anal Appl 1998;1(2):91—104.

Distributed, intelligent, interactive visualization and exploration of time-oriented clinical data...

Documents

Transcript of Distributed, intelligent, interactive visualization and exploration of time-oriented clinical data...