Pictorial analysis: a multi-resolution data visualization ...

Pictorial analysis: a multi-resolutiondata visualization approach for monitoring

and diagnosis of complex systems

V.G. Grishin a, A.S. Sula a, Mihaela Ulieru b,*

a View Trends Ltd., ElectroCardioVision, Inc., 250 Chatham Way, Ste. 845, Cleveland,

OH 44124, USAb Faculty of Electrical and Computer Engineering, The University of Calgary,

2500 University Drive NW, Calgary, Alberta, Canada, AL T2N 1N4

Received 10 March 2001; accepted 27 November 2002

Abstract

This paper aims to describe the prospective methodology (Pictorial Analysis) of a

human–computer interaction capable to select and adjust visual representations for

better feature and pattern selection. As an adequate tool for on-line monitoring and

diagnosis of very complex systems which require human supervision in addition to the

computerized one, our approach exploits human capabilities of pattern recognition in

doing data analysis and selecting appropriate representations, features and class de-

scriptions by means of interactive human–computer learning. It is based on the analysis of

multidimensional relations through transforming the initial data (heterogeneous arrays,

signals and fields) into artificial pictures. Practical applications of this approach prove it

extremely useful in critical areas of safety, such as flight control, power plant moni-

toring, etc.

� 2003 Published by Elsevier Science Inc.

Keywords: Data modeling; On-line monitoring; Process safety; Data visualization;

Interactive human–computer learning; Fuzzy sets; Neural networks

Information Sciences 152 (2003) 1–24

www.elsevier.com/locate/ins

*Corresponding author. Tel.: +1-403-220-8976; fax: 1-403-282-8406.

E-mail addresses: [email protected] (V.G. Grishin), [email protected] (M. Ulieru).

URL: http://isg.enme.ucalgary.ca/people.

0020-0255/03/$ - see front matter � 2003 Published by Elsevier Science Inc.

doi:10.1016/S0020-0255(03)00044-6

mail to: [email protected]

http://isg.enme.ucalgary.ca/people

1. Introduction

It has often been said that a picture is worth a thousand words or a thou-

sand numbers as the case may be. This is the basic premise behind the ap-proach outlined in this article for presenting vast amounts of process

information to operators or researchers that must solve complex decision

making problems associated with a system (plant, engine, process) model

choice as well as with monitoring, diagnosis and control tasks in industrial

systems [1,2].

Visual representations of data are very important for the automation and

control of large-scale industrial processes, such as fossil fuel or nuclear power

generation and distribution systems, as well as stand alone gas-turbine engines,powerful pumps for gas or oil pipelines, chemical, mining and metallurgical

processes and many other complex systems of different industries. In the de-

velopment of a control system for such complex processes one needs to develop

adequate models that enable quick visualization of any critical misbehavior.

Decision-making and control tasks are often organized in a hierarchical

configuration that includes direct level and supervisory level decision and

control functions. Direct level control functions are intended to respond to

disturbances and commands in the natural time-scale of the process to main-tain safe and acceptable system performance. The decision-making and control

tasks in the supervisory level can include planning for operation, process

monitoring, fault detection and isolation, and adaptive and self-organizing

control functions.

Human operators are usually not involved at the direct level of the auto-

mation and control hierarchy but they are an integral part of the supervisory

automation and control functions, especially during abnormal system opera-

tion, for example during large transients resulting from disturbances, equip-ment malfunctions and emergency operating situations [3,14].

Typically, control systems include distributed sensors, actuators and control

processors supplying a vast amount of data on control panels monitored by

human operator. Consequently, one of the important goals of control system

design is a synthesis of visual representations that can on one side underline the

distinction between normal and abnormal situations and on the other side

uniquely identify each abnormal state of the controlled process. To achieve this

goal we need an adequate model capable to outline the variables which candiscriminate/differentiate among these behaviors. These models are higher level

models than the ones for the direct level control and they do not require the

same level of precision.

In this article we introduce a data visualization technique, which we named

pictorial analysis, suitable for on-line monitoring and diagnosis of complex

systems. The method proved useful in the following circumstances which will

be exposed in the sequel:

2 V.G. Grishin et al. / Information Sciences 152 (2003) 1–24

1. in process monitoring and equipment diagnostics to specify the set of differ-

ent important states requiring human operator decision making and how

these states connect to the same parameters;

2. in direct level control to find how important ‘‘outputs’’ of the process, forexample product quantity and quality, environment pollution, etc. depend

on a system�s measured parameters, including its ‘‘inputs’’,

2. A review of data analysis tasks involved in monitoring and diagnosis and their

geometrical interpretation

2.1. Model selection for state monitoring and equipment diagnostics

For both monitoring and diagnosis the sought model usually has to connect

some limited set of process or system states, possible faults or equipments mal-

functions to the limited amount of classes fXigk of measured parameter vectorsor matrices Xi. These states, faults, etc. are selected by experts or by experimental

data to provide a required effectiveness, reliability and safety of the process.

Two types of data units are useful: primary data units and sample structures.

2.1.1. Primary data units

In the case of a static system, monitoring and control are completely spec-

ified by the current values of system parameters. If xd denotes the value of the

nth measured parameter of an system, then its state will be described by meansof a vector Xi ¼ fxdðiÞ; d ¼ 1; 2; . . . ;Dg. Possible variations of a state could berepresented by these vectors collection fXig, i.e. a matrix of generalized state.Examples of simple representations of these data vector Xi are given in Fig. 1.

These are different images of just one sample of a gas-turbine engine state.

The current state of a dynamic system is defined by some history of para-

meter dynamics (e.g. a technological process, market trends). In this case, each

state of a system, instead of a vector Xi, is specified by a data matrix Xi with a

separate row for the time series of each parameter value (temperatures, pres-

sures, flows, etc.). This matrix is ‘‘the digital image of a state sample’’ (see Fig. 2).

2.1.2. Data sample structures

Pattern recognition learning: So-called learning sample [1,5,17,20] is made of

K classes (collections) of vectors or matrices for dynamic systems ffXigkg,k ¼ 1; 2 . . . ;K, where k denotes the sequential number of the class and i denotesthe specific realization of the class. Every class consists of data vectors which

are known to be in similar system states for a particular operating condition

that is to be monitored and diagnosed. The system state can be verified by

experts, by direct possibly invasive inspection of systems, or it can be determinedfrom historical data about the process. In the latter instance, it is necessary to

V.G. Grishin et al. / Information Sciences 152 (2003) 1–24 3

learn how to distinguish between various system states by monitoring param-

eters. Every state that is to be monitored for diagnostic or analysis purposes is

represented in the learning sample by a set of its variations. Collectively, these

realizations are used to account for the variability of the process data and states.

Clustering or pattern recognition self-learning: If there are new classes of

process states, unknown in advance and indistinguishable by available meth-ods, it is necessary to automatically identify coherent classes for these states.

The learning sample is not divided into classes a priori and the analysis is

expected to indicate those decompositions that follow from the structure of the

data and make possible a meaningful interpretation of the data in terms of the

features of the process data. Once such decomposition has been determined,

the process model can be chosen or updated, the process dynamics can be

described in detail, and monitoring, detection and diagnosis functions can be

improved.

2.2. A data modeling approach for direct level control

The above mentioned discrete models of system state classification by theirobserved parameters are used for automatic control, but for a continuous

Fig. 1. (a) Linear and (b) polar (star) contour representation of 32 parameters vector image of the

current values for a gas-turbine engine.


process control and optimization the control system needs some equations as amodel. The model has to give some analytical dependencies of process ‘‘out-

puts’’, as productivity, quality, etc. from its ‘‘independent inputs’’ which can be

changed for outputs control. The modeling process goes through several stages

involving different techniques, as explained in the sequel.

Multiple regression analysis is used first to get a static model of the system by

its experimentally measured ‘‘input-outputs’’ dependencies. It could be linear

or non-linear, multidimensional, etc. [5,17,20].

Systems identification approach is used to model the dynamic system forcontrol and forecasting, as differential equations representing well the con-

nections between inputs and outputs. It could be non-stationary, non-linear

and multidimensional simultaneously. In both approaches to get an appro-

priate quantitative model we have to select its correct qualitative properties

such as structure, types, and power of non-linearity, etc. [17]. This structural

identification often starts from:

Fig. 2.


• clustering of input/output mappings of data samples calculated with the raw

model;

• a classification of relationships between these clusters for different model

structures;• selection of final model structure by matching the appropriate clusters rela-

tions.

Both clustering and pattern recognition learning prove to be extremely

useful in modeling and this is a main point where data visualization methods

can help for a structure selection of the equation model. Therefore our focus

will be on these techniques here.

3. Geometrical representation of data

Consider Xi a vector of parameters for some system state, and a point in

EðDÞ representing it. We represent values of the parameters from

Xi ¼ fxdðiÞ; d ¼ 1; 2; . . . ;Dg by means of orthogonal axes and get a D-dimen-sional space EðDÞ in which Xi is a vector or point and a class fXigk, corre-sponding to kth system state, covers some region Sk (Fig. 3) which can consistof a few unconnected regions. The goal of pattern recognition learning usuallyis to find a decision, e.g. a description of boundaries between structures Sk inEðDÞ that is effective enough to ensure a correct definition of current systemstates by new vectors or Xi which were not in the learning sample. Usually, a

part of the verified data vectors of learning sample are randomly excluded from

the learning process and are used instead to check its result by an examination

sample. A clustering approach can be useful in finding and describing unknown

Fig. 3. Geometrical interpretation; data structure in data space.


structures Sk in EðDÞ. When connected with a new systems classification

they can become in turn useful in the fault detection and identification process.

An illustration of a possible Sk complexity is shown in Fig. 3.

4. On human–computer interaction during the modeling process

Practically the modeling process (including selection and evaluation of the

most appropriate model) involves human–computer interaction even if the

approach is called ‘‘automatic’’. In these Sections we will review the most

popular approaches.

4.1. Discrimination methods of pattern recognition and clustering

In discrimination methods of pattern recognition and clustering [5] a human

being searches for some combinations of N analytical hyper surfaces (potentialfunctions, functional correspondences, etc.) which together with some discri-

minant or probabilistic decision rule provide an acceptable accuracy in de-

scribing the boundary separating different Sk from the learning sample.

Usually, it is a search of a system of hyperplanes:

fPn ¼ a1ðnÞx1 þ a2ðnÞx2 þ � � � þ aDðnÞxD ¼ 0g; n ¼ 1; 2; . . . ;Nand a decision rule defines which current vector Xi belongs to which Sk when itis in different places of EðDÞ relatively to these hyper planes.Below, we primarily consider this analysis approach as it provides the most

vivid example essential for understanding of this article. The computational

complexity C of searching for these hyper surfaces is directly proportional tothe problem dimension D, the problem non-linearity N (amount of hyper

planes in simplest case) and the number of classes K, i.e.C ¼� D Nk ðK 1Þ;

where Nk is some average amount of hyper planes for one Sk separation. Dcould be up to a few thousands (as in the case of nuclear power plant diag-

nostics), K––up to hundreds and N usually unknown and often very large.Human expert role: To build the effective discriminating function for a

complex system a human expert has to select:

• informative features providing either a way of decreasing dimension D or a‘‘linearization’’ problem, i.e. decreasing of N in a new features space and soon;

• the most appropriate features combinations forming different state patterns

which may be useful for a following decision rule selection and for human

operator monitoring;

• the structure of a decision rule, i.e. a quantity and types of hyper surfaces for

Sk boundary approximation;• initial positions of these hyper surfaces;


• the algorithm and criteria for their position adjustment in the learning pro-

cess in order to get an acceptable recognition accuracy for each learning and

examination sample.

The role of the computer: According to specialists [5], most methods ofcomputer pattern recognition are almost equitably effective, and the overall

performance of a particular approach depends most critically on the expertise of

the problem domain experts. For example, if the expert cannot successfully solve

the above listed problems of feature and model structure selection the computer

will be wrongfully guided to check a huge amount of the local extremes and will

not reach an acceptable solution. Moreover, in this case it could be very difficult

to understand the solution obtained. In any case, and especially if domain

knowledge is not enough to avoid these problems, experts usually analyzegraphical representations of data like that shown in Fig. 1(a) and (b). However,

as shown below, separately from matrix representations as in Fig. 2 they are not

effective for visual analysis of multidimensional data and complicated time se-

ries. Sometimes color matrix representations of stand-alone vectors as well as of

parameter dynamics matrix are used, which could be very profitable for this

problem solution, but unfortunately are used poorly. The reason is that the vast

majority of researchers and system developers are not oriented towards emu-

lating the human pattern recognition process because they do not understand thehuman vision system in depth and mainly do not know how to use its strength

properly. Further many researchers face a wide spread modern prejudice that

computers can do everything. At most they try to minimize the human functions

by a computer substitution. Sometimes this may work but not for the problems

considered here. As a result the abilities of the control system are decreased.

In response to the above mentioned problems we propose a methodology

able to develop pictorial representations for complicated non-linear problems

and non-stationary process data that are designed to take advantage of theunique processing and feature extraction capabilities of the human vision

system. To emphasize the high value of our method we will first discuss the

drawbacks of two existing techniques which aimed to overcome the mentioned

drawbacks by means of a human task simplification:

Artificial neural networks: One of the automated techniques that has gained

wide popularity recently is artificial neural networks (ANNs). Through train-

ing, usually formulated as a mathematical programming problem, ANNs can

learn a non-linear discrimination rule to separate features and Sk in the data.However for large data sets with strong non-linear dependencies in the data,

the networks are very difficult to train and computational time can be pro-

hibitively large. Another drawback of the ANN approach is that it is very

difficult, if not impossible, to gain insight into the clustering mechanism by

examination of the network parameters, for example the weights and thresh-

olds of the network. Therefore, it is particularly difficult for an expert to

interpret the results of the network solution and to make sense of it.


Computer search of reduced data representations: A second direction of a

data analysis enhancement is targeted on a computer search of reduced data

representations, which may simplify a visual analysis. They will be critically

analyzed in the next subsection.

4.2. Projection pursuit and visualization methods

For a choice of qualitative models and decision rules, we need to understandsomething about multidimensional spatial structures fSkg of data samples inEðDÞ and their boundaries. Because human vision can not perceive more thanthree-dimensional space structures many methods of EðDÞ and X projection

visualization have been developed. Some of them involve projecting the initial

data sample onto various planes of its main (statistically optimal) components

(this is commonly referred to as principal component analysis). Methods of this

type are linear and can only be applied to problems with D < 40–50. Theprimary restriction is that a particular data matrix is of full rank and an inversemust be computed.

The ‘‘multidimensional scaling’’ type of methods automatically arrange a set

of L ¼ jfXgj data points on a plane. This can be accomplished by optimizingsome performance function, e.g. a standard linear least squares problem that

captures the important characteristics of a sample. This reduces to the problem

of multiextremal optimization for L variables.Other methods are based on systems of the Peano-Hilbert type with em-

bedded developments (scans), which generate, for example, the density curvesof the sample vectors distributed along regular trajectories. Evidently, it is very

difficult to estimate visually the features of multidimensional, even two- or

three-dimensional structures by such a curve. Other visualization methods that

are also used can be found in [2,6], but all of them use only a small part of the

overall capability of the human vision systems.

All these type of visualization are based on the misunderstanding, that be-

cause human vision cannot see more than 3D space, we have to represent

multidimensional data structures by their 2D or 3D projections. This is notcorrect because human vision can really analyze very high dimensional struc-

tures without seeing their space by means of visual analysis of data relation-

ship. In the following we will plead for this argument.

5. Pictorial analysis

5.1. Direct visual exploration of multidimensional relationships

Multidimensional structures and relationships can be visually analyzedwithout using spatial 2D–3D projections, if to properly map X onto visible


features of two or three-dimensional pictures. Although diagrams, bar graphs

and other graphical representations of data have been used for many years [2],

they are empirical constructions useful in the estimation of a few simple fea-

tures of low dimensional data. In fact, the questions of their use for compli-cated problem solutions were not considered in depth first of all because of

absence of a quantitative model of the human visual perception and secondly

because of the very complicated connections between picture features and the

multidimensional data structures represented by them. In the following we will

discuss essential human vision capabilities which pictorial analysis exploits.

Features hierarchy: Humans are capable of distinguishing and comparing of

many pictures in parallel with hundreds of local attributes and features relating

to the shapes, textures, colors, or brightness of the images (see Fig. 4). A standalone spot on a picture or a spot cluster has certain boundary shapes which can

be segmented by simple local features such as ‘‘straight line’’, ‘‘concave’’,

Fig. 4. Plane picture analysis by human vision.


‘‘convex’’, ‘‘angle’’, ‘‘hole’’, etc. These features have attributes such as ‘‘sizes’’,

‘‘orientation’’, ‘‘symmetry’’, etc. and are connected by means of ‘‘upper-

lower’’, ‘‘left-right’’, ‘‘inside-outside’’ positioning. Local features are visually

joined in more complicated shapes as ‘‘wave’’, ‘‘leaf’’, ‘‘a face profile’’ andothers associated with real world systems and also have above mentioned and

more complicated attributes synthetically expressed easily in a linguistic

manner. This features generalization continues until obtaining a holistic image

of the spot which integrates features such as ‘‘complexity’’, ‘‘symmetry’’,

‘‘elongation’’ besides the usual size, orientation, position on the picture, etc.

Plus, color and textural properties can be described.

These attributes and features are then organized into a multilevel hierarchy

that can be partially verbalized. If a picture contains many different separateshapes, a nested hierarchy can be applied which zooms out from shapes to

clusters to groups to pictures. Additionally, combinatorial and statistical fea-

tures can be visually detected and estimated. The capability of human vision to

rapidly process through this hierarchy, searching for more details or general-

izing the attributes, allows for the simultaneous examination of many facets

of the image in terms of a variety of attributes and features.

Linguistic and formal representations: A human-linguistic description of

these features and patterns consists of some combinations of features Qi andattributes Ai connected via AND/OR operators. For example:

‘‘<middle size ðQaÞ shape> AND <with pretty symmetrical ðQbÞ aroundhorizontal axis ðAcÞ> AND <compact ðAhÞ>. . .. AND <small Qk (concave)with AiðkÞ (for example, a position)> OR Qm AND QjðmÞ AND QkðmÞ (withothers attributes)’’, and so on. Last disjunction ‘‘OR’’ describes at least two

shapes and a few of them some class of shapes. Using ranges of attribute values

we will get different classes of continuous shape variations. These descriptions

are invariant for global rotations, shifts and projective transformations ofwhole shapes as well as their parts (up to some limit). This result is in tune with

the well-proven robustness of fuzzy reasoning systems [15].

The relationships: visual features––data structures: For any representation, a

certain shape sample is a point in the space EðDÞ. A shape pattern including itsvariations covers some region around this point. In general, these regions for

different shapes are not similar, convex or symmetrical because even for simple

contour displays as in Fig. 1(a) and (b) the stability of a complicated feature�svisual perception depends on the relationships among its components andbetween them and the surrounding features. For example, if the values of

neighbor parameters Xk, Xkþ1, Xkþ2 and Xkþ3 in Fig. 1(b) form a ‘‘concave part’’,

this will be visible so for Xk > Xkþ1 < Xkþ2 < Xkþ3, for Xk > Xkþ1 > Xkþ2 < Xkþ3and others of their combinations depending on the current values of each

parameter, on the differential and absolute thresholds of vision, etc. This

subjective factor is quite well encapsulated linguistically as presented in the

previous subsection. For the corresponding color matrix representations of


structures in data space EðDÞ even simple patterns are more complicated anddepend on mapping of Xi or Xi into picture.

The exceptional flexibility and power of visual features hierarchy is the best

argument for using our approach. It allows detecting and describing verycomplicated high dimensional non-linear system models and structures Sk inEðDÞ depending on the appropriate mapping of initial data onto pictures.

5.2. Visual representation classes

There are several possibilities of mapping fXig into pictures fPig. Eachrepresentation has its own advantages and disadvantages and will fit someproblem classes well, but an estimation of their capabilities is a difficult task.

For example, at first glance, it seems promising to map each data parameter

value into some visible feature attribute to distinguish essential Sk by con-ventional pictures. For example [8], statisticians tried to find such mapping of

patients parameters into features of face parts which map ‘‘smile’’, ‘‘quiet’’

faces for healthy humans and ‘‘sad’’, ‘‘angry’’, ‘‘bad’’ ones for different dis-

eases. During a decade, many attempts were made to develop and apply this

technique, but these attempts were not successful. Why? An analysis shows thatstructures corresponding to ‘‘smile’’, ‘‘sad’’ or other expressions of faces in an

Euclidean space of their features are not simple and often consist of a few

unconnected regions with complicated unknown boundaries. Consequently,

this approach requires finding effective ways of mapping unknown complicated

Sk structures of data into other complicated non-linear also unknown structures

of facial expressions. Researchers using this approach found some essentially

non-linear representations of a few classes identifying different facial expres-

sions through certain learning samples but did not check if these representa-tions were validated by the examination samples.

Our approach is based on the analysis of multidimensional relations through

transforming the initial data (heterogeneous arrays, signals and fields) into

artificial pictures fPg. It exploits human capabilities of pattern recognition indoing data analysis and selecting appropriate representations, features and

class descriptions by means of interactive human–computer learning––the very

essence of ‘‘pictorial analysis’’ [9].

5.3. Human–computer interaction in pictorial analysis

The concept: The essence of our approach is to conduct visual analysis prior

to a computer analysis. No data is rejected without either a priori knowledge

about its uselessness for following model choice or the same assurance from in

depth visual analysis. Computer methods based on reduced data representation

(such as projections or principal component analysis, as well as pattern rec-ognition learning or model identification) have to be directed by holistic images


of the system�s states figured out via a full visual analysis of all data. Fortu-nately, modern monitors display simultaneously more than 4096� 4096 in-dependent color dots (pixels) and with color printing these values could be tens

times more on the same display size. Consequently, complete information oftens and hundreds of vectors X with dimension D up to many thousands can besimultaneously represented on the screen by separate pictures via the ‘‘matrix

representation’’.

Criteria for choosing a representation for visual analysis: Color matrix rep-

resentation is the most effective choice to start feature and model selection for

monitoring of highly dimensional, non-stationary systems (such as power

plants). In Fig. 5, visual representations of data are hierarchically classified

according to the power of their usage of different sub channels (sub modalities)of vision as well as the dimension and connectivity of formed images. It is very

important to correctly choose effective mapping of each system’s parameter intopicture features. It could be not only matrix cell color-intensity or star ray

length as on Fig. 1(b) but as well other local and integral features of a picture,

as listed in Fig. 4. A complexity of connection between structures in the state

Fig. 5. Static visual representations of data.


space EðDÞ and visually selected features Q depends on this mapping. This

connection can be fairly complicated even for the simple stars and matrix

representations of static vectors. Therefore, to effectively apply visual analysis

results to pattern recognition, identification and other model selection, we haveto do an analysis for extracting representations providing a well interpretable

connection of EðDÞ with a picture space P ðQÞ.From visual perception psychology it is known that on the average it is best

to equally share data between sub modalities, features and attributes of a

picture and to represent with each of them no more than 5–7 different values of

some data parameter. This restricts the representation choice.

5.4. Human and computer roles in pictorial analysis

The role of the human: We refer to Fig. 6. In general, at each step of the

dialog the human-researcher performs the following tasks:

• specifies a current representation and its parameters (for example, color

scale, an order of Xi coordinates placement in matrix cells or along contour,

etc. . .), as a set of vectors for on screen representation and may be some oftheir previous state transformation of X (for example, statistical or spectral,for signals);

• visually compares displayed pictures P ðiÞ and selects features separatingsubgroups of Pi;

• builds descriptions Qfqg allowing to distinguish classes fPig of the learningsample, corresponding to different system states or equipment malfunctions;

• determines informative Pi clusters, if this is a self-learning, and a search ofnew important system states, dangerous situations and so on;

• selects a representation change to enhance a picture�s visibility and a class�sseparation;

• checks with the computer the qualitative model selected.

The role of the computer: A computer has to process the human requests by:

• data transformations and analyses (if needed);

• data representations and their possible computational enhancement;

• creation and development of selected representations, fqg and fQg databases;

• testing these features and descriptions for the separation of a class fPig inthe feature space EðQÞ;

• automatic testing of selected models, discriminating decision rules and their

quantitative adjustment by means of conventional pattern recognition or

identification algorithms.

Through an iterative approach of these steps, the most effective pictorial

representation, i.e. the one that leads to the most effective set of discriminants,

is selected. Indeed, any reliable a priori knowledge should be used to facilitatethe choice of the above transformations and the corresponding features.


However, to get effective algorithms for real-life control problems, the ob-

tained fqg and fQkg have to be from pictures well interpreted in terms of thefXigk data structures in EðDÞ (recall Section 3). This is not always possible.Because of the subjective nature and fuzziness of human perception and verbal

descriptions of the patterns recognized by classification in picture classes, theirmathematical expression or even algorithmic approximation can be very dif-

ficult. A computer model will be derived from a pictorial analysis in any case

and if it does not provide a complete problem solution, a control system has to

use it together with human-operator monitoring to cover those detected rep-

resentations, situation features and descriptions that could not be modeled. Of

course in this case the control system has to have a database of standard

patterns of important situations and human operators have to be trained for

visual comparison of these patterns with an image of the current situation.

Fig. 6. Pictorial analysis.


5.5. Main issues in pictorial analysis theory

In Fig. 4 possible structures of Q were described. Their search complexityCðQÞ depends on the number of inherent features, their attributes, relationsand logical operations. A subset of the shortest descriptions fQ : CðQÞ ¼ Cming exists for each sample fXg and human can construct descriptions of thecomplexity no larger then some l-max in reasonable time. Therefore, at leastfor comparison with considered conventional methods, the first question of our

theory is: ‘‘What structures of data can be detected and described by visual

perception on different representations in a reasonable amount of time?’’ The

answer requires:

• human vision modeling, i.e. mathematical definition of the human-used al-phabet fqg, their attributes and relations frðqðiÞ; qðjÞ; . . . ; qðzÞÞg;

• search of the sets of probable descriptions of various pattern structures

(properties, characteristics), of an estimation of different Q search time;• analyze which structures in EðDÞ are corresponding to different q, fqg, Q forspecific pictorial representation and dependencies of CðQÞ-min on sample Xproperties.

Unfortunately the decades of work by many scientists have led to the

construction of formal models of only some specific features of perception forvery simplified stimulus. After years of experimental research and modeling of

human vision capabilities of different pictorial representations analysis we have

just a partial solution to estimate learning prospects and results for different

data structures [1]. Fortunately, this only limits the algorithmic implementa-

tions of the above analysis for automatic control but not for a human-operator

process monitoring. If stable and distinguishable patterns of important situa-

tions were found, they could be used for visual system monitoring without

knowledge of corresponding fXig structures in EfDg. Moreover, this knowl-edge is not needed to figure out what real events in a system generate different

features and patterns of its states. These relationships usually have a simpler

interpretation than with EðDÞ. So we also get a system model, which gives thevisual and verbal description of system�s states by its parameters.A few decades of our experience has shown the pictorial analysis always helps

to better understand a controlled process and always gives information about

data structures for enhancement of regression, pattern recognition and identi-

fication models, via this approach and using an appropriate dialogue interface.

5.6. Methodology of pictorial analysis

The essence of our methodology of pictorial analysis is based on sequential

looking through some contour and matrix representations specified by the

problem, data, above considered criteria and principles and intermediateanalysis results.


Recalling Section 4.2, the simplest data, containing a few tens of vectors Xi

(L < 100) with up to 30–40 parameters (D < 40) can be visualized and com-pletely analyzed with only simple contour representations as illustrated in Fig.

1(a) and (b). However, even for this simple case, a linear display (Fig. 1(a)) isworse than a polar one (‘‘star’’) which has two dimensionally organized con-

nected shape more closely resembling natural shapes than the linear ones and

therefore allowing to better estimate and compare integral features as sym-

metries, elongations, inside figure relations, etc. (See Fig. 4). This stands even

more for greater D and L. Therefore in the sequel we will consider just onecontour display––the star.

We have to visually compare stars of a few vectors of one state with stars of

another state and to look for features, state patterns and their descriptions todistinguish these states with needed accuracy. Logical combinations Q of

shapes features [1] can probably describe any structures in EðDÞ, but forpractice it is important to know which structures in E for the given represen-tation can be described by sufficiently short combinations of features and at-

tributes known to be easily found in a reasonable time. For stars, we

experimentally found that Q from 10–15 features can be selected by the spe-cialist for a few hours on average and this is usually one of the shortest problem

solutions [1,3]. Due to the flexibility and diversity of visual features and de-scriptions, a set of Q with this length can cover a majority of real problems.Practically, many multidimensional problems for which we have to start the

data analysis from the color matrices, as in Fig. 2, can be reduced to their small

subspace visualization by contour displays [1].

For ‘‘stars’’ to interactively search the state�s patterns by the means of datarepresentation, the following main tools can do it:

• Mapping of xi in to ray length should map the most important xi value rangesinto the best visible ray changes. It could be specific non-linear mapping foreach xi.

• Coordinates permutations of Xi on the star permits to make shorter Qfor structures with the long initial descriptions, and this is a main goal of

human–computer dialogue, i.e. this is the synthesis of the better ‘‘visual’’

representation.

• Origin shifts: For example, the stars of EðDÞ vectors laying in tubes aroundstraight lines crossing coordinate axes near their origin will have similar

shapes because of close coordinates xi proportions. Therefore, by joiningthe origin with the ends of different Xi, we can detect all sample tubes pass-

ing near Xi and all other tube structures in EðDÞ by moving the origin to an-other Xi. Other easily visible features, which can be detected by means of

their appropriate mapping, are shape transformations such as shifts, rota-

tions, etc.

Matrix representations: Even built from data samples consisting of 20–30

vectors Xi, a matrix representation, as on Fig. 2, can be a fruitful supplement to


stars in order to map all vectors in to one matrix. The matrix gives a pattern of

all learning sample giving some general connections between its vectors, which

could be useful for star parameters choice. Sometimes, this technique will yield

the problem solution. Three interactive tools were applied for a solution visi-bility enhancement: a selection of effective mapping xi values into gray inten-sities of cells, binarization of these intensities for further application of special

algorithms and then permutations of rows and columns for picture quality

criteria maximization by these algorithms. We proved that for most samples

fXg one of these algorithms, based on the analysis of X graph structure, en-sures the improvement of picture connectivity up to L D times by permu-tations of columns and rows of the picture [10].

When D > 50 and/or L > 100 the stars shape can be too complicated to startanalysis from them. In this case, the matrix representations obviously have

Fig. 7.


priority because they allow simultaneous illustration of a large amount of in-

formation on one screen. If we were to represent all realizations fXigk of each kstate of a system by a separate matrix, these state�s portraits would show theirinternal variations and stable patterns, from which we have to interactivelyselect patterns and features for state�s separation.Except for the most important tool of dialogue, (e.g. considered above matrix

elements permutations) it is important to consider the mapping of xn values intogray scale or colors [11], picture improvement by smoothing, edge sharpening,

and many other methods (see Fig. 7). On these pictures, besides the contour

features, many other ones appear (textural, statistical, space spectral, color,

etc.). The dependencies between data structures and picture properties become

considerably more complicated in this case. Simultaneously, the possible deci-sion variety increases many times. Each choice of state�s patterns and theircomparative variations estimations, i.e. for static systems some general vector

Xk for each state, can be more precisely compared by means of stars again.

For dynamic systems, when an image of a state sample is already a matrix of

parameter dynamics, many of these matrices represent each state variation. To

detect state patterns, other informative features and descriptions of the same

methods of a representation adjustment are applied, except column permuta-

tion to avoid time dependencies destruction [2,12,13].For time series, and especially vibrations and acoustic signals for equipment

diagnostics, the spectral analysis is an additional powerful tool for the detection

of useful information in noisy data/environments. The human–computer-

interaction goal is to find the appropriate basic functions, intervals, weights,

etc.

6. Critical areas of application where pictorial analysis is most suitable

6.1. Visual system monitoring and diagnostics

After obtaining all possible information for the control model choice, the

state patterns, features and visual decision rules can be applied for human

operator decision-making. These results of a pictorial analysis usually containthe information, which cannot be exhausted by algorithms and formal models

due to its fuzziness and absence of appropriate models of human perception.

Only the subjective character of this fuzzy information can encapsulate the

complexity of a visual representation required by monitoring and control in

complicated situations. From the answers to a survey done in the Soviet Union

(in 1986–87, after the Chernobyl�s disaster), leading developers and chief en-gineers of nuclear power plants considered as either impossible or inexpedient a

complete automation of the following functions, for which pictorial analysisproves very promising:


• transient modes, preparation and initiation of an automatic mode, start–

stops;

• choice of reserves (mechanisms or channels) and the majority of tasks deal-

ing with system structure changes;• account of non-formal characteristics of a situation (technical, natural and

others) and corresponding corrections of control;

• control of effectiveness in normal plant functioning, in particular at the cost

of its ‘‘individuality’’;

• early diagnostics, forecasting and troubleshooting of different types of faults

in process equipment (e.g. small leaks, vibrations induced by mechanical

failures in rotating equipment, actuator or sensor failures) and control sys-

tem malfunctions; preventing• equipment failures and emergency situations through proper recognition

of abnormal operating conditions and the timely initiation of appropriate

decision and control strategies, etc.

Each of these tasks requires decisions involving a human operator for the

information and control loops of the process. In order to present the state

patterns obtained by pictorial analysis and simultaneously represent a current

situation image generated by the same representation, trained operators can

match real-time patterns with the templates for fault detection, diagnosis orcontrol tasks [3].

6.2. Conventional human operator decision making

Today, the design of information displays (a combination of tables, text,

hard panels, etc.) requires an operator to solve many complex formal-logic typeproblems in order to reach the desired conclusion. For example, consider the

problem of an operator attempting to estimate the operating state of an in-

dustrial process from real-time plant data in order to make a control decision.

Information and decisions might include: Valve A is open, pressure B is in somedefinite range, temperature C is decreasing, etc.; this means that variable Tneeds to be decreased to maintain acceptable operating performance. Fuzzy

control [18] has proven extremely useful in dealing with this kind of problems,

however there are cases when the human operator is needed to make the finaldecision (especially in high risk situations). This is often cumbersome and

difficult for human operators, especially during abnormal operating conditions

of a complex non-stationary process. By transforming plant data via pictorial

analysis and representation, decision making by the operator is enabled in the

image domain.

It is well known from engineering psychology investigations that operators

cannot simultaneously handle more than 7–12 variables for logical decision-

making tasks. For more complicated problem solving, either a hierarchy (tree)of conventional windows and/or an expert system is required. The first en-


hancement to the operator information/decision making system requires that

the operator looks through many levels/resolutions of windows to obtain rel-

evant information. The operator is required to search through various bran-

ches of a tree-structured database. This increases the decision making time andvery detailed operator training is required.

By means of a pictorial representation, an operator can perceive the current

situation as a holistic image and quickly estimate its general state. Typically, an

operator works in a supervisory mode about 40% of the time, i.e. he observes

whether ‘‘all is in order’’ or if it is necessary to interfere with the process. If so,

he/she can compare a current image with standards by means of learned fea-

tures and rules and make a decision. When more complicated situation occurs,

then this approach could be more profitable because the pictorial analysisforms an open semiotics system of situation descriptions [16]. It can generate

patterns and descriptions for states unrepresented in the learning sample.

Pictorial features obtained in the analysis together with interpretation of their

connections with real events in the system create a system of a pictorial de-

tection of new combinations of these features and their generating events. They

can be easily integrated with fuzzy control systems to enable human inter-

vention whenever needed. We are currently working on the development of a

system that integrates pictorial analysis within a fuzzy control software [19].Pictorial methods of data representation, recognition and analysis can be

used as a tool to interact with domain experts and also to incorporate the

history of past operational data into the knowledge base of the expert system.

Fuzzy expert systems primarily use verbalized expert knowledge and heuristic

expert skills and ‘‘know-how.’’ Difficulties often encountered include differ-

ences of opinions from two experts commenting on the same aspect of a be-

havior and the difficulty for a novice in the problem domain, the expert system

builder for example, to ask the correct questions to obtain the necessary in-formation to construct an appropriate knowledge base. Pictorial representa-

tions, state patterns and decision rules give new tools and such an enhancement

to the knowledge base of an expert system may help to increase the eventual

practical uses of such systems in real-time industrial decision-making and

control problems.

7. Pictorial functional diagnostics for gas-turbine engines (GTE)

We have applied pictorial analysis to solve the following problems: electro-

and phono-cardio diagnostics; early prediction of throwing of oxygen steel

converter by acoustic noises: checking of cutting regimes of mining machines;

search of informative features of dolphin signals; clustering of geological

sample; visualization of seismic cuts for geophysicists; pictorial diagnosticsof nuclear power plants (not published due to confidentiality agreement


bindings.) In the following we present the most relevant application which

emphasizes the radical advantages of pictorial representations for non-sta-

tionary process monitoring. Binded by non-disclosure agreements we cannot

give as many details as we would like to.Purpose: Early preventive functional diagnostics and forecasting of GTE

technical state during flight operation.

Basic Method:

• Pictorial representation of GTE thermal, gas and dynamic parameters (op-

erating process parameters) in transient mode of flight; current fuel con-

sumption; oil system parameters; positions of passage section controlled

elements and engine controls.

• Visual analysis and comparison of pictorial representations of these processdata by trained human-operator on the basis of special methodology and

supporting expert system.

• Decision-making, checking and verification by the technician using an ex-

pert system and knowledge base.

Advantages: Pictorial visualization allows a technician to effectively use the

human vision system capabilities to analyze the transient modes of the engine in

repeated flights. As a result, it is possible to observe the effect of incipient failures

through observable symptoms of the malfunctions in the transient modes ofengine operation. Therefore, improvements in the reliability of diagnostic

conclusions were achieved as compared to conventional analysis techniques.

Performance evaluation:

• Average time of obtaining a diagnostic conclusion: 3–5 min.

• Diagnostic period: 1–3 flights.

• Reliability of correct classification using deviations from the norm which are

expressed by recorded parameters (probability of correct detection of symp-

toms) better than 0.99.• Systematic reliability of early detection of malfunction 0.8–0.95 (probability

of detecting a malfunction no less than a flight before the critical event).

• Forecasting interval: 1–5 flights.

• Using the technology of a pictorial dialog allowed for the early diagnosis of

engine malfunctions with almost 100% reliability resulting in timely mainte-

nance of the engine, which significantly increased to engine�s availability.Practical results: The diagnostic system was implemented on one type of by-

pass GTE that had fuel control equipment with an electronic corrector, anadjustable compressor and an adjustable nozzle. Real-time measurements were

recorded for ten parameters of the process. After the development and im-

plementation of the pictorial functional diagnostic system for interflight pro-

cessing of the recorded data in a ground-based unit, the following results were

obtained:

• in the early stages, before 1–20 engine hours-to-failure, 80% of the operating

malfunctions were detected;


• preprocessing of the data and the use of correlations practically eliminated

false detections: the probability of a false detection was decreased by 15

times;

• the average flight time till in-flight failure was increased by 40%;• seven new types of malfunctions were detected and classified during opera-

tion; this establishes the high adaptability of the system to any deviations

from the norm;

70% of the detected malfunctions were correctly forecasted for 3–7 flights.

8. Conclusions

The essence of our approach is to conduct visual analysis prior to a com-

puter analysis. No data is rejected without either a priori knowledge about its

uselessness for following model choice or the same assurance from in depth

visual analysis. Computer methods based on reduced data representation (suchas projections or principal component analysis, as well as pattern recognition

learning or model identification) have to be directed by holistic images of

the system�s states figured out via a full visual analysis of all data. The methodhas proven useful in several critical domains. Current work is focused on the

integration of our method within a fuzzy control software [4,7,19].

Acknowledgements

Our most grateful thoughts go to Dr. A.S. Soula, Chief Developer with

View Trends Ltd., who led the work for the gas turbine engine diagnostics

presented in Section 7.

References

[1] V.G. Grishin, Pictorial Analysis of Experimental Data (in Russian), Nauka Publishing,

Moscow, 1982.

[2] P.C. Wong, R.D. Bergeron, 30 Year of Multidimensional Multivariate Visualization, in:

Scientific Visualization––Overview, Methodologies, Techniques, IEEE Computer Society

Press, 1997.

[3] V.G. Grishin, CAM operator functions and pictorial representation of information, in:

Preprints IFAC/IFIP/ Int. Conference on Analysis, Design & Evaluation of Man–Machine

Systems, vol. 2, 1988, pp. 422–425.

[4] D.J. Guilmore, Interface design: Have we got it wrong?, in: Human–Computer Interaction:

Interact �95, Chapman & Hall, London, 1995.[5] V.N. Vapnik, The Nature of Statistical Learning Theory, Springer, London, 1995.

[6] J. Foley, A. van Dam, S. Feiner, J. Hughes (Eds.), Interactive Computer Graphics: Principles

and Practice, Addison-Wesley, Reading, MA, 1995.

[7] B.M. Velichkovsky, D.M. Rumbaugh (Eds.), Communicating Meaning: The Evolution and

Development of Language, Lawrence Erlbaum Assoc, Mahwah, NJ, 1996.


[8] H. Chernoff, The use of faces to represent points in k-dimensional space graphically, J. Am.Stat. Assoc. 68 (1973) 361–368.

[9] V.G. Grishin, Multivariate Data Visualization for Qualitative Model Choice In Learning

Systems, in: Proceedings of IEEE ISIC/CIRA/ISAS International Symposium on Intelligent

Control/Systems, 1998, pp. 622–627.

[10] L.L. Verin, V.G. Grishin, Algorithm for interactive forming matrix data representation and

estimation, Pattern Recogn. Lett. 4 (1986) 193–200.

[11] V.G. Grishin et al., The Color Coding Models For Data Matrix and Scalar Fields (in Russian),

Institute of Control Sciences Publishing, Moscow, 1991.

[12] V.G. Grishin, Dynamic spectroscopy in problems of visual analysis and identification of

compound acoustical signals, Automat. Rem. Control (2) (1967) 71–77.

[13] SAP AG Collaborates With Human-Intelligence Scientists on the Management Cockpit for R/

3, Press Release of SAP AG, Walldorf, Germany, 30 March 1998.

[14] D. Roverso, Soft computing tools for transient classification, Inform. Sci. 127 (2000).

[15] M. Ulieru, in: J. Baldwin (Ed.), Fuzzy Logic, Fuzzy Logic in Diagnosis: Possibilistic Networks,

invited Chapter, John Wiley and Sons, 1996, ISBN 0471 962813.

[16] A. Meystel, Semiotic Modeling, 1995.

[17] T. S€ooderstr€oom, P. Stoica, System Identification, Prentice Hall, London, 1989.

[18] H. Hellendoorn et al., Fuzzy Control, Springer Verlag, 1994.

[19] TransferTech, 2000 �––Fuzzy Control Manager, Version 5.1, User�s Manual, TransferTechGmbH, Germany www.transfertech.de.

[20] M. Ben-Horim, H. Levy, Statistics, Random House, New York, 1984, p. 870.


http://www.transfertech.de

Pictorial analysis: a multi-resolution data visualization ...

Documents

Transcript of Pictorial analysis: a multi-resolution data visualization ...