Automatic scoring of virtual mastoidectomies using expert examples

Int. JCARS manuscript No.(will be inserted by the editor)

Automatic Scoring of Virtual Mastoidectomies Using Expert Examples

Thomas Kerwin · Gregory Wiet · Don Stredney · Han-Wei Shen

the date of receipt and acceptance should be inserted later

AbstractPurpose Automatic scoring of resident performance on avirtual mastoidectomy simulation system is needed to achieveconsistent and efficient evaluations. By not requiring imme-diate expert intervention, the system provides a completelyobjective assessment of performance as well as a self-drivenuser assessment mechanism.Methods An iconic temporal bone with surgically impor-tant regions defined into a fully partitioned segmented datasetwas created. Comparisons between expert-drilled bones andstudent-drilled bones were computed based on gradationswith both Euclidean and Earth Mover’s Distance. Using thefeatures derived from these comparisons, a decision tree wasconstructed. This decision tree was used to determine scoresof resident surgical performance. The algorithm was appliedon multiple expert comparison bones and the scores aver-aged to provide reliability metric.Results The reliability metrics for the multi-grade scoringsystem are better in some cases than previously reported bi-nary classification metrics. The two scoring methods givenprovide a trade-off between accuracy and speed.Conclusions Comparison of virtually drilled bones withexpert examples on a voxel level provides sufficient infor-mation to score them and provide several specific quality

T. Kerwin · D. StredneyOhio Supercomputer Center, Columbus, Ohio, USAE-mail: [email protected]; [email protected]

G. WietDepartment of Otolaryngology and Biomedical Informatics, Nation-wide Children’s Hospital, Columbus, Ohio, USAThe Ohio State University Medical Center, Columbus, Ohio, USAE-mail: [email protected]

H.-W. Shen · T. KerwinDepartment of Computer Science and Engineering, Ohio State Univer-sity, Columbus, Ohio, USAE-mail: [email protected]; [email protected]

metrics. By merging scores from different expert examples,two related metrics were developed; one is slightly fasterand less accurate, while a second is more accurate but takesmore processing time.

Keywords Automatic evaluation, Objective assessment,Mastoidectomy, Surgical simulation, Temporal bone

1 Introduction

An integral and essential part of surgical training is evalu-ation. This is true whether the training is through a virtualsimulation system, a physical simulation (i.e. a cadavericspecimen), or supervised interaction with real patients. Inone-to-one training methods, an experienced surgeon pro-vides immediate feedback to the resident. However, this typeof training time is intensive, costly, often unavailable, andcan potentially be influenced by the evaluator’s personal bi-ases. With a virtual simulation system, objective evaluationof performance and active feedback can be provided to theuser. We have previously reported on the development oftemporal bone simulator system for learning the anatomyand surgical techniques associated with a basic mastoidec-tomy [2,9]. This system employs volume rendering of tem-poral bone data acquired from CT scans, delivers haptic feed-back during drilling using a 3D joystick, and plays auralfeedback to the user with modulated drilling sounds. Thesystem provides a realistic multi-modal environment for learn-ing surgical technique. A screenshot and photo of the sim-ulator system can be seen in Fig. 1. Our intent is not to re-place one-on-one training, but to create a system that servesas additional training experience without the need for directsupervision by an attending surgeon. Part of the goal of thissystem is to give feedback to users in a way that can sup-port its use in an educational curriculum by providing bothformative and summative evaluations.

2 Thomas Kerwin et al.

(a) Photo of the simulator in-use

(b) Screenshot of the simulator

Fig. 1: The surgical simulator used in the study.

A multi-institutional trial consisting of eight institutionswas designed to test the efficacy of temporal bone surgicaltraining in a simulator versus traditional training in a cadav-eric laboratory [23]. All participants were asked to drill thesame virtual bone (referred to as bone ID number 9413R)before and after receiving training in performing the surgi-cal techniques associated with a mastoidectomy. This taskwas in addition to drilling other bones, real and virtual. Ex-perts performed the same exercise on that same virtual boneto provide a standard for comparison with the trainee drilledbones. We will refer to a data volume consisting of the endproduct of a resident performing a mastoidectomy in thesimulator as a resident bone and a data volume from a ex-pert performing the same task on the simulator as an expertbone. The combination of expert and resident bones formsour dataset. This set of varying surgical performances start-ing from identical initial conditions forms the basis of ouranalysis.

Virtual simulation performance in otology and other sur-gical specialties have generally been evaluated by direct ob-

servation by trained experts in the given domain, and manydifferent scales have been proposed for grading technicalskills performance. Our work thus far has been based onthe Welling scale [3], which is designed to measure perfor-mance in specific tasks in a temporal bone dissection labexecuted on a cadaveric specimen. Other scales have beenpresented as well, including one by Laeeq et al. [11], whichhas similar goals but also attempts to expand applicability tosurgical performance. A composite scale of available met-rics on temporal bone dissection/mastoidectomy has beenpublished with a classification schema for potential appli-cation to computer scoring [22]. Since these types of scalesare meant to be applied by experienced surgeons, they oftencontain terms that are extremely difficult to quantify. Thedefinition of “proper saucerization” in the Welling scale isa good example of this problem. Additionally, use of suchterms potentially introduces error in expert evaluation: sincethere is no clearly defined quantitative definition, evalua-tions are subject to the expert’s own interpretation whichmay vary from person to person and even from time to time.In order for drilled mastoid bones to be scored on these met-rics by an algorithm, we require either a precise definition ofthese types of terms or a data-driven approach. This articledetails a data-driven method to automatically score virtuallydrilled bones based on expert evaluation of these types ofsubjective metrics.

2 Related Work

Much of the existing literature on automatic scoring sys-tems for surgical simulation deals with hand-motion analy-sis. This type of analysis usually incorporates hidden Markovmodels to classify expert from non-expert surgical perfor-mances. Murphy [14], Cotin et al. [5] and Megali et al. [13]have demonstrated that the use of hand motion analysis todescribe and score surgical performances is a useful methodof evaluating simulation performance. However, there arelimits to hand motion analysis. Porte et al. [16] have shownthat expert feedback is better than motion economy feedbackfor one month retention of suturing skills. While both meth-ods are shown to give improvement in post-test validity, theyconclude that long-term skills are better learned by feedbackthat is more salient than through simple hand-motion analy-sis. Therefore, it is reasonable to integrate other types of au-tomated feedback into evaluation systems along with hand-motion-based scores.

The most comprehensive work specifically on algorith-mic analysis of mastoidectomy simulation performances todate is by Sewell et al. [19] The broad goals in their work arethe same as in ours – to develop metrics to score mastoidec-tomy performances. In their work, they developed methodsto distinguish between experts and novices using a simu-lation system. In this work, we use automated analysis of

Automatic Scoring of Virtual Mastoidectomies Using Expert Examples 3

the final product of a virtual mastoidectomy to duplicate thesame results as an expert analysis of the same final product.This type of analysis is important if simulation systems areto be used in the certification process. Repeatability and re-liability is a key goal of assessment by simulation systems,and these properties are critical for integration into certifica-tion exams [20].

Part of Sewell’s analysis included using mutual infor-mation to make a classifier that chooses between expert andnovice performances based on the 1000 most informativevoxels to make that decision in a training set. He also usesthe sum of voxels that had at least 0.8 probability of expertremoval that were not removed by the user and voxels thathad a 0.2 or less probability of expert removal that were re-moved by the user. This type of analysis is similar to whatwe propose with the added and removed functions as de-scribed in Sec. 3.3.

Rosen et al. [18] used hidden Markov model analysis forbinary classification of novice and expert surgeons perform-ing a laproscopy procedure. By defining motion signaturesthrough both direct measurement of the forces on the tooland through video analysis, they were able to achieve a ac-curacy of 0.875. Rosen et al. [17] use Markov models to an-alyze knot tying performance with a more advanced lapro-scopic system (Blue DRAGON) which is able to record 27channels of data during the procedure. Mackel et al. [12]used a similar framework to classify the users of a phys-ical pelvic examination simulator with accuracy of 0.927.Cristancho et al. [6] have developed a technique using videorecording and tool position to score laproscopic skills. Ah-midi et al. [1] used a combination of eye tracking and toolposition to classify between expert and novice surgeons inendoscopic sinus surgery with an accuracy of 0.825.

An important difference between our work and the pre-vious work on hand motion-based analysis of surgical pro-cedure is that we use final product analysis – the end resultof the surgery – rather than procedural analysis to evalu-ate the surgery. In mastoidectomy procedures, review of anovice during training is commonly performed by examin-ing the end product of a training task on a cadaver specimenin a dissection lab. Our work uses existing surgical metricsthat are currently used in training. We describe a voxel-basedalgorithm to evaluate a portion of those metrics using an au-tomatic system.

We use earth mover’s distance (EMD) as a metric whencomparing parts of an expert-drilled bone with a student-drilled bone. The EMD has been used to great effect in otherregions of image and volume analysis, especially with his-togram comparisons. In work by Janoos et al. [8], an approx-imation of the EMD is used to cluster brain activity data asrecorded by fMRI. Sun and Lei [21] outline a method toclassify images acquired using optical coherence tomogra-phy that uses EMD as a processing step on features before

using a classification algorithm. EMD is a flexible metricand has been used in speaker identification [10] as well as inaudio searching [24].

3 Methods

3.1 Data acquisition

Under The Ohio State University Institutional Review Board(IRB) approval, as well as approval by each individual insti-tution’s IRB, residents in otolaryngology with a wide rangeof experience in performing mastoidectomies were asked touse a surgical simulator developed by our team. After per-forming a non-surgical virtual drilling task to give them afeel for the simulation environment, we asked residents toperform a complete mastoidectomy on bone 9413R. Fortysubjects drilled the bone twice (before and after training),therefore 80 different versions of this bone were availablefor analysis. Although our simulator records the drilling mo-tions used during the procedure, we consider only the finalproduct of the drilling for this study.

In order to determine meaningful differences betweenthe users, we first create a fully partitioned volume that re-flects anatomical distinctions between the regions that arerelevant to surgery. The volume 9413R was hand segmentedby experts into 37 segments that have relevance to mastoidsurgery. Most of the voxels containing bone were not as-signed a segment. The result of this segmentation is shownin Fig. 2. However, we require all voxels to be assignedto a region instead of only the voxels that are part of crit-ical structures. Since the drilling performed during the mas-toidectomy is in regions that are in close proximity to the im-portant anatomical structures that have been segmented, anintermediate goal is to completely partition the volume. Wecould divide the volume into blocks along a regular grid, butthese blocks lack a relationship to the anatomy and wouldbe clinically meaningless.

In order to completely divide the bone into meaningfulregions, we use a voxelized Voronoi tessellation of the seg-mented volume. We want to give all voxels an id numberthat corresponds to the nearest segmented structure. In otherwords, for all voxels v in the volume, we find the nearestsegmented voxel p to v. We then assign v to the Voronoicell associated with the segment id of p: v ∈ cell(id(p)). Incontrast with the normal Voronoi tessellation, we have manyvoxels sharing the same value for p.

Distance fields are employed to find the final tessella-tion. For each segment, we calculate a distance field thatcontains the distance to the nearest segment. Then, for eachvoxel, we then select the distance that is lowest out of all thesegment distances. This algorithm assigns segment Voronoicells to all voxels in the volume. Since we are not interestedin voxels that are not bone, the id volume is then masked


(a) Original segments (b) Voronoi cells of segmented bone

Fig. 2: The regions created from the 3D Voronoi tessellation of the segments in the 9413R dataset. This is a Right Lateralview of the regional anatomy.

by the intensity values of the original bone volume. All vox-els that are not associated with bone are assigned an id of0. In this way, all voxels are assigned an id on proximity toanatomical segments. The result of this process is shown inFig. 2.

This segmentation of the volume relies on a previoussegmentation by an expert of the voxels assigned to eachstructure. These expert segmentations are common, but donot give a structure id for every voxel. Our technique doesnot explicitly take into account anatomical boundaries, but itfinds a full tessellation of the bone based on key anatomicalareas or subregions of surgical significance.

3.2 Expert data

Based on a survey performed on members of the AmericanNeurotology Society by Wan et al. [22], we identified cri-teria considered important to a correct mastoidectomy pro-cedure that were not captured directly by existing scoringmechanisms. We asked an expert in otolaryngology to visu-ally rate the results of the virtual mastoidectomy procedureson a scale of one to five (with one being ’poor’ and five being’good’) on five separate criteria:

1. Antrum entered2. Posterior canal wall thinned3. Appropriate depth of cavity4. Complete saucerization5. Overall performance

We selected the above metrics because they can be dif-ficult to quantify and they are more readily analyzed by fi-nal product analysis. Other metrics such as “maintains burr

visibility” or “does not penetrate facial nerve” are impor-tant factors to measure performance as well, and have beenexamined extensively by Sewell et al. [19] Those metricscan be determined from simulation data fairly easily and ofcourse should be incorporated into any system that gradesmastoidectomy procedures completely. However, our workfocuses on a subset of important criteria that should be in-corporated with other metrics for a complete final score.The metrics considered here, especially complete sauceriza-tion, are considered important but constructing an algorith-mic test for them is quite difficult. The inclusion of the over-all performance metric in our study is for reference only, andwe recommend the addition of burr visibility and violationof critical structures to be included in a final product analy-sis, as well as incorporating hand-motion analysis scores.

A surgeon was asked to provide us with final products ofmastoidectomies performed on 9413R. We used four exam-ples that the surgeon considered his best, after familiarizinghimself with the simulator. From these four example vol-umes, we constructed three composite volumes: minimum,maximum and mean. The minimum contains the voxels thatwere removed by all expert examples while the maximumremoves any voxels from the original dataset that were re-moved by any expert example. The mean simply takes theper-voxel mean of all the expert examples.

3.3 Extracting distance features

Our goal is to construct an algorithm that provides a scorefor a resident bone based on the expert examples. An impor-tant step is extracting a set of features out of the millions ofvoxels in the volume. Using a set of features rather than raw


voxels, we can use a machine learning algorithm to com-pute a classification based on the features. Sewell used a setof “most significant voxels” in his classification. In our anal-ysis, we use four distance measures between the previouslycalculated segment regions as our feature set.

The distance measures fall into two categories: euclideanand earth mover’s distance. We find two euclidean metrics:voxels removed and voxels added. Both of these can be cal-culated as a sum of a pair-wise operation between the twovoxel sets. The position of the voxels has no bearing on thesemetrics, besides their occupancy in a region or not. The defi-nitions for these functions are found in Eqs. 1 and 2. S is theresident volume, E is the expert volume and R designates asubset of the volume. occ (Eq. 3) is an binary function thatdetermines if the voxel is in the selected region, behavinglike a mask. These two functions are found for all 37 val-ues of R, corresponding to each of the segmented regions.An efficient algorithm can calculate both of these metrics si-multaneously in O(n) time, where n is the number of voxelsin the volume.

removed(S,E,R) = ∑occ(i,R)max(ei− si,0) (1)

added(S,E,R) = ∑occ(i,R)max(si− ei,0) (2)

occ(i,R) =

{1 if i ∈ R

0 otherwise(3)

The removed function (Eq. 1) describes the number ofvoxels that have been removed in the resident’s drilled vol-ume but not in the expert’s drilled volume (i.e., excess drillingby the resident). Conversely, the added function (Eq. 2) de-scribes the number of voxels that have been removed in theexpert’s drilled volume but not in the resident’s drilled vol-ume (i.e., not enough drilling by the resident).

The second category of distance measures consists of theearth mover’s distance (EMD) and a signed version of theEMD. We use a fast implementation of EMD described byPele and Werman [15], for which there is source code pub-licly available. The EMD was originally designed to com-pare probability distributions, but in this case we apply itdirectly to the amount of bone in the volume. This measurecan be thought of as the total work that needs to be doneto change one volume into another, by moving the voxels.In our case, it is better to think of the work in moving thedrilling operation from one place to another in the bone. Thework for moving the drilling operation from one voxel toanother to match the expert volume is based on the distancebetween them, called the ground distance in the EMD algo-rithm. In our case, the ground distance used is the Euclideandistance between the two voxels.

Although the earth mover’s distance is normally usedfor histogram comparison, it has some properties, both in-tuitively and mathematically, that make it a good candidatefor volume comparison features. The EMD between two dis-tributions increases as the work needed to change one distri-bution to the other increases as well. The work in our caseis the amount of drilling, since this is the only operation thatthe users can perform on the bone. There are two types ofdrilling work when comparing a resident’s performance tothe expert: drilling that should have been done, but was not,and drilling that should not have been done, but was.

The EMD algorithm finds the minimum cost to transportmaterial in voxels in the final resident volume from placesthat should have been drilled to places that were drilled andshould not have been. Any remaining voxels that have a dis-crepancy between the resident and the expert bone are addedas a penalty to the final cost. The idea behind using the EMDis that the choice of drilling or not drilling performed by theresident is a locally bounded decision: if extra drilling oc-curs in a spot that is close to a place where experts drilled,then the penalty should be lower than when extra drilling oc-curs far away from expert drilling. Although real bone can-not be moved from one place to another in a cost based man-ner that the EMD is based on, this cost is an abstraction forthe magnitude of the error when drilling in an incorrect spot.Like its use in histogram comparison, the EMD captures aquality of similarity that direct pixel-wise vector euclideandistance does not. This idea is valid for metrics that dealwith shape of the drilled cavity.

Computation of the EMD is expensive. The thresholdedversion of the EMD that we use [15] has a computationalcomplexity of O(N2UlogN) where U depends on the thresh-old value used. Pele and Werman’s experiments were on2D images containing 1538, three channel pixels, and theirsearch on 773 images took around 6 seconds. Our datasethas many fewer 3D images, but each image has many vox-els. Some of the partitions have over 50,000 voxels, even af-ter removing identical voxels in the two volumes to be com-pared. Due to the complexity, calculating the EMD com-pletely on these partitions is not practical. We do not wantusers of our automated assessment tool to wait days to findout they did something wrong in the simulation.

To improve the performance of the algorithm, we subdi-vide our segment-based partitions of the volume into clus-ters of around 5000 voxels. The clusters are determined byk-means clustering, giving k a value of bN/5000c+1. Eachcluster comparison takes around 15 seconds to compute.

We also compute a signed EMD (sEMD) value. If thetotal mass of the expert bone is less than the total mass ofthe resident bone, sEMD = EMD, if not, sEMD =−EMD.This is obviously not a metric, but this quantity does reflectthe asymmetry between expert and resident. In this model,we are not computing distances between different resident


mean.Posterior.Canal.Wall..added.p < 0.001

1

≤ 450757 > 450757

mean.Facial.Canal..Nerve...added.p = 0.034

2

≤ 296462 > 296462

Node 3 (n = 35)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

Node 4 (n = 14)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

Node 5 (n = 31)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

Fig. 3: The resultant decision tree for Posterior canal wallthinned based on the composite feature vectors.

bones or different expert bones, just between experts andresidents. The sEMD measure captures the distinction be-tween too much removal and too little removal.

3.4 Determining appropriate classifications

Using the previously described distance functions, we gen-erate a feature vector. The vector has values from all fourdistance measures on all 37 segments. However, we elimi-nate any measure in the vectors that has no variance acrossall 80 samples. Doing this results in a feature vector thathas a length around 50 for each sample for our data. Withdifferent input volumes, there might be a greater or fewernumber of elements with zero variance, leading to a differ-ent length of the feature vector. From the feature vector andthe expert scores for each bone, we can use machine learn-ing techniques to determine a decision method that convertsan arbitrary feature vector (as from a new resident perfor-mance) to a score for each of the five scored measures.

Initial attempts at classification using simple linear re-gression were not promising. A decision tree approach de-livered much better results. Unbiased recursive partitioning[7] was used to construct the trees. This algorithm only splitsgroups of data elements into different nodes if the split hasa p-value less than a minimum threshold. For our purposeswe considered a value of p < 0.05 to be sufficient, althoughmany of the splits had a value of p < 0.01.

The computed decision tree is applied like a filter to theuser bone drilling performances, based on the values of thefeature vector. One of the trees is shown in Figure 3. The top

of the diagram shows the decision questions. The first ques-tion asked is “Is the value of the feature MEAN POSTERIOR

CANAL WALL ADDED greater than 45075”. If so, then thebone drilling result is filtered to Node 5. This is a terminalnode, so a score is assigned; in this case the score is 2. Thescore assigned for this and each terminal node is determinedby the score that has the plurality out of all the expert scoresthat were assigned to that bin. A histogram of the expertscores assigned to that bin can be seen on the bottom of thefigure. There are some expert scores that are 1 and some thatare 3, but most are 2, so this is the assigned score for this bin.If, however, the answer to the first question is no, then thetree algorithm goes to Node 2 and another question is asked,continuing down the tree. In this way, all feature vectors areassigned scores. Example decision trees computed from thecomposite final scoring method are shown in Figure 5.

Most of the trees had only enough information to clas-sify the bones into three separate categories. This is due tothe lack of examples for some of the bins. For example, onlyone bone was given a score of 1 by the human reviewer forantrum entered and only four bones were given a score of 4for overall score. It is likely that with more examples thesecategories would be better represented, and a decision treecould be computed that outputs the full range of values.

3.5 Evaluation

Two approaches were used to calculate final scores. In thefirst approach, feature vectors were constructed using thefour distance measures between each resident bone and theminimum, maximum, and mean expert bone, as described inSec. 3.1. We call this the composite method, since compositevolumes were made from the expert examples. These featurevectors were used to optimize a decision tree. In the secondapproach, feature vectors were constructed for comparisonbetween each resident bone and each expert bone. Decisiontrees were then constructed for each expert comparison, anda form of voting was used to determine the final score. Weinvestigated two methods for the voting for the ordinal scor-ing. One takes the mean of all expert sub-scores for the fi-nal score (the mean method), while the other one uses themedian of all expert sub-scores (the median method). Forlarger numbers of expert sub-scores, other types of votingalgorithms other than these two may be appropriate. In thecase of the binary classification, we use only a majority test,rounding up for ties.

We use two statistical approaches to evaluate the qual-ity of our ordinal automated assessment scores: correlationand inter-rater reliability. The correlation method used isSpearman’s rank correlation coefficient, which is a measureof monotonic association between two variables. For inter-rater reliability, we use Cohen’s kappa, which is the mostcommon method of determining reliability ratings between


two sets of graders. For the binary classification task, wecomputed the accuracy measure, which is the percentage ofcorrectly classified items. Table 1 shows the quality assess-ment scores. These scores were generated using leave-one-out cross-validation.

We use a sunflower plot [4] to demonstrate the correla-tion between the subjective scores determined by the expertand the computed scores in Figs. 4 and 6. In this type of plot,more petals represent more items assigned to a location onthe plot.

4 Results

We evaluated several ways to score the virtual surgical per-formances using our framework, various scoring scales andcalculation methods. We employed two scoring scales, a four-rank ordinal score and a binary classification. The results ofthese methods can be seen in Table 1. We collected data ona five-rank ordinal scale but for all metrics except for over-all score, we had less than 0.07 of the data values fall intothe 1 category. The data for categories 1 and 2 were mergedfor these metrics. For overall score, we had the same prob-lem with categories 4 and 5 and these were merged. We alsoperformed binary classification, by merging the categoriesfurther: the two higher categories merged into one as well asthe two lower categories. This binary classification task iscommon in the literature on automatic evaluation of surgi-cal simulator performance, while a ordinal classification isless common.

A statistical comparison between the original expert sub-jective scores and the computed scores shows validity forour approach. The range of inter-rater reliability found forthe Welling scale [3], is 0.49–0.64. The complete sauceriza-tion and antrum entered metrics achieve scores in this range.The overall score metric falls in this range with the me-dian voting method. The other metrics are under this range,with depth of cavity in the composite scoring method fallingwell below. Correlation scores, determined by Spearman’smethod, are moderate, with complete saucerization as wellas antrum entered again being quite strongly correlated, butdepth of cavity being comparatively weak.

The median and mean methods gave much better resultsthan the composite method does. These methods are slowerto compute than the composite method. Using the medianmethod (or the mean method), one decision tree must be fol-lowed for each expert bone. Using the composite method,only one decision tree is used. The preprocessing time forthe voting methods is longer than it is for the compositemethod when the number of expert bones is four or more,as it is in our tests.

The results for the binary classification task are shownin the right two columns in Table 1. For this task, the expert

1 2 3 4 5

12

34

5

Complete saucerization

Computed score

Sub

ject

ive

scor

e

●

●●

● ●

●

●

●

●

●

●

●

Fig. 4: 2D histogram petal plot for complete saucerizationscores using the composite method. The number of petalsequal the number of items in that particular bin. A singledot represents one item. Most of the items fall along thediagonal, which means that the item’s computed score andexpert-given subjective score are equal. More petal plots ofthe composite method can be found in Fig. 6.

scores were divided into two groups, expert and non-expert.Values of four and above were counted as an expert perfor-mance; three and lower were considered non-expert. The de-cision tree method was then applied. The fraction of correctanswers are reported as accuracy (ACC). Similarly to theordinal results, the majority voting method achieved higheraccuracy than did the composite method, with the majoritymethod reaching 81%–89% accuracy for the individual met-rics and 75% accuracy for overall score.

In Fig. 4, a two-dimensional histogram comparing thecomputed score and the expert-given subjective score for allthe trainee-drilled bones for the metric of complete saucer-ization using the composite scoring method. The items areconcentrated along the diagonal, which indicates a high co-incidence of the subjective scores with the automated scores.Fig. 6 shows the remaining plots for the composite scoringmethod; the plots for the mean and median scoring methodare not dramatically different. Not all the metric categoriesare represented by the automated scores, since there was notenough data gathered for those bins to make significant de-cisions, as explained in Sec. 3.4.


Ordinal Classification Binary ClassificationMean Median Composite Majority Composite

κ ρ κ ρ κ ρ ACC ACC

Complete saucerization 0.51 0.85 0.54 0.85 0.61 0.79 0.83 0.80Antrum entered 0.56 0.84 0.46 0.75 0.32 0.69 0.89 0.76Depth of cavity 0.36 0.70 0.37 0.68 0.13 0.60 0.85 0.63

Posterior canal wall thinned 0.31 0.75 0.38 0.76 0.31 0.71 0.81 0.80Overall score 0.47 0.80 0.50 0.66 0.32 0.46 0.75 0.45

Table 1: For the ordinal classification task, the the inter-rater reliability (Cohen’s kappa) and correlation (Spearman’s rho) aregiven for each metric. The mean and median columns show the scores from computing a final score from the four separateexpert scores. The composite column shows the results from the evaluator trained using the three composite datasets. For thebinary classification task, overall accuracy (ACC) is given for the majority voting and composite scoring methods. Pleasesee Sec. 4 for more detail.

5 Discussion

5.1 General comments

Many of the criteria seem intuitively likely due to the anatom-ical basis of the metric. The posterior canal wall metric de-cision tree example shown in Fig. 3 is automatically cal-culated based on the added function (Eq. 2) applied to theposterior canal wall region as well as the facial canal nerveregion, which is adjacent. Likewise, the antrum entered au-tomated metric depends only on the voxels of the mastoidantrum region. Some of the metrics depend on computed re-gional EMD values, while other metrics use only the simplerEuclidean distances for classification.

However, because of the incomplete diversity of the datathat we have obtained through the study, the decision treeswe generate can have counter-intuitive results. In Fig. 5, forthe overall score metric, the tree gives a higher score tobones that have a higher value for MEAN FACIAL CANAL

NERVE ADDED. Intuitively, more added voxels should resultin a lower score, not a higher score. The result here is dueto factors in the data that are correlated to performance butare not causal. Many of the bones that our expert scored asa 2 overall are from students that did not drill away enoughbone. Many of the bones scored as a 1 were over-drilled.Because of this, the algorithm constructed a decision tree toclassify these categories using this difference. With a largerand more diverse set of examples, issues like this will bereduced.

Most of the decision trees had only enough informationto classify the bones into three separate categories. This isdue to the lack of examples for some of the bins. For ex-ample, only one bone was given a score of 1 by the humanreviewer for antrum entered and only four bones were givena score of 4 for overall score. It is likely that with more ex-amples these categories would be better represented, and adecision tree could be computed that outputs the full rangeof values.

Problems with the depth of cavity metrics could be dueto the lack of stereo vision in the test environment used togather these resident bones. Due to hardware limitations atthe time, we were not able to deploy our systems with a 3Dstereo display device. Some users complained about diffi-culties in determining depth during drilling. Even though theanalysis described in this article is theoretically independentof the quality of the simulator, it is influenced by the train-ing set, and for this metric the training set might not havebeen sufficient. With the use of 3D stereo in the next revi-sion of our simulation system, we will see if correlation willimprove for this metric.

With a tool to give automated assessment on difficult todefine metrics such as complete saucerization, we can de-velop simulation environments that give feedback to resi-dents during the early stages of their training. Although ex-pert assessment is still needed during the course of study,these types of tools, along with hand-motion analysis, couldaccelerate training for formative development. More stud-ies are required to determine the correlation of the use ofautomated assessment tools as a part of simulation use ina surgical curriculum to actual performance and outcomeson patients. This article presents evidence that an algorithmcan be used to assess shape-based results from bone drillingprocedures, but refinements will no doubt be needed as newevidence and more data comes in from larger-scale testing.

5.2 Limitations

A limitation to this technique is that it must be done foreach dataset. We have constructed decision trees for bone9143R. We can apply the same procedure to other bones.However, we will require both representative expert finalproducts from a mastoidectomy procedure and expert grad-ing of sample resident bones. This takes about four hourstotal: two hours for drilling four mastoid bones at around 30minutes per bone and around two hours to grade 80 bones,since it takes between one and two minutes per bone (on


min.Mastoid.Antrum..added.p < 0.001

1

≤ 818743 > 818743

Node 2 (n = 36)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

max.Mastoid.Antrum..added.p = 0.003

3

≤ 1154986 > 1154986

Node 4 (n = 26)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

Node 5 (n = 18)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

(a) Antrum entered

max.Sigmoid.Sulcus..Sinus...added.p < 0.001

1

≤ 1658114 > 1658114

mean.Mastoid.Antrum..sEMD.p = 0.005

2

≤ −19595.3> −19595.3

Node 3 (n = 31)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

Node 4 (n = 16)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

max.Suprameatal.Triangle..added.p = 0.012

5

≤ 50659 > 50659

Node 6 (n = 21)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

Node 7 (n = 12)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

(b) Complete saucerization

min.Mastoid.Antrum..EMD.p < 0.001

1

≤ 40053.8 > 40053.8

Node 2 (n = 34)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

mean.Mastoid.Antrum..EMD.p = 0.05

3

≤ 89500.6 > 89500.6

Node 4 (n = 29)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

Node 5 (n = 17)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

(c) Appropriate depth of cavity

min.Tympanic.Ring..EMD.p = 0.003

1

≤ 2143.4 > 2143.4

Node 2 (n = 51)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

mean.Facial.Canal..Nerve...added.p = 0.004

3

≤ 230927 > 230927

Node 4 (n = 11)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

Node 5 (n = 18)

1 2 3 4 5

0

0.2

0.4

0.6

0.8

1

(d) Overall score

Fig. 5: The decision trees determined from the data to give the best division between classes.

average) to grade, based on the time taken by our expert.The more time-consuming part is the acquisition of the 80resident-drilled bones. These do not have to be drilled all bydifferent residents, as it was in our case, but a wide enoughvariance in performance is important to avoid over-fitting ofthe data. Once this work is performed, automatic grading ofeach bone can be done without any more user interaction.

We performed an analysis using the same feature vec-tors and decision tree approach as in Sec. 3 but targeting vi-olation based metrics, but this was unsuccessful. An expertgraded the bone on violations of the tegmen, sigmoid sinus,facial nerve, and lateral canal. Most of the resultant decisiontrees had only one node, which means that there was notenough information in the feature vectors to justify splittingthe dataset at a low enough p-value. The metrics that we


1 2 3 4 5

12

34

5

Antrum Entered

Computed score

Sub

ject

ive

scor

e

●

●

●

● ●

●

●

●

●

● ●

1 2 3 4 5

12

34

5

Post. canal wall thinned

Computed score

Sub

ject

ive

scor

e

●

●

●

●●

●

●

●

●

●

●

●

●

1 2 3 4 5

12

34

5

Appropriate depth of cavity

Computed score

Sub

ject

ive

scor

e

●

●

●

●

●

●

●

●

●

1 2 3 4 5

12

34

5

Overall score

Computed score

Sub

ject

ive

scor

e

●

● ●

●

●

●

●

●

●

Fig. 6: The remainder of petal plots of the scores from the composite score method. A continuation of Fig. 4.

used originally are shape-based metrics, and these can beanalyzed well by the distance measures that we have chosen,on regional partitions. However, violation based metrics aremore accurately based on exact voxel analysis based on thestrict segmentation boundaries provided by the experts.

The k-means clustering necessary for efficient imple-mentation of the EMD algorithm, so that it completes in areasonable time, does add an artificial separation betweenregions that is undesirable. Although a k-means-based par-titioning approach appears to be more desirable than a rect-angular tessellation of the structure, the effects of this par-titioning on the classification outcome have not been tested.

In addition, more anatomical structures can be defined. Theinclusion or exclusion of structures from the complete parti-tioning of the bone will affect the grading performance, butit is not clear if the inclusion of more structures always re-sults in better performance.

6 Conclusion

To our knowledge, the use of EMD for the comparison atthe voxel level of virtual surgical volumes has not been at-tempted before and deserves further study. Furthermore, our


techniques provide automatic scores on shape-based perfor-mance metrics that can be difficult to quantify in other waysrather than economy of motion metrics that are not proce-dure specific. The use of a simulation system to obtain mul-tiple expert and trainee performances from a single originaldataset removes any noise in the system due to inexact reg-istration between multiple anatomical specimens. This givesmore confidence and objectivity in the resulting scores. How-ever, further efforts include the creation of a complete objec-tive scoring system for mastoidectomy simulation. We wishto include these metrics in an assessment module for oursimulation system, and plan to do this in the next large-scaletest of the system.

We have demonstrated a method of performing auto-matic scoring for a mastoidectomy simulator. Using deci-sion trees and feature vectors generated from several dis-tance measures, ratings on a multi-level scale can be givento users of a simulation system without action from an ex-pert.

Acknowledgements This work is supported by a grant from the Na-tional Institute of Deafness and Other Communication Disorders, ofthe National Institutes of Health, 1 R01 DC06458-01A1.

References

1. Ahmidi, N., Hager, G.D., Ishii, L., Fichtinger, G., Gallia, G.L.,Ishii, M.: Surgical task and skill classification from eye track-ing and tool motion in minimally invasive surgery. In: T. Jiang,N. Navab, J.P. Pluim, M.A. Viergever (eds.) MICCAI, pp. 295–302 (2010)

2. Bryan, J., Stredney, D., Wiet, G., Sessanna, D.: Virtual temporalbone dissection: a case study. In: IEEE Visualization, pp. 497–500(2001)

3. Butler, N.N., Wiet, G.J.: Reliability of the Welling scale (WS1) forrating temporal bone dissection performance. The Laryngoscope117(10), 1803–8 (2007). DOI 10.1097/MLG.0b013e31811edd7a

4. Cleveland, W.S., McGill, R.: The Many Faces of a Scatterplot.Journal of the American Statistical Association 79(388), 807 – 822(1984)

5. Cotin, S., Stylopoulos, N., Ottensmeyer, M.P., Neumann, P.F., Rat-tner, D., Dawson, S.: Metrics for Laparoscopic Skills Trainers:The Weakest Link! In: MICCAI, pp. 35–43 (2002)

6. Cristancho, S.M., Hodgson, A.J., Panton, O.N.M., Meneghetti,A., Warnock, G., Qayumi, K.: Intraoperative monitoring of la-paroscopic skill development based on quantitative measures.Surgical endoscopy 23(10), 2181–90 (2009). DOI 10.1007/s00464-008-0246-9

7. Hothorn, T., Hornik, K., Zeileis, A.: Unbiased Recursive Parti-tioning: A Conditional Inference Framework. Journal of Com-putational and Graphical Statistics 15(3), 651–674 (2006). DOI10.1198/106186006X133933

8. Janoos, F., Machiraju, R., Sammet, S., Knopp, M., Morocz, I.: Un-supervised Learning of Brain States from fMRI Data. In: T. Jiang,N. Navab, J. Pluim, M. Viergever (eds.) MICCAI, Lecture Notesin Computer Science, vol. 6362, pp. 201–208. Springer Berlin /Heidelberg (2010)

9. Kerwin, T., Shen, H.W., Stredney, D.: Enhancing realism of wetsurfaces in temporal bone surgical simulation. IEEE Transactions

on Visualization and Computer Graphics 15(5), 747–758 (2009).DOI 10.1109/TVCG.2009.31

10. Kuroiwa, S., Umeda, Y., Tsuge, S., Ren, F.: NonparametricSpeaker Recognition Method Using Earth Mover’s Distance. IE-ICE - Transactions on Information and Systems pp. 1074–1081(2006)

11. Laeeq, K., Bhatti, N.I., Carey, J.P., Della Santina, C.C., Limb, C.J.,Niparko, J.K., Minor, L.B., Francis, H.W.: Pilot testing of an as-sessment tool for competency in mastoidectomy. The Laryngo-scope 119(12), 2402–10 (2009). DOI 10.1002/lary.20678

12. Mackel, T., Rosen, J., Pugh, C.: Data mining of the E-pelvis simu-lator database: a quest for a generalized algorithm for objectivelyassessing medical skill. In: MMVR, vol. 119, pp. 355–60 (2006)

13. Megali, G., Sinigaglia, S., Tonet, O., Dario, P.: Modelling andevaluation of surgical performance using hidden Markov mod-els. IEEE Transactions on Biomedical Engineering 53(10), 1911–9 (2006). DOI 10.1109/TBME.2006.881784

14. Murphy, T.E.: Towards Objective Surgical Skill Evaluation withHidden Markov Model-based Motion Recognition. Master’s the-sis, Johns Hopkins University (2004)

15. Pele, O., Werman, M.: Fast and Robust Earth Mover’s Distances.In: International Conference on Computer Vision. Kyoto, Japan(2009)

16. Porte, M.C., Xeroulis, G., Reznick, R.K., Dubrowski, A.: Ver-bal feedback from an expert is more effective than self-accessedfeedback about motion efficiency in learning new surgical skills.American journal of surgery 193(1), 105–10 (2007). DOI 10.1016/j.amjsurg.2006.03.016

17. Rosen, J., Brown, J.D., Chang, L., Sinanan, M.N., Hannaford, B.:Generalized approach for modeling minimally invasive surgery asa stochastic process using a discrete Markov model. IEEE Trans-actions on Biomedical Engineering 53(3), 399–413 (2006). DOI10.1109/TBME.2005.869771

18. Rosen, J., Hannaford, B., Richards, C.G., Sinanan, M.N.: Markovmodeling of minimally invasive surgery based on tool/tissue inter-action and force/torque signatures for evaluating surgical skills.IEEE Transactions on Biomedical Engineering 48(5), 579–91(2001). DOI 10.1109/10.918597

19. Sewell, C., Morris, D., Blevins, N.H., Dutta, S., Agrawal, S.,Barbagli, F., Salisbury, K.: Providing metrics and performancefeedback in a surgical simulator. Computer Aided Surgery 13(2),63–81 (2008). DOI 10.1080/10929080801957712

20. Shaffer, D.W., Gordon, J., Bennett, N.: Learning, Testing, andthe Evaluation of Learning Environments in Medicine: GlobalPerformance Assessment in Medical Education. InteractiveLearning Environments 12(3), 167–178 (2004). DOI 10.1080/10494820512331383409

21. Sun, Y., Lei, M.: Method for optical coherence tomography im-age classification using local features and earth mover’s distance.Journal of Biomedical Optics 14(5), 054,037–6 (2009)

22. Wan, D., Wiet, G.J., Welling, D.B., Kerwin, T., Stredney, D.:Creating a cross-institutional grading scale for temporal bonedissection. The Laryngoscope 120(7), 1422–7 (2010). DOI10.1002/lary.20957

23. Wiet, G.J.: Virtual Temporal Bone Dissection System: Develop-ment and Testing. Triological Society Thesis (Submitted) (2010)

24. Yuxin, P., Cuihua, F., Xiaoou, C.: Using Earth Movers Distancefor Audio Clip Retrieval. In: Y. Zhuang, S.Q. Yang, Y. Rui,Q. He (eds.) Advances in Multimedia Information Processing,Lecture Notes in Computer Science, vol. 4261, pp. 405–413.Springer Berlin Heidelberg, Berlin, Heidelberg (2006). DOI10.1007/11922162

Automatic scoring of virtual mastoidectomies using expert examples

Documents

Transcript of Automatic scoring of virtual mastoidectomies using expert examples