Engineering Research Center for Computer Integrated Surgical Systems and Technology1 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

Group15:Yiwei JiangMentors:EhsanAzimi,Dr.PeterKazanzides

SeminarPresentation:DisplayCalibrationforHolographic

OST-HMD

Engineering Research Center for Computer Integrated Surgical Systems and Technology2 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

ReviewedPaper

Qian,Long,etal."Comprehensivetrackerbaseddisplaycalibrationforholographicopticalsee-throughhead-mounteddisplay." arXiv preprintarXiv:1703.05834 (2017).

Engineering Research Center for Computer Integrated Surgical Systems and Technology3 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

OurProject

• ThegoalistointroduceimageguidanceviaaugmentedrealityonHoloLens

• The image guidance is AR overlay ofventricle model from MRI imageandcatheterguideoverlay.

Catheterguideoverlay(s)

Ventricle

Azimi,E.,etal.:Canmixed-realityimprovethetrainingofmedicalprocedures?In:IEEEEngineeringinMedicineandBiologyConference(EMBC),pp.112–116,July2018

HMD-BasedNavigationforVentriculostomy

Engineering Research Center for Computer Integrated Surgical Systems and Technology4 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

Relevancy

• Theaccuracyofoverlayingavirtualobjectmodeltoitstruecounterpartwilldirectlyaffectthecorrectnessoftheguidanceitprovides.

• Oursituationisquitesimilartotheirhead-mountedtrackercase,onethingdifferentisthatforthiscasetheyusedtheembeddedcameraonHoloLensratherthanweuseanexternalZEDcameramountedtoHoloLens.

• Thisblackbox approachenablesnon-expertstousethisapplicationonanyHMDwithoutworryingaboutthetechnicaldetailsofeachindividualsystem.

Engineering Research Center for Computer Integrated Surgical Systems and Technology5 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

Summaryofproblem

Misalignmentbetweenthetrackerspaceand3Dvirtualscene

Engineering Research Center for Computer Integrated Surgical Systems and Technology6 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

Technicalapproach

ThegoalofacalibrationprocessistofindatransformT(·)whichmaps3Dpointsfromtheworldcoordinatestoa3Dvirtualholographicenvironment.

Engineering Research Center for Computer Integrated Surgical Systems and Technology7 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

Technicalapproach

Given: 𝑝" (Pre-Defined)𝐺$%(ARTracking)=>𝑞$=𝐺$%*𝑞'

𝑝" =𝐺"$* 𝑞$

Find: 𝐺"$

CalibrationwithHead-AnchoredTrackingSystem

Engineering Research Center for Computer Integrated Surgical Systems and Technology8 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

Technicalapproach

Given:𝑝" (Pre-Defined)𝐺)"(SLAM-basedspatialmapping),𝐺)*, 𝐺*%(Tracking)𝑞*=𝐺*%*𝑞' 𝐺"* =𝐺)",- ∗ 𝐺)*

𝑝" =𝐺"** 𝑞*

Find:𝐺"*

CalibrationwithWorld-AnchoredTrackingSystem

Engineering Research Center for Computer Integrated Surgical Systems and Technology9 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

Experiments

RecordedthroughPortal,notassameaswhattheuserseeingbehindtheHoloLens

Engineering Research Center for Computer Integrated Surgical Systems and Technology10 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

EvaluationOnlytheuserwearingitcanobservethesuperimposedobjectsthatresultedfromthecalibration

1,Train-and-Test:Collect8additionalsamples,andaretestedagainstthecalibrationcalculatedwiththetrainingdatasetsconsistingofthe20alignments.

2,Double-Cube-Match:Asecondcubemarkerisusedasanauxiliaryreference.Avirtualcubeisdisplayedwithapredeterminedoffsetwithrespecttothefirstmarkercubeatfourdifferentequidistantpositions.Theuserisaskedtoalignthesecondrealmarkercubewiththedisplayedvirtualcube.

Engineering Research Center for Computer Integrated Surgical Systems and Technology11 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

KeyResultsOverall,thecalibrationprocesswasabletosuperimposetheholographiccubeinthecorrectposewithregardtoitsrealtrackedcounterpart.

Perspective (4.04±1.04mm)Affine (3.96±1.06mm)Isometric (5.86±0.81mm)

Perspective (5.88±1.81mm)Affine (5.83±1.78mm)Isometric (8.92±1.60mm)

Train-and-Testwithhead-anchoredtrackerandworld-anchoredtracker

Engineering Research Center for Computer Integrated Surgical Systems and Technology12 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

KeyResults

Thex-yplaneisperpendiculartotheuser’sview,andthezaxisisparallelwiththeuser’slineofsight,indicatingthedepthofalignment.

Thex,y,andzaxisareparalleltotheworldcoordinatesystem{W}.Theuserisabletomovearoundandmakealignmentsfromdifferentviewingperspectives.Therefore,noaxisisassociatedwiththedepthdirectioninthiscase.

Engineering Research Center for Computer Integrated Surgical Systems and Technology13 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

KeyResults

Perspective(5.47±4.26mm)Affine(4.45±3.00mm)Isometric(6.44±4.15mm)

Perspective(0.999,0.001,0.001,0.001)Affine(0.999,0.005,0.002,0.007)Isometric(0.999,0.009,0.002,0.003)

Double-Cube-Matchforthecalibrationwithworld-anchoredtracker

Engineering Research Center for Computer Integrated Surgical Systems and Technology14 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

StrengthandLimitation

Cons:

1. Onlyusedthreeofthefivemarkers,andtheposesofcornersarealmostthesame.

2. TheseexperimentswereperformedbytwoexperiencedHoloLensusers.Theresultswouldbemoreconvincediftheydidthetestsamongnon-experts.

3. Thesymmetryofthecubeisunfriendlytocolorblind,anasymmetricalobjectcanbeusedtoimprovethis.

4. Assumptionoflineartransformation.Maybethetransformationissomewhatrelatedtotheposition.

Pros:

1. Ablackbox approachforsolvingthetransformation, regardlessoftheinternalfeaturesofaspecificHMD.

2. Twomathematicmodelarestudied,makethismethodmoregeneralandwidelycompatible.

3. Designedtwoapproacheswhichcaneliminatethesubjectivitytosomedegree.

4. Accuracyaround4mmisgoodformostapplications.

Engineering Research Center for Computer Integrated Surgical Systems and Technology15 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

FutureWork

1. Integratehead-anchoredandworld-anchoredtrackingsystemsintheholographicOST-HMDusingsensorfusion,therebyovercomingissuessuchasocclusionorlimitedfieldofview.

2. Thiscalibrationmethoddependsonanexternalobjectwithmarkersattachedtoit.Itwouldbebetteriftheprocesscanbemarkerless,itwouldbegreatifapplicationcoulddetectanobjectthatisthere,andthenletusermovethatindifferentlocationsforcalibrate.

3. UseastereocamerainsteadofhumanbehindtheHMD,performthealignmentsautomatically.

Engineering Research Center for Computer Integrated Surgical Systems and Technology16 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

Reference

[1]C.B.Owen,J.Zhou,A.Tang,andF.Xiao.Display-relativecalibrationforopticalsee-throughhead-mounteddisplays.InThirdIEEEandACMInternationalSymposiumonMixedandAugmentedReality(ISMAR),pages70–78.IEEE,2004.

[2]M.Tuceryan,Y.Genc,andN.Navab.Single-pointactivealignmentmethod(SPAAM)foropticalsee-throughHMDcalibrationforaugmentedreality.Presence:TeleoperatorsandVirtualEnvironments,11(3):259–276,2002.

[3]E.Azimi,L.Qian,P.Kazanzides,andN.Navab.Robustopticalseethrough head-mounteddisplaycalibration:Takinganisotropicnatureofuserinteractionerrorsintoaccount.InVirtualReality(VR).IEEE,2017.

[4]S.Reichelt,A.Schwerdtner,G.Futterer,H.Stolle,N.Leister,andR.Haussler.Holographic3-Ddisplays-electro-holographywithinthegraspofcommercialization.INTECHOpenAccessPublisher,2010.

[5]Lee,C.K.,Tay,L.L.,Ng,W.H.,Ng,I.,&Ang,B.T.Optimizationofventricularcatheterplacementviaposteriorapproaches:avirtualrealitysimulationstudy. Surgicalneurology, 70(3),274-277.2008

Engineering Research Center for Computer Integrated Surgical Systems and Technology17 600.456/656 CIS2 Spring 2019Copyright © R. H. Taylor

Question？