Collaborative 3D Visualization onLarge Screen Displays

Daniel MendesINESC-ID / IST-UTLdanielmendes@ist.utl.pt

Pedro CamposUMA / INESC-IDpcampos@uma.pt

Maurıcio SousaINESC-ID / IST-UTLantonio.sousa@ist.utl.pt

Luciano SoaresTecgraf / PUC-Riolpsoares@tecgraf.puc-rio.br

Bruno AraujoINESC-ID / IST-UTLbrar@vimmi.inesc-id.pt

Alberto RaposoTecgraf / PUC-Rioabraposo@tecgraf.puc-rio.br

Alfredo FerreiraINESC-ID / IST-UTLalfredo.ferreira@ist.utl.pt

Joaquim JorgeINESC-ID / IST-UTLjaj@inesc-id.pt

Hildegardo NoronhaUniversity of Madeirahildnoronha@gmail.com

Figure 1: Our collaborative 3D virtual environment mixinglarge scale displays with mobile devices.

Copyright is held by the author/owner(s).CHI 2013 Extended Abstracts, April 27 - May 2, 2013, Paris,France. ACM 978-1-4503-1952-2/13/04.

AbstractLarge Scale Displays, besides their visualizationcapabilities, can provide a great sense of immersion to ageographically distributed group of people engaging incollaborative work. This paper presents a system that usesremotely located wall sized displays, to offer immersive,interactive collaborative visualization and review of 3DCAD models for engineering applications.

Author KeywordsVirtual Environments; Large Scale Displays; MobileDevices; 3D User Interfaces; Collaboration

ACM Classification KeywordsH.5.2 [User Interfaces]: Graphical user interfaces (GUI).Input devices and strategies (e.g., mouse, touch-screen),Interaction styles (e.g. commands, menus, forms, directmanipulation)

IntroductionLarge Scale Displays offer a unique visualizationenvironment favorable to both individual and collaborativedesign tasks. During the last decade they became bothaffordable and easier to setup, providing highly immersivevirtual environments with higher resolution and supportfor stereoscopic images. Combined with emerging inputdevices, these provide natural ways to interact with virtual

content as well as enabling new applications to supportcollaborative engineering tasks. Following seminal work in

Figure 2: Powerwall with4.0x2.3 m, using fourstereoscopic projectors in Lisbon,Portugal.

Figure 3: Cave-like system inRio de Janeiro, Brazil. Eachscreen has 4.1x2.3 m.

Figure 4: iPad user navigatingand sharing his camerapoint-of-view with the Wall.

multimodal interfaces [2], new devices have becomeavailable to enhance interaction including multi-touchtablets combining several sensors, non-intrusive trackingsolutions based on depth cameras and affordable braincomputer interfaces. We present a virtual reality (VR)system designed to support collaborative visualization of3D environments, applied to engineering (CAD) designreview activities. Our approach combines stereoscopicLarge Screen Displays with multiple input devices toprovide natural interaction techniques for 3D visualization,while fostering collaboration and remote communicationbetween users across different VR facilities simultaneously.

Related WorkThere is considerable research on combining severalmodalities to interact with Large Screen Displays. Ni etal. [13] surveyed several interaction techniques usingmid-air spacial gestures, handheld controllers or touchsurfaces to navigate in 3D space. As shown by Jota etal. [8], gestures provide a hands–free solution and can beused precisely to acquire targets on the screen, whileallowing mobility around the space in front of the display.Techniques developed for horizontal surfaces can also beapplied to mobile touch devices. Boring et al. [3] proposesmanipulating content displayed on a vertical display, usinga mobile touch device. Nancel, studied [12] uni- andbi-manual interaction techniques to conclude thatnavigational activities, such as pan and zoom, using touchdevices are both more efficient and cause less fatigue thanthose in mid-air.

Different navigation methods have been proposed fortouch–enabled mobile devices. Navidget [7] presents a 2Dnavigation widget for 3D navigation using a lasso to

define the point of interest and rotate around it. Otherworks, such as Dabr [6], ScrutiCam [5] and Drag’nGo [11] try to propose easier navigation techniques.However these methods do not fully explore the mobilityoffered by mobile devices. The Tether project [1] mimicsthe moving window paradigm using iPad devices.However, this approach requires expensive trackingsolutions. On a complementary note, brain-computerinterfaces (BCI) can enable users to effectively navigatethrough virtual environments. Larrue et al. [9] concludedthat BCI combined with visual information, eliminatesintrusive devices for navigation while promoting userexperiences closer to reality. However BCIs are still farfrom being reliable devices.

Virtual environments enhance spatial perception of 3Dcontent during design activities. Additionally, they canprovide valuable insights to better decision support withrisk mitigation when used in conjunction withcollaboration techniques. Following Daily et al [4] work onvirtual reality for distributed design review, differenton-line collaborative CAD systems [16, 15] have beendeveloped. However their limited interaction capabilitiesprevented wide acceptance [10]. Human-to-humaninteraction techniques to share design ideas instantly arethus crucial to make these systems useful. We believethat technologies such as immersive visualization cansignificantly improve collaborative CAD by providing newpossibilities to both understanding information andinteracting with it.

CEDAR SystemCEDAR proposes a design review tool to supportcollaborative tasks from the oil industry field. We followeda user-centered methodology starting with a task analysisinvolving engineers operating traditional design review

CAD tools on a daily basis. Via both structured interviewsand informal talks, we were able to clearly identify bothchallenges to be tackled and real usage scenarios.

CEDAR server

CEDAR server

Figure 5: CEDAR Systemarchitecture overview.

In oil industry settings, large teams usually gather toreview, manipulate and discuss around large CAD models,which are often difficult to visualize and to navigate owingto their complexity. These teams typically include fieldengineers working at remote locations and technical staffoperation at a central location. Maintenance in specificinfrastructures inside an oil platform is very frequent andtechnical staff have to plan how to access thembeforehand. Furthermore, changes to these installationsare frequent, removing and installing new components.One key goal of ours is to support engineers’ field work byallowing them to visualizing in loco the procedures neededto repair devices while assessing possible interferencesthat are not reflected in the 3D model but exist in the realscene in close cooperation with headquarters.

Setup OverviewOur system was developed and evaluated using twogeographically-distinct facilities based using large scaledisplays with different specifications.

One is a 2x2 tiled display system using four OptomaEW775 projectors in rear-projection mode located inPortugal as depicted in Figure 2. Each projector has 4500lumens and displays 720p @ 120 Hz. The mosaic generates3.6 megapixel stereoscopic images thanks to a smalloverlap (64 px horizontal and 19 px vertical) blending ateach projector. The visualization system is driven by a HPZ820 Workstation coupled with two NVIDIA Quadro 5000behaving as a single desktop and thus simplifying thedevelopment and deployment of applications. The stereosetting uses NVIDIA 3D Vision 2 shutter glasses and anIR emitter located behind the screen.

The second visualization facility is located in Rio deJaneiro (Brazil). It is composed of four Barco NH-12projectors in a trapezoidal arrangement creating acave-like system as shown by Figure 3. While lateralscreens use rear-projection, the floor uses front-projection.Each projector has 12000 lumens and is capable ofrendering 1080p stereoscopic content using Infitec activetechnology. The projectors are connected to a HP Z800Workstation through one NVIDIA Quadro Plex D2. Thedesktop uses the mosaic feature and each screen isoriented side-by-side. Unlike traditional cave-type systems,we prefer a 60◦ lateral glass screen configuration, allowingeither a visualizations by a sizable audience or severalusers to operate within the environment.

System ArchitectureThe CEDAR system follows a star network topologyconnecting several clients to a central server as depictedby Figure 5. We consider two different clients to interactwith the CAD model under revision during a reviewsession. A client running on a desktop computerconnected to a Large Scale Display provides a uniquevisualization environment for several users discussingengineering problems. Another client, running on an iPad,offers a personal window displaying its own view of thescene. This view affords multi-touch 3D navigation andcan be synchronized (or not) to other clients, namely theLarge Screen Display.

The main server coordinates the different clients andmanages the engineering review session, by forwardingboth local and remote communications. By using theclient application on each visualization facility, users canconnect to the main server either locally, or using theInternet and start sharing data between them.

3D NavigationThe CEDAR system offers two natural and easy-to-usealternatives to navigate the virtual environment andmanipulate the visualization. Users can either use gesturescombined with a BCI device to interact directly with theprojection or use an iPad device synchronizing its viewwith the large screen’s as depicted in Figure 4.

Figure 6: Screenshot of userinterface in mobile device.Virtual joystick on the right tocontrol the navigation withbuilt-in sensors.

Figure 7: Remote user avatarrepresentation highlighting atarget viewed by another user inthe large screen display.

Users in front of the Large Screen Display are trackedusing a Microsoft Kinect depth camera, enabling them topoint at specific locations on the 3D scene. Pointinggestures use the Kinect-computed skeleton. A ray is castfrom the user head position to their hand position, overthe screen to define a 2D cursor. Currently, only oneactive user is supported within the range of the Kinectcamera. Using an Emotiv Epoc device, the user cantrigger via a pre-defined brain pattern to move to aspecific location on the 3D scene or issue other commandsto the graphical user interface, such as (un/)sharing theview with remote sites.

The iPad navigation uses the device built-in inertialsensors to aim the camera in a first-person view mode, asif the user were holding a real camera to visualize thescene. Combined with an on-screen joystick, users canmove around the scene as illustrated in Figure 6. Thisjoystick is exponentially scaled to allow both finemovements on the center of its working area and rapidones at its outer edges. This allows for fine navigationwhile examining small structures as well as very fastnavigation through large models. We also provide a”camera-freeze” mode allowing users to freeze the currentview and move around the tablet without their movementschanging the camera position on the large screen andwithout assuming stressful body postures. Further detailsregarding such technique can be found in [14].

Collaborative FeaturesBesides providing 3D model visualization and navigation,the CEDAR system supports several collaborativefeatures. The following aim to improve the collaborationbetween users by providing both direct and indirectcommunication, visibility and awareness.Speech Communication. This allows remote, naturalcommunication between the users, which is crucial to thesuccess of the collaborative tasks;Viewport Sharing. The point of view presented on adisplay can be shared with other users, allowinggeographically apart teams to see the same. Suggesteduses are: Displaying the location of small objects, guidedtours and simply knowing were the other colleagues are;Avatars. An avatar representing the current position andorientation is presented on the environment. This affordsfaster recognition of where other users are located andtheir look–at direction;Target Identification. When users point to specificlocations in the scene, the system highlights target objectsat that position in addition to displaying an avatar (seeFigure 7). Thus, other users can know which object isbeing referred to, without requiring verbal descriptions;Annotations. During reviewing sessions, it is oftennecessary to take notes. Instead of taking these on anotebook or using a different application, CEDAR offersboth sketched (using the iPad) and speech annotations,that can be viewed or replayed later by users.

Current prototypeBoth Large Screen Display and iPad clients implementspecific components to manage hardware,communications and input from users as depicted by thesoftware architecture presented in Figure 8. For eachclient, the User Interface is responsible for providinginteraction support and visual feedback. Both server and

clients were developed on Unity3D game engine. Thisprovides a common framework for the client applicationswhile reducing software integration efforts on differentplatforms. Unity3D also provides a fast and reliable

Input

Hardware

Sensor

Interface

Navigation

Display

3D Scene

Network

Avatar R

epresentation

Rem

ote Visualization

View

port Sharing

User Interface

Figure 8: The CEDAR clientsframework: software componentsused based on Unity3D GameEngine.

support to rendering complex 3D scenes. 3D Scene Unitycomponents are created representing the 3D scenerepository, virtual camera and avatar representations.

Three separate components are responsible for sharingvirtual representation across the network using Unityremote procedure calls to support collaborative tasks. TheAvatar Representation synchronizes the point of viewamong all clients on every frame, i.e. the position and theorientation of all connected users are sent to the serverand are distributed among other clients. This componentworks together with the Remote Visualization, whichpresents remote client’s point-of-view either as a previewor a full–display visualization. The Viewport Sharing isresponsible for broadcasting the current viewport of theclient to connected clients over the network.

The Navigation component implements different travelmethods on the VE depending on the sensor interfaceprovided. On the iPad client side, the Sensor Interfaceuses the iPad built-in sensors to support multi-modalinput. On the Wall client side, the Sensor Interfaceprovides different input modalities. Using Open SoundControl as a network protocol for communication amongdifferent and independent input applications, the system isscalable enough to handle navigational traffic originatingfrom motion sensing input devices (Microsoft Kinect),brain-computer interfaces (Emotiv Epoc) and the moreconventional WIMP input devices.

A preliminary evaluation of our prototype has alreadybeen carried out. We asked different pairs of participantsto guide each other through the model using the iPad

navigation method. While a large formal user evaluationis yet to be done, we gathered valuable qualitativeinformation. Notably, Viewport Sharing was preferredover the Avatar when guiding or following another user.The probable reason for this is that the model has a lot ofobstacles which occlude the Avatar most of the time. It isonly when users are near to each other that they can takeadvantage of the Avatar. To overcome this and get morevisual cues and landmarks, some users navigated to aposition high enough that they could see the whole model.On the other hand, Viewport Sharing can be disorientingsince fast rotational movements performed by the user incontrol can be hard to follow by other users who are notcontrolling the navigation.

Some users resisted to using the sensor–based iPadnavigation, maybe owing to its novelty. However withpractice, they did find it both easier and more fun to usethan traditional touch-based interfaces. The maindrawbacks were the thumb-stick metaphor on the screenwhich caused some problems to users less experiencedwith game-inputs-devices and how tiring the handlingposition could prove after prolonged use. To address thelatter, we proposed to relax the virtual axis and to use thecamera freezing feature to work as a clutch, which wasalso useful to prevent awkward user body postures.

Conclusions and Future WorkThis paper explores collaborative 3D visualization on largeand small screen displays. We presented our setupcomposed of two (extensible to more) geographicallydisjoint large screen stereo displays. To navigate thevirtual environment, we propose two alternatives: iPads,that besides offering multi-touch navigation can work as asecond personal window on (or off) the scene while ofdepth cameras and brain-computer interfaces provide

novel approaches to interaction. To support remotecollaboration between users, we developed a set of specificgroupware features. While our preliminary evaluationprovided positive results and invaluable feedback fromusers, further extensive user tests of the system are inorder. Our current and future work focuses on improvingthe prototype, both in terms of user collaboration andimmersion.

AcknowledgementsThis work was partially supported by the PortugueseFoundation for Science and Technology (FCT): projectCEDAR PTDC/EIA-EIA/116070/2009, project 3DORuSPTDC/EIA-EIA/102930/2008 and projectPEst-OE/EEI/LA0021/2011.

References[1] M. Blackshaw, D. Lakatos, H. Ishii, and K. Perlin.

T(ether) a tool for spatial expression, 2012.http://kiwi.media.mit.edu/tether/.

[2] R. A. Bolt. ’put-that-there’: Voice and gesture at thegraphics interface. In Proceedings of SIGGRAPH ’80,pages 262–270. ACM, 1980.

[3] S. Boring, D. Baur, A. Butz, S. Gustafson, andP. Baudisch. Touch projector: mobile interaction throughvideo. In Proceedings of CHI ’10, pages 2287–2296.ACM, 2010.

[4] M. Daily, M. Howard, J. Jerald, C. Lee, K. Martin,D. McInnes, and P. Tinker. Distributed design review invirtual environments. In Proceedings of CVE ’00, pages57–63, NY, USA, 2000. ACM.

[5] F. Decle, M. Hachet, and P. Guitton. Tech-note:Scruticam: Camera manipulation technique for 3d objectsinspection. In Proceedings of 3DUI ’09, pages 19–22.IEEE Computer Society, 2009.

[6] J. Edelmann, A. Schilling, and S. Fleck. The dabr - amultitouch system for intuitive 3d scene navigation. In3DTV Conference, pages 1 –4, may 2009.

[7] M. Hachet, F. Decle, S. Knodel, and P. Guitton.Navidget for easy 3d camera positioning from 2d inputs.In Proceedings of 3DUI ’08, pages 83–89. IEEE ComputerSociety, 2008.

[8] R. Jota, J. M. Pereira, and J. A. Jorge. A comparativestudy of interaction metaphors for large-scale displays. InProceedings of CHI EA ’09, pages 4135–4140. ACM,2009.

[9] F. Larrue, H. Sauzeon, L. Aguilova, F. Lotte, M. Hachet,and B. NKaoua. Brain computer interface vs walkinginterface in vr: the impact of motor activity on spatialtransfer. In Proceedings of VRST ’12, pages 113–120.ACM, 2012.

[10] W. Li, W. Lu, J. Fuh, and Y. Wong. Collaborativecomputer-aided design: research and development status.Computer-Aided Design, 37(9):931–940, 2005.

[11] C. Moerman, D. Marchal, and L. Grisoni. Drag’n go:Simple and fast navigation in virtual environment. InProceedings of 3DUI ’12, pages 15–18. IEEE, 2012.

[12] M. Nancel, J. Wagner, E. Pietriga, O. Chapuis, andW. Mackay. Mid-air pan-and-zoom on wall-sized displays.In Proceedings of CHI ’11, pages 177–186. ACM, 2011.

[13] T. Ni, G. Schmidt, O. Staadt, M. Livingston, R. Ball, andR. May. A survey of large high-resolution displaytechnologies, techniques, and applications. In VirtualReality Conference, pages 223 – 236, march 2006.

[14] H. Noronha, P. Campos, J. Jorge, B. d. Araujo, L. Soares,and A. Raposo. Designing a mobile collaborative systemfor navigating and reviewing oil industry cad models. InProceedings of NordiCHI 2012. ACM, 2012.

[15] M. Tang, S.-C. Chou, and J.-X. Dong. Conflictsclassification and solving for collaborative featuremodeling. Advanced Engineering Informatics,21(2):211–219, 2007.

[16] F. E. H. Tay and A. Roy. Cybercad: a collaborativeapproach in 3d-cad technology in a multimedia-supportedenvironment. Computers in Industry, 52(2):127–145,2003.