Visualizing and Analyzing the Mona Lisa

60 November/December 2007 Published by the IEEE Computer Society 0272-1716/07/$25.00 © 2007 IEEE

Real-Time Interaction with Complex Models

A s technologies for acquiring 3D data andalgorithms for constructing integrated

models evolve, very large data sets representing objectsor environments are emerging in various applicationareas. As a result, significant research in computergraphics has aimed to interactively render such mod-els on affordable commodity computers. Interest isgrowing in the possibility of integrating real-timeanalysis and transformation tools in interactive visu-alization environments as they become more available.

Large meshes built from measurements are complexnot only because of their typically huge size but alsobecause of their multiple scales of information. High-resolution and high-precision 3D scanning providesinformation not only about overall shape and color, butalso about fine surface details and variations that must

be accessed within a larger contextand can be hidden within global sur-face features. This problem differsslightly from that of large modelscomposed of complex collections ofabstract geometric entities, such asCAD models. But a common taskremains: to efficiently access infor-mation at a variety of scales, and topresent it at different interest levels,without creating visual artifacts orpotential misinterpretations. Here,accessing the data means not onlydisplaying it photorealistically, butalso transforming it to enhance the

viewer’s understanding of its contents, and thereforemaximize the data set’s value (see the “Previous Workin 3D Modeling for Cultural Heritage” sidebar).

This article presents the set of interactive graphictools that we used to assist experts in analyzing adetailed complete 3D scan of Leonardo Da Vinci’s MonaLisa (see the “Scanning the Mona Lisa” sidebar). Thetechniques we used in the application project aimed toboth help document this famous artwork and betterunderstand Leonardo’s painting technique. Our toolslet us interactively explore the data on commodity hard-ware using various representations and at differentscales.

Multiresolution modelingFigures 1a and 1b show NRC’s high-resolution poly-

chromatic laser scanner we used to digitize the MonaLisa. We assembled the Mona Lisa laser scans into a 333-million polygon color-per-vertex model (see Figure 1c).The model represents the entire poplar panel on whichthe painting was executed, with higher resolution onthe obverse (front) surface. Such a model is still too largefor even the most recent high-end graphics hardware,and this is in no way an exceptional situation. By com-bining sensors and digital photography, one can pro-duce models of several hundred millions of 3D samplesand hundreds of gigabytes of 2D images in a matter ofdays, if not hours.

Our general technique for interactively displayingsuch large scanned surface data sets easily handles mod-els composed of hundreds of millions of polygons andtens of gigabytes of associated texture data.1 Themethod is an extension of view-dependent hierarchicallevels of detail (LODs), where we use geomorphing toensure temporal and spatial continuity between the dif-ferent LODs and across the entire model.

Our rendering technique combines several importantadvantages for this application context.

First, by using static preoptimized geometry units asthe primitives for the view-dependent computations,we strike a better balance between GPU and CPU usage.Thus, we can benefit from on-GPU geometry and tex-ture caching and from better use of the GPU vertexcache. On a model such as the Mona Lisa, we can ren-der more than 100 million geomorphed polygons persecond when computing per-vertex lighting. This per-formance level lets us render the multiresolution mod-els with subpixel accuracy at interactive frame rates.

We also minimize the visual artifacts associated withthe view-dependent transformation of the displayeddata through geomorphing. One of the known inconve-niences of LOD-based approaches is the visible transi-tional artifacts associated with changes in the resolutionlevel. When rendering a model such as the Mona Lisa athigh resolution, no artifacts are visible even if we set atarget resolution slightly over one pixel.

In addition, our technique requires little CPU usagebecause we perform all geomorphing on the GPU. We use

Size and scale issues present acomplexity problem invisualizing detailed 3D modelsbuilt from sensor data. A modelof Leonardo da Vinci’s MonaLisa, with its thin pictorial layer,illustrates the need for intuitivereal-time processing tools thatare seamlessly integrated with amultiresolution visualizationenvironment.

Louis Borgeat, Guy Godin, Philippe Massicotte,Guillaume Poirier, François Blais, and J.-Angelo BeraldinNational Research Council of Canada

Visualizing andAnalyzing the Mona Lisa

the CPU only to cull the coarse hierarchical LOD structure,as with any scene graph, and to prefetch data during nav-igation. Resources are therefore available for other taskssuch as audio/video transcoding for telecollaboration (seethe “Displaying the Imagery” sidebar on page 63).

Finally, we display the original full-resolution modelat the finest level of detail. This ensures that the usercan differentiate between real features present in thedata and potential artifacts that could be caused by theprocessing and rendering steps associated with manag-ing a large data set.

Before we can interactively display a model withthis tool, we must precompute the appropriate mul-

tiresolution data structure. For a model the size of theMona Lisa, this process will take a few hours.1 Prepro-cessing converts the triangular mesh model into acompressed LOD hierarchy of optimized geometrypatches and associated texture or attribute data. Thefirst step of this preprocessing is to simplify the entiremodel into a sequence of discrete LODs using an algorithm based on vertex aggregation. This choice ofsimplification technique is important because theaggregation paths become the geomorphing paths inthe real-time rendering phase. We then decomposethe lowest resolution LOD into a set of triangle patch-es. We partition the next higher resolution level along

IEEE Computer Graphics and Applications 61

1 Creating a model of the Mona Lisa: (a and b) We used our high-resolution polychromatic laser scanner to digitize the painting. (c) We can then render the model using the 3D and color data.

Previous Work in 3D Modeling for Cultural Heritage

The cultural heritage field has provided a stimulatingtesting ground for many developments in 3D acquisition,modeling, and display. Elsewhere, we review surface sensor-based data-set-rendering techniques of large models.1

Large-scale cultural heritage documentation projectsinclude the Digital Michelangelo Project,2 and the scanningof the Florentine Pietà3 and of the Great Buddha ofKamakura.4 A description of other projects in culturalheritage and archaeological applications is availableelsewhere.5 Some work has also aimed to enhance therendering of 3D models for analysis purposes. Andersonand Levoy process and enhance models of cuneiformtablets using curvature and accessibility coloring.6 Cohen etal. use interactive 2D+ visualization combined withnonphotorealistic lighting to also analyze such tablets.7

Rusinkiewicz et al. describe a nonphotorealistic renderingtechnique to highlight shape details of 3D models that’sderived from hand-drawing shading techniques.8

When sufficient resolution and precision become available,paintings can no longer be considered flat 2D objects.Rather, they enter the realm of 3D imaging applications andbecome massive data sets comparable to those obtained onsculptures. Our work on modeling the complete Mona Lisafrom laser scans is one such recent project (see the “Scanningthe Mona Lisa” sidebar on the next page).

References1. L. Borgeat et al., “Gold: Interactive Display of Huge Colored and

Textured Models,” Proc. Siggraph, ACM Press, 2005, pp. 869-877. 2. M. Levoy et al., “The Digital Michelangelo Project: 3D Scanning

of Large Statues,” Proc. Siggraph, ACM Press, 2000, pp. 131-144.3. F. Bernardini et al., “Building a Digital Model of Michelangelo’s Flo-

rentine Pietà,” IEEE Computer Graphics and Applications, vol. 22,no. 1, 2002, pp. 59-67.

4. D. Miyazaki et al., “The Great Buddha Project: Modeling Cultur-al Heritage through Observation,” Proc. 6th Int’l Conf. Virtual Sys-tems and Multi-Media (VSMM), Int’l Soc. on Virtual Systems andMultimedia, 2000, pp. 138-145.

5. G. Godin et al., ”Active Optical 3D Imaging for Heritage Applica-tions,” IEEE Computer Graphics and Applications, vol. 22, no. 5,2002, pp. 24-36.

6. S.E. Anderson and M. Levoy, “Unwrapping and VisualizingCuneiform Tablets,” IEEE Computer Graphics and Applications, vol.22, no. 6, 2002, pp. 82-88.

7. J. Cohen et al., “iClay: Digitizing Cuneiform,” Proc. Symp. VirtualReality, Archaeology, and Cultural Heritage (VAST), A.K. Peters,2004, pp. 135-143.

8. S. Rusinkiewicz, M. Burns, and D. DeCarlo, “Exaggerated Shad-ing for Depicting Shape and Detail,” ACM Trans. Graphics (Proc.Siggraph), vol. 25, no. 3, pp. 1199-1205.

(a) (b) (c)

the same boundaries, and subpartition each group inthis level until we reach the desired granularity. Weapply this process sequentially to all levels of themodel, which gives us a hierarchy of group subdivi-sions spanning the whole sequence of LODs (see Fig-ure 2a).

We can shape groups according to criteria such ascompactness, common orientation, texture/viewpointassociation, culling requirements, model structure,existing partitions, and number of primitives per group.These criteria change depending on the model type to bedisplayed. For example, for a model with much more

texture than geometry, we might create patches thatsegment images in texture space to minimize the sizeand number of texture units. For a color-per-vertexmodel such as the Mona Lisa, we might prefer to createmore compact patches to minimize errors in the view-dependent computations. Finally, we individually con-vert groups into triangle strips optimized for the GPUvertex cache to maximize rendering speed. Thesegroups or patches constitute the basic units for all theview-dependent computations.

The runtime culling process selects a front in theLOD patch hierarchy (see Figure 2b, bottom).The


62 November/December 2007

Scanning the Mona LisaAt the request of the Louvre’s Paintings Department, the

French Center for Research and Restoration for Museums(C2RMF) undertook the largest scientific examination everconducted on the Mona Lisa. This study coincided with the painting’s move to the newly renovated Salle des États.A team of 39 specialists with backgrounds in art history,conservation, photography, physics, engineering, chem-istry, optics, and digital imaging from seven institutionstook part in various aspects of the study.

As part of this project, a team of 3D imaging scientistsfrom the National Research Council of Canada (NRC) wasinvited to Paris in October 2004 to undertake the 3Dscanning of Leonardo’s most famous painting. Theobjective was to acquire a high-resolution 3D image dataset of the obverse, reverse, and sides of the Mona Lisa tobuild a model of the complete painting. Leonardo paintedthe Mona Lisa on a poplar panel measuring 79.4 � 53.5 cm.The resulting 3D data and model provide a digital record ofthe painting’s condition, and help in the computerizedstudy of Leonardo’s structure and technique.

The Mona Lisa presented a unique research anddevelopment challenge for 3D-imaging technologies,because of the flat pictorial layer and absence of apparentbrush strokes, and the surface’s translucency. Moreover, theteam had access to the painting for a few hours over twonights, thus requiring careful planning of the scanning

operations. Restorers performed all manipulations of theMona Lisa, and the painting’s environmental conditionswere monitored at all times. Preliminary results arepublished elsewhere.1,2

The complete model of the painting contains 333 milliontriangles, with an average sampling density of 60 �m on theobverse. The NRC’s scanning system measures the surfacegeometry by triangulating a projected low-power laserspot. The laser used is a mixture of wavelengths, which aredecomposed at observation to estimate the surface color inperfect registration with the geometric measurements. Thismeasurement process leads naturally to a color-per-vertexrepresentation. We achieved complete surface coverage bymultiple overlapping scans that are integrated into a unifiedmesh. Researchers can then examine the 3D and colormodel using the techniques we’ve described, either in arealistic manner or in transformed representations thatenhance features of interest.

References1. J.-P. Mohen, M. Menu, and B. Mottin, eds., Mona Lisa, Inside the

Painting, Abrams, 2006.2. F. Blais et al., “More than a Poplar Plank: The Shape and Subtle

Colors of the Masterpiece Mona Lisa by Leonardo,” Proc. SPIE,vol. 6491, SPIE, 2007.

2 Modeling processes. (a) The recursive group subdivision process across the level-of-detail (LOD) sequence. (b) Three representationsof the same view-dependent rendering of the model. The top image shows a simple color rendering. In the middle image, each LOD isassociated with a unique color. We color groups by blending the colors of the two levels between which they’re morphing, according totheir current geomorphing ratio. The bottom image shows the connected assembly of selected patches spanning three LODs.

(a) (b)


algorithm selects groups based on a chosen maximumscreen pixel error and a unique worst-case precom-puted error measurement associated with each group.We compute geomorphing ratios between selectedgroups and their lower resolution parent for eachframe. Geomorphing boundary points betweengroups maintain seamless continuity between neigh-boring LOD groups at all times according to a specif-ic set of rules. A vertex program on the GPU morphsthe other points using a uniform ratio per individual

patch, maintaining a more uniform on-screen resolu-tion (see Figure 2b, middle). Because this program ison the GPU, this process requires almost no CPUresources. We apply geomorphing to spatial and tex-ture coordinates, normals, and colors, and performout-of-core prefetching asynchronously on a differentthread to ensure smooth navigation and interaction.We assign groups for prefetching during culling byselecting another front in the LOD hierarchy, this timeusing a lower pixel-size error.

Displaying the ImageryHigh-resolution models yield high-resolution imagery,

which should be appropriately displayed to be fullyappreciated. When in a workstation configuration, we canleverage our data sets by using one or two relativelyaffordable digital 30-inch screens with a 2,560 � 1,600-pixel resolution. However, when working collaboratively onlarger screens, we’re quickly limited by commodityprojectors’ much lower resolution. Indeed, a 1,024 � 768

meeting-room projector is more expensive than our high-resolution digital display, but displays five times fewerpixels. Typical solutions to this problem involve usingmultiple projectors and image tiling to obtain moreresolution. But such setups can become expensive andrequire large installations, especially in the context ofstereoscopic visualization, which requires either doublingthe number of projectors for passive techniques, or usingtime-multiplexed stereo projectors.

We’ve developed an alternative technique that combinesstereoscopic visualization with a focus�context, or foveatedapproach (see Figure A). On a large display wall, we projecta high-resolution, high-brightness stereo image within alarger, lower-resolution one. A pair of commodity PCs andprojectors produces each image. Such a setup combinesthe advantages of accessing a high-resolution area of inter-est while keeping a larger context within sight. Additionaltechniques ensure that the inset’s presence doesn’t affectthe stereoscopic perception along its boundary, and thataligning the projectors doesn’t require special manualefforts.1

We’re also developing an application that integrates ourrendering and display tools with other components usefulin processing and analyzing 3D data sets. We’ve alreadyincluded modalities to support collaborative work on ourdisplay wall.2 Figure B illustrates a collaborative session inwhich participants can share a virtual world based on amultiresolution model, and interact by inserting X3Davatars with video insets and audio links in theenvironment. Participants can annotate the model using 3Ddrawing and guide the viewpoint of other participants tocollaborate. We designed the application with large datasets in mind, so we’ve implemented many features tooptimize resource usage. For example, the GPU partiallyencodes and decodes video as part of the 3D rendering,and all the 3D annotations are done using information fromthe depth buffer so there’s no need to compute rayintersection directly on data sets. We also update the scenegraph using efficient differential encoding.

References1. G. Godin, P. Massicotte, and L. Borgeat, “Foveated Stereoscopic

Display for the Visualization of Detailed Virtual Environments,”Proc. 10th Eurographics Symp. Virtual Environments, ACM Press,2004, pp. 7-16.

2. L. Borgeat et al., “Collaborative Visualization and Interaction forDetailed Environment Models,” Proc. 10th Int’l Conf. Virtual Sys-tems and Multimedia, OCSL Press, 2004, pp. 1204-1213.

A A foveated display used to collaboratively analyze high-resolution 3D models (shown here in monoscopic mode for clarity).

B A collaborative environment for remote interaction withlarge 3D data sets.



Different representations from a single viewpoint

Current GPUs provide much more control of theimage-generation pipeline than their early counterparts.They’re also significantly more powerful and can per-form more real-time computations. We can harness thisadditional power to further process the rendered mul-tiresolution model by implementing transformationsuseful as analytical tools. Such hardware will let usimplement simple techniques such as rendering en-hanced depth information instead of color to visualizeother aspects of the data sets. We can also perform, in realtime, complex multistep filtering and image-compositiontechniques that would require significant effort andmanipulation in conventional 2D image-processing toolssuch as Adobe Photoshop. Recent GPUs also provide 32-bit float buffers as rendering targets, so we can executehigh-precision computations and measurements direct-ly on the GPU.

When rendering large data sets, implementing suchmultipass filters on the GPU doesn’t necessarily result ina linear increase of the rendering time with the numberof required passes. Passes in which we actually render themultiresolution mesh are costly because they use a largeamount of graphics memory and involve processing mil-lions of primitives. But we can often implement complexrendering stages by processing the mesh once and work-ing iteratively on intermediate results. Those results areonly pixel data buffers, not 3D models, located in fastgraphics memory that we can rapidly transform and com-bine. We’ll describe, as an example, a real-time filter thatwe implemented so we could separate small detail vari-ations in an object’s local shape at a given resolution fromits global structure, acting as a high-pass filter for the dataat the current LOD. In this representation of the MonaLisa, we in effect separate the surface variations of thewood grain structure and paint layer from the moreimportant warping of the poplar panel. Figure 3 illus-trates some steps in this process.

First, we render the multiresolution model from thechosen viewpoint. This process is similar to that used

for the photorealistic mode. We use the same interpo-lation process between LODs as when processing thevertices. However, in this case the target buffer is a sin-gle-channel 32-bit float buffer. At the end of the firstpass, we obtain a 32-bit depth image of the 3D data ineye space. Figures 3a and 3b show the results of render-ing both color and depth from the same viewpoint. Werescaled the depth in the figure across the entire gray-scale spectrum for illustrative purposes: this is analo-gous to the technique used to produce Figure 4b and thedepth component of Figure 4c.

The problem with such an image is that the humaneye can only distinguish a limited number of shades ofgray. Even if small shape details are present in this 32-bit high-precision image, they aren’t visible becausethey’re hidden in the panel’s global warping, which con-sumes the dynamic range’s most significant bits. Onlythe warping is apparent in the figure. We therefore needa way to highlight these details. The simplest solutionis to virtually unwarp the poplar panel so other detailscan emerge.

We achieve this by first creating a representation ofthe overall shape in the form of a second float image thatwe obtained by convolving the depth image with a largeGaussian blur kernel. We can efficiently implement con-volution on a GPU2 by filtering sequentially with a uni-dimensional kernel along both image axes, resulting infewer texture fetches. At the scale at which we apply thefilter, we effectively filter out all the fine details we seekto recover, while preserving the panel’s global warping.Because the details we want to filter out aren’t visibleon the gray image, we also apply the filter to the corre-sponding color image for illustrative purposes (see Fig-ure 3c).

Next, we subtract the original image from the filteredone in a new rendering pass. This process results in animage containing only the local variations that were con-tained in the original buffer. We now simply need totransform those small variations into color values fordisplay. Because the GPU must rescale values betweenzero and one to produce color values, we need to find

10 14

1216139

8415

7326

1115 1 16

1 6 3 8

9 14 11 16

3 The rendering process: (a) a rendering ofthe Mona Lisa model, (b) a rendering of thedepth from the same viewpoint, (c) the samerendering of the color with a Gaussian filterapplied, and (d) the result obtained afterapplying our detail highlight operator. (e) Thereduction process involves aggregating neigh-boring pixels in successive rendering passesinto smaller and smaller buffers. Minimumvalues are accumulated in the red channel,maximum values in the green channel.

(a) (b)

(c) (d)

(e)


the minimum and maximum values in our depth image.The fastest way to achieve this on a GPU is to hierarchi-cally combine neighboring pixel values by renderinginto smaller buffers until we’re left with a one-by-oneimage containing our result. This technique is called par-allel reduction.2 We use two color channels to find theminimum and maximum at the same time. Figure 3eillustrates this reduction process. We can map the finalresult to a color or gray-scale spectrum (see Figure 3d).In practice, for a typical rendering at 2,560 � 1,600 pix-els with subpixel geometry resolution, we observe areduction in rendering speed by a factor between 2 and3, even if we perform six full passes (plus the smallerones for the reduction) instead of a single pass.

This example emphasizes the strong advantage wegain from working in our viewer’s 3D multiresolutionspace instead of working with a 2D image-processingtool such as Photoshop or the GNU Image ManipulationProgram. Indeed, to achieve the same image processingas described, we would have needed to first render the3D model’s depth into a 2D image, then crop it andchoose appropriate filter sizes, create and combine lay-ers, and apply histogram manipulations. In 3D, we spec-ify the kernel filter’s resolution in terms of screen pixels.Thus, by zooming into the model, we naturally crop theview and select the filter size in model units. We alsoensure that the amount of data to be processed is con-stant without having to manipulate multiple storedintermediate representations, or to crop and explicitlywork on multiple versions on the image in parallel. Wecan modify the viewpoint and instantly switch from onerepresentation to the other while still navigating in 3D.Filter parameters are interactively modifiable within thenavigation application. This greatly increases our abil-ity to efficiently analyze the models. These advantagesalso apply to all filters used to produce the results in thefollowing sections.

Many faces of the Mona LisaTo people who approach heritage preservation from

the computer graphics or remote-sensing world, remov-ing an object’s color to explore its shape using synthet-ic shading is a given, if not the default representationmost often associated with laser scanning of objects andscenes. But for curators used to working with photo-graphs or directly with the original artifacts, this sim-ple representation often comes as a revelation. Indeed,to them, a mode in which there are no colors or ambigu-ous shadows can be of even greater interest than a pho-torealistic rendering. Raking light is a common tech-nique for observing and analyzing a painting’s surfaceshape. This technique is more often than not limited by ambiguities produced by cast shadows and color variations, or difficulties posed, for example, by globalwarping in the surface. Applying synthetic shading onlyon the model geometry conceptually corresponds to raking light without cast shadows (unless they’re explic-itly computed) on a uniformly colored surface, thusrevealing surface details without being affected by actual surface color variations.

Using various representations, we illustrate someinteresting technical aspects of Leonardo’s famous

painting. A more in-depth analysis and historical inter-pretations of sensor imagery of the Mona Lisa are avail-able elsewhere.3,4 All images in this section aresnapshots from Atelier3D, our prototype interactivevisualization and analysis tool.

The wood panelOne of the 3D scan’s key goals was to precisely mea-

sure the shape of the poplar panel on which the MonaLisa is painted. This would let conservators documentthe painting’s current condition and help to assess itsconservation state. Figure 4 shows different represen-tations for visualizing the panel’s global shape. Figures4b and 4c are based on encodings of the model’s depthrelative to the observer. In Figure 4c, we blended thedepth information with color data to help correlate theshape and appearance. To facilitate navigation, the Atelier3D interface provides tools to perform principalcomponents analysis on the model’s visible part to alignthe observer with the inspected data. This process isequivalent to fitting a plane within the depth buffer andrealigning the viewing axis perpendicularly to it. Thisalignment tool is important for observing detailsbecause, as we stated earlier, we can only observe a lim-ited number of color differences. The interface lets usersadjust the histogram to obtain more detail in certainareas, similar to what’s done in 2D tools. We implementthe 3D cutting tool (see Figure 4a) at minimal cost onthe GPU using either clipping planes or explicit geome-try computations in the fragment program. Thosedynamic cut tools are an intuitive way of understand-ing large-scale shape variations, especially when addinga reference frame to the scene. GPUs now support fullfloating-point buffers, so we can easily extract precise

4 Global shape of the poplar panel: (a) cut of thepainting model with a virtual reference plane, (b)shape of the panel with color depth-coding after align-ing the viewpoint using principal components analysis,and (c) contour lines overlaid on a depth and colorblend.

(a)

(b) (c)

measurements or profiles in real time from the buffersused to render those images.

Figure 5 shows the result of applying the local varia-tion filter. This image shows the panel’s wood grainstructure in a less ambiguous way than when using rak-ing light techniques. Because we apply the filter at thepixel resolution chosen by navigating the 3D space, wecan easily process the image in real time while navigat-ing. The kernel size (in model units) adapts naturally aswe zoom to areas of interest. Again, performing theequivalent task in a 2D tool from an initial full-size depth

rendering would involve selecting and cropping areas,applying the filter to data that’s at an excessive resolu-tion, and waiting for the result. We’d then have to fig-ure out numerical values for the filters, and so on, notto mention that many of the common imageediting/processing software tools don’t operate on 32-bit floating-point images.

The paint layerThe paint layer’s most striking conservation aspect is

the craquelure visible over the entire painting. Craque-lure can indicate many phenomena of interest to con-servators, such as panel deformation, improper paintingtechnique, or normal paint aging. We can easily extractthe 3D craquelure patterns using edge detection or otherfilters, or highlight the patterns using specularity toshow both color and shape at the same time, as in Fig-ure 6b.

Art historians and curators have expressed interestin understanding Leonardo’s technique in painting theMona Lisa. Figure 6a is a local-variation depth render-ing similar to Figure 5, but at a finer scale. The goal isto provide some indication concerning local variationsin the paint layer thickness. We can clearly see thathigher areas generally correspond to darker sections ofthe face’s actual color. One theory is that Leonardoapplied extremely thin successive layers of tinted glazeto produce those fine lighting and shading effects. Thistechnique is called sfumato, which means “vanished”in Italian. The image tends to confirm this hypothesis.One visible result of Leonardo’s subtle painting tech-



6 Paint layer views. (a) Local elevation variations in the face area. (b) Using virtual specularity on the paintingmodel is a good way to display information about the shape and color at the same time. The actual painting is alsospecular. (c and d) Detail of the pictural layer of a Renoir painting. Contrary to the Mona Lisa, brushstroke detailsare easily visible in the 3D data. (e) Separation of the color channels. Some details of the composition areenhanced at selected wavelengths. In this case, details of the hair are more visible in longer wavelengths.

5 Local depthvariations of thepainting surfacecomputed at aglobal scale.Wood grain and paint layerthickness variations are clearly visible.

(a) (b)

(c) (d) (e)

nique is that no brush strokes are apparent onthe paint layer. To demonstrate the relevanceof using laser scanning for painting surfaceanalysis, we compare this surface with thework of another renowned painter, Pierre-Auguste Renoir. Figure 6d shows an artificial-ly shaded area of Renoir’s Femme nue dans unpaysage (Nude woman in a landscape). Wecan compare this shading to the bottom partof Figure 7a, similarly produced from theMona Lisa model.

The 3D laser-scanning system we designedand used simultaneously provides color mea-surement in registration with the geometricinformation. The reflectivity at differentwavelengths and different scattering levelsinfluenced these readings. We can gatherinteresting information by looking separate-ly at the painting’s three color channels. Fig-ure 6e shows the three color channels for asingle viewpoint. The red image gives a muchbetter view of the fine hair details, which aremuch more difficult to see in the compositecolor version.

Traces of historyIn addition to the craquelure pattern and the varnish’s

yellowing, time has left us with several interesting fea-tures to visualize on the Mona Lisa. The most obvious oneis an 12-cm split on the top of the wood panel for whichmany restorations have been done over the years. Thoseare mostly visible on the back of the panel (see Figure 8).Conservationists have inserted butterfly-shaped wood-en pieces transversally to stabilize the painting. Wormholes are visible on sides of the wood panel (Figures 7aand 7b). Many other smaller restorations are also appar-ent on the surface. To find these restorations, we couldlook in independent color channels. Even if the paint usedhas the same color as the painting under normal lighting,it might have been executed using pigments differentfrom the original, and therefore exhibit a different spectral response. We canobserve these differenceswhen using different lighting�a phenomenoncalled metamerism. Thesecolor differences whereenhanced much furtherusing other imagingmodalities as part of theoverall documentationproject.3

One enduring myth isthat thieves cut the MonaLisa wooden panel whenthey stole it in 1911. The3D data in Figure 7a caneasily debunk this myth,which probably originatesfrom the existence ofcopies of the painting thatincluded wider columns.

We can see the accumulation of paint that formed alongthe original support frame. If the painting had been cut,that crest along the original frame’s border would nolonger be present. We can also see in that image a fewblue specks, highlighted in Figure 7b. Those are believedto be pigments that were protected from aging by anoth-er frame, and could indicate the sky’s original color,which turned from blue to green as the varnish yellowedand the painting became much darker.

Conclusion3D imaging technologies play an increasingly impor-

tant role in numerous application domains, such asmanufacturing, architecture and infrastructure, andsecurity. 3D sensors related to the one used for the MonaLisa project are now used in space to inspect the shuttle


7 Paint layer views: (a) paint layer border, and (b) magnifiedview of the bluish pigment. Paint accumulation is on the side,confirming that the painting was never cut. The blue area mightbe much closer to the original sky color.

(a) (b)

8 Restoration of the 12-cm split in the poplar panel. (a) Rendering of the back of the painting. (b) Virtualraking light on a close-up of the split area, showing the consolidation work.

(a) (b)

before re-entry. The need to visualize and interactivelyanalyze very large data sets is expanding with such datasets’ size and complexity. The software that we devel-oped in the specific context of the Mona Lisa project pro-vides us with a generic framework that can be appliedto those other fields.

One key aspect of 3D data analysis not discussed inthis article is the sequence of modeling steps that trans-forms raw sensor data into a complete and valid modelready to be processed for visualization and analysis. Thismodeling process is often long and tedious, and mostcommercial software currently fails to handle very largeor overly complex models. Our future work will aim toprovide tools that scale better with data size and com-plexity, and minimize the amount of human interven-tion required in the modeling phase. ■

AcknowledgmentsWe thank our colleagues Luc Cournoyer, Michel

Picard, Louis-Guy Dicaire, Marc Rioux, John Taylor, andDaniel Gamache. We also thank Christian Lahanier andhis colleagues at the French Center for Research andRestoration for Museums (C2RMF) in Paris, as well asthe Paintings Department of the Louvre Museum.

References 1. L. Borgeat et al., “GolD: Interactive Display of Huge Col-

ored and Textured Models,” ACM Trans. Graphics (Proc.Siggraph) vol 24, no. 3, ACM Press, 2005, pp. 869-877.

2. R. Fernando, ed., GPU Gems: Programming Techniques,Tips, and Tricks for Real-Time Graphics, Addison-Wesley,2004.

3. J.-P. Mohen, M. Menu, and B. Mottin, eds., Mona Lisa,Inside the Painting, Abrams, 2006.

4. F. Blais et al., “More than a Poplar Plank: The Shape andSubtle Colors of the Masterpiece Mona Lisa by Leonardo,”Proc. SPIE, SPIE, vol. 6491, 2007.

Louis Borgeat is an associateresearch officer with the Visual Infor-mation Technology Group at theNational Research Council of Cana-da. His current research interestsinclude modeling and visualization oflarge sensor data sets, telecollabora-tion, real-time computer graphics,

and 3D computer vision. He is completing is PhD in elec-trical engineering and has an MS in computer engineeringfrom Laval University. Contact him at [email protected].

Guy Godin is a senior research offi-cer with the Visual Information Tech-nology Group at the NationalResearch Council of Canada. Hisresearch interests include 3D comput-er vision and image analysis, shapeand appearance modeling, and inter-active visualization. He has a MEng

in electrical engineering from McGill University. He is amember of the IEEE and ACM. Contact him [email protected].

Philippe Massicotte is a comput-er scientist with the Visual Informa-tion Technology Group at the NationalResearch Council of Canada. Hisresearch interests include modelingand interactive visualization of largedata sets, multiresolution displays,and virtual environments. He holds a

BSc in computer science from the Université du Québec àHull. Contact him at [email protected].

Guillaume Poirier is a computerscientist with the Visual InformationTechnology Group at the NationalResearch Council of Canada. Hisresearch interests include real-timecomputer graphics, nonphotorealisticrendering and perception, andhuman–computer interaction. Poirier

has a MMath degree from the University of Waterloo. Con-tact him at [email protected].

François Blais is a principalresearch officer with the NationalResearch Council of Canada. Hisresearch interests cover various fieldsin digital and image processing, con-trol, 3D, vision systems, and theirapplications. He has an MSc in elec-trical engineering from Laval Univer-

sity, Quebec City. Contact him at [email protected].

J.-Angelo Beraldin is a seniorresearch officer at the Visual Informa-tion Technology Group in the Instituteof Information Technology at theNational Research Council of Cana-da. His research interests include sen-sor systems as well as engineering andsignal processing for 3D vision systems

and their application to the documentation of cultural her-itage. He has an MS in electrical engineering from the Uni-versity of Ottawa. Contact him at [email protected].



Video Web ExtraView the author’s video supplement to this article at

http://doi.ieeecomputersociety.org/10.1109/MCG.2007.162

Visualizing and Analyzing the Mona Lisa

Documents

Transcript of Visualizing and Analyzing the Mona Lisa