Metrics-Based Approach for VR Technology Evaluation in Styling Product Design

1 Copyright © 2009 by ASME

Proceedings of the ASME 2009 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference

IDETC/CIE 2009 30 August-2 September, 2009, San Diego, California, USA

DRAFT DETC2009-86228

METRICS-BASED APPROACH FOR VR TECHNOLOGY EVALUATION IN STYLING PRODUCT

DESIGN

Michele Germani PhD Researcher

Department of Mechanical Engineering Polytechnic University of Marche

Ancona, Italy [email protected]

Margherita Peruzzini PhD candidate



Maura Mengoni PhD Researcher



ABSTRACT In recent years, the interest of small and medium sized enterprises towards Virtual Reality (VR) systems is strongly increased thanks both to the improvement of VR tools effectiveness and to the cost reduction of technologies implementation. Due to the growing number of installed systems, many SMEs companies require robust methods for evaluating technology performance. In this context, the present paper presents a metrics-based approach in order to analyze the VR system performance. It is specifically dedicated to the design review process during styling product design. The evaluation parameters are related to the effective communication and preservation of design intent. Metrics are classified in two main classes. The first one is related to the product, the process and the characteristics of VR technology. The second one is related to the design intent meanings preservation along the design process. Two experimental case studies are reported in order to test the approach in different operative fields. Keywords: Human Computer Interaction, Virtual Reality, Metrics, Design Intent

1 INTRODUCTION One of the main engineering design tasks in large industries, such as automobile, aerospace, and oil & gas, is the use of integrated information systems to manage product development. Computing systems aim at offering resources for the 3D visualization of design models with enough realism for supporting Design Review (DR), styling evaluation, structural analysis, manufacturability, and training.

Virtual Reality (VR) represents technologies in order to achieve the above-mentioned goals [1], [2], [3] improving virtual prototypes interaction and manipulation. VR systems efficiency depends on immersion and interactivity capabilities, which requires a real time response to user demands. Thanks to the emerging low-cost technologies, VR systems are becoming economically available and technically sound for SMEs. While Large Enterprises (LEs) have human and financial resources to choose the more suitable VR technology on the basis of their specific process needs, SMEs differ from LEs due to organization, human skill and processes flexibility. Hence, SMEs have to be properly supported in order to choose it. VR performance measurement in terms of usability, scalability and design process lead-time, is the one feasible way to support the system evaluation. In parallel, VR application context is the second important factor for analyzing system adoption. Circumscribing the attention to styling product design, VR influences tasks performance as it affects design intent communication and teamwork participants interactions. The analysis of styling products design reveals [4] that the most time consuming activities are both styling and technical Design Reviews. DRs aim at achieving a compromise between form and function while assessing technical feasibility. The growing complexity of products requires an increasing involvement of a variety of viewpoints and disciplines: marketing staff, stylists, manufacturing engineers, etc. Communication problems emerge. The result is a conflict that generates a high number of design iterations and aesthetic errors, mainly due to the difficulty of effectively transmitting the designer intent to


the other process actors. The design intent can be defined as the set of product design values representing the stylist’s creative ideas. The functional and aesthetic product features are the concretization of the design intent. It should remain unchanged throughout the whole product development process. DRs are generally supported by physical prototypes as representational means for knowledge sharing, users interaction and product design evaluation. It has been observed that physical prototyping not only stretches time to market and design costs but also does not effectively reduce design iterations. Physical prototypes implicitly contain non-structured design information. Data retrieving in real time is not a trivial task and process efficiency decreases. Representing new modes of human-product interaction, Virtual Reality technology can support DR activities only if it enables the explicit representation of design intent on virtual prototypes and simultaneously allows non-expert users to explore prototypes surfaces, manipulate them simulating assembly/disassembly, evaluate dimensions, ergonomics, aesthetics, etc. Although there has been a substantial amount of research in the general area of Virtual Reality, in the context of styling DRs involving SMEs, additional research is required. VR customization in accord with SMEs needs and aesthetic values communication by virtual prototypes in DRs are important research issues for VR improvement. The remainder of this paper, after an overview of research background, points out the developed approach. A model of interaction between all actors involved in the virtual experience is the starting point in order to define a new simple to use evaluation technique for VR performance measurement in DRs. By adopting methods and criteria of Human Computer Interaction (HCI), the process of “perceiving information” is analyzed. Protocol analysis is then used to monitor DR activities. Two industrial design case studies are presented to test the approach and, finally, a discussion of results is reported.

2 VR AND DESIGN REVIEW PROCESS VR technologies are largely used for supporting the DR process in styling design. The ideal VR environment for preserving the design intent should provide the following main functionalities: • the assessment, in real time, of full-scale realistic virtual

prototypes in different product scenes; • the integration of different viewpoints in the product

common space; • the visualization and manipulation of 3D models from

geographically separated sites; • the creation, visualization and storing of enriched virtual

prototypes that contain design information from multiple sources by recording documents, images, 3D models;

• the evaluation, in real time, of different functions, colors, materials, in order to explore different design alternatives;

• the effective implementation of full immersivity in combination with an high sense of presence;

Anyway, current VR systems lack some of them, such as the integration of multiple points of views in a common space, the management of a multi-site collaborative environment and the easy design history retrieving. In order to satisfy the desired requirements several DRs tools based on virtual environments have been studied. A complete and interesting research work [5] studied the ability of advanced computer-based technologies and virtual and augmented prototypes to provide object-presence during design reviews activities in order to improve participation, time to market and process efficiency. The modalities and the related experimental tools for simultaneously working on a virtual model have been presented in [6] and [7]. Fu and East [6] introduced a virtual DR system that allowed a group of reviewers to simultaneously work on a 3D representation of an incomplete building model to perform their DRs process. While Knopfle and Voss [7] describe a VR user interface for applications in the area of digital DR for automotive applications. They report the functional requirements for allowing navigation, manipulation, examination and documentation of flaws in the design of the models. According to the limitations of 3D static CAD models for DRs, different visualization and user interaction approaches are reported in literature. ENVIRON (ENvironment for VIRtual Objects Navigation) is the robust application presented in [8] to control projects in VR environment starting from 3D CAD models. Burtnik et al. [9] describe an interactive system for DR, built to exploit the visual perception cue of motion parallax, in order to enhance shape perception and aesthetic evaluation. They replace static views of traditional CAD application with moving "shots" intersperse. By cinematic visual transitions users can access shots by picking object features on the 3D model, which invokes a spatial search over all shots, selects the ones deemed relevant to the feature, and plays back those shots. Bochenek et al. [10] tested different VR-based technologies for product conceptual DR in order to determine if VR contributes to cross-functional team collaboration. Research results show that while stereoscopic imaging facilitates the detection of design errors, no improvements are achieved in terms of supporting problem-solving activities. Furthermore, their experimental test highlighted that the sense of presence plays an important role in improving DRs practice. A more complete system is described in [11] (Boom Chameleon), an input/output device consisting of a flat-panel display mounted on a tracked mechanical boom. The display acts as a physical window into 3D virtual environments, through which a one-to-one mapping between real and virtual space is preserved. The Boom Chameleon is further augmented with a touch-screen and a microphone/speaker combination. A 3D annotation application allows simultaneously capturing viewpoint, voice and gesturing information. If Boom Chameleon annotation facilities have the potential to be an effective 3D DR system, it is limited in terms of visualization, collaboration and multi-sites management. The distributed teams for VR-based DRs processes are studied in [12]. Authors describe the integration of several components to enable distributed virtual DR in mixed multi-


party, heterogeneous multi-site 2D and immersive 3D environments. The system provides higher layers of support for collaboration including avatars, high fidelity audio, and shared artifact manipulation. The system functions across several interface environments ranging from CAVEs to Walls to desktop workstations. All these systems represent good examples of VR technologies usable in DRs activities, but they are far from the optimal system. On the one hand, the user-model interaction cannot be simulated as in physical prototype based situations; on the other hand, the advantages deriving from management of digital data are not completely achieved. Finally, a further difficulty in order to perform DR with virtual prototypes consists in correlating design process activities with technical performance of different VR technologies. Their use in DR activities can be successful if their implementation is supported by a structured method to select and design the proper VR combination according to the company’s requirements. It is also important to quantify advantages and limitations of implemented system in order both to verify the correspondence between company’s expectations and achieved performance and to plan possible upgrades for DRs improvements.

3 THE VR TECHNOLOGY EVALUATION APPROACH

The proposed approach starts from the schematisation of DR activities. The analysis of DR process, both aesthetic and technical, highlights the following general model: • Presentation: definition of the design tasks; • Interrogation: definition of the design intent and

description of the possible design solutions from each working session stakeholder;

• Discussion: evaluation of the individual solutions and of the different interpretations of the product design;

• Evaluation: verification of the coherence between the proposed solutions and the specified tasks definition of a whole solution.

All these activities involve users (team members), equipments and the external world. They are the main elements of Human-Computer Interaction (HCI) scenario. Hence, in order to measure VR technologies performance in DR, methods and tools of the HCI can be adopted. The user-system interaction mechanisms and the interface usability evaluation techniques are the basis of the approach. The definition of proper analysis techniques guarantees not only the observation of the interaction mechanisms between user and VR technology, but also the verification of its reliability with regard to the specific task that consists in the evaluation of VR systems in order to facilitate design intent communication and avoid subjective interpretations. The design intent communication is analyzed thanks to the definition of specific metrics used to verify the system performance. The metrics-based approach consists in the following steps: • to define the evaluation parameters (metrics); • to choose the evaluation techniques to assess the defined

parameters;

• to analyse the operative questions that must be faced for the experimental test;

• to correct the evaluation taking into account the application context;

• to evaluate the performed test, collecting data and interpreting the results.

3.1 The definition of the evaluation parameters The definition of “what to assess” is not a trivial task. A wrong identification of metrics may invalidate the analysis results. Metrics are quantitative estimates of the VR technology performance in order to verify the DR process results in terms of efficiency and preservation of design intent. It can be stated that in order to evaluate VR technologies performance it is necessary to identify usability and presence metrics. Usability refers to the effectiveness, efficiency and satisfaction through which users can achieve predetermined tasks in a specific context. The usability of a human computer interface is determined by its ability to be adopted by given users to perform particular activities to achieve a task within specified environmental scenarios [13]. Presence is defined as subjective experience of being in one place or environment, even when one is physically situated in another [14]. Presence forms an important subjective measure of a user’s virtual experience. It is assumed that the more natural the display feels, the greater its usefulness. This naturalness may better enable a user to utilize ‘real-world’ skills in a virtual environment. It is the perceptual illusion of non-mediation, specifically related to VR presence can be considered as a perceptual flow requiring directed attention. Presence is based on the interaction of sensory stimulation, environmental factors, and internal tendencies. If one feels present in a virtual environment, then they are thought to perceive themselves as being in the computer-generated environment rather than in their actual physical location. The conventional definition of presence suggests that non-immersive displays are inadequate, even for tasks that do not require the user to be surrounded and isolated. In this case a new measure called object-presence [15] is suggested to detect the extent to which information presented with a non-immersive display seems natural to the user. This concept replaces the feeling of being surrounded by an environment with the sense of being co-located with a collection of objects. This sense of object-presence is an important element for coincident haptic and visual displays that use a physical model as a three dimensional screen for projected visual information. Lombard and Ditton [16] report a complete review of relevant literature for conceptualisation of presence. As virtual prototyping goal is to provide a mediated experience in order to create for the users a sense of presence, meaningful metrics should related with the conceptualization of presence as perceptual realism proposed by Lombard and Ditton. Presence metrics successively used are designed according to their definition of presence as the measure of the degree to which a medium can produce seemingly accurate representations of objects.


Furthermore, other factors are representative of the interaction between user, VR equipment, external world and design tasks: immersion, navigation and interaction. Interaction is the ability of a specific equipment to create a mutual or reciprocal action of transmitting and exchanging information between two entities. The interaction establishes relations between the human senses and the action tasks. Immersion is the user’s emotive reaction to the virtual environment. It stimulates the multi-sensorial involvement of the user in the virtual space [17]. The immersion is strictly correlated to presence, but it specifies the relation between the external world and the user human senses. Navigation represents the process of managing users’ movements in the virtual scene in order to achieve the specified tasks. Navigation metrics are common both to presence and usability, but they are also meaningful for the immersion and interaction in the virtual environment. The diagram below (figure 1) expresses the subject of interaction process and the main classes of metrics.

Figure 1. A schema of the main actors of the human-computer interaction, the evaluation metrics fields and

their relationships

The metrics are classified in two typologies: • metrics exclusively related to technology, product and

process • metrics that contain also the design intent meanings. The design intent communication can be viewed as the identification of aesthetic and functional features considered fundamental for the success of a product. It can be approached from two complementary perspectives: the product viewpoint and the process viewpoint. In the first case it is necessary to fix the essential product characteristics to represent the stylist’s intentions. The process has to allow the transmission and the elaboration of the product model in a robust way for preserving the characteristics. The design intent communication from the process viewpoint can be supported through proper “actions”. A list of “actions” is reported in section 2 and it can be resumed as follows: • to examine full-scale prototypes in different product

scenes,

• to integrate different viewpoints in the prototype common space,

• to manipulate prototypes collaboratively from geographically separated sites,

• to manage the design process history, • to evaluate alternative aesthetic and functional features, From the product perspective the design intent can be translated in a set of “concrete” and “evocative” features. Concrete features, classifiable in geometric and non-geometric ones, are measurable parameters, while evocative features are related to the stylist creative process. Table 1 reports the product features classification oriented to design intent representation.

Table 1: Product features classification oriented to design intent representation

Concrete Evocative

Geometric Non-geometric

2D styling curves 3D styling curves

3D shapes Size

Dimensional proportions External phys. interfaces

…..

Material Finishing

Colour Movements

Weight Flexibility/rigidity

Transparency ……

Images Shapes from

analogy Shapes from

mutation ………

Table 2 shows the identified usability and presence metrics. Each metric contains also immersion, interaction and navigation metrics. Some of them can be correlated to the above-mentioned “actions” and design intent oriented product features. In bold are highlighted the design intent oriented metrics.

Table 2. Identified metrics for presence and usability

Usability Metrics Presence Metrics

Input devices customization Degree of movements

Control and navigation support Variety of input information

Intuitiveness

Variety of output information Events prediction

Software reuse and integration Natural engagement

Imaging resolution Environment image complexity

Haptic feedback Multi-sensorial modalities involvement

Sound feedback Interactivity

Motion feedback Unconsciousness of the VR devices

Background noise Discriminating attention Concentration Realism of the scene and of the

interpreted information Ease of learning Meaningfulness of the experience Achievement of design tasks Bewilderment Tasks result quality


Number of detected errors

Number of remained errors

Mental load

Centrality of the user

User motivation

Barrier free

Latency

Easy-to use

For instance if we consider the “concrete” feature finishing, its evaluation requires high haptic feedback in terms of usability and high multi-sensorial interaction in terms of presence. Furthermore if we consider an “evocative” feature as shapes from analogy, its evaluation requires variety on input information (2D drawings, movies, photos etc.), sound feedback and motion feedback. Similar reasoning can be applied for “actions”. For instance if we want to examine full-scale prototypes in different product scenes it is necessary to have high values in terms of natural engagement and realism of the scene and of the interpreted information.

3.2 The choice of the evaluation technique Once the metrics are defined, it is necessary to compute each of them by analysing the VR-based DR process. The metrics evaluation can be achieved by applying proper investigation techniques. A possible classification of the investigation techniques (figure 2) is based on the interrelation between the user involvement, the evaluation context and the type of requested results [18].

Figure 2. Different investigation techniques used to measure the performance of VR technologies

The Post-hoc Questionnaire is a largely diffused technique adopted to retrieve information from the direct contact with the user after his/her VR technology employment. In order to fulfil the questionnaire the observer must previously define the metrics, the units for their assessment and a scale of values to

correlate user’s preferences and metrics values. It is an intrusive technique. The Interview technique is similar to post-hoc questionnaire but differs from the first for the high level of detail that can be reached with expert users and the subjective reaction of the respondents. The Heuristics Evaluation is a technique that does not imply the involvement of final user. VR experts define a list of guidelines in order to evaluate the previously identified metrics by observing users involved in the virtual environment. The result of the heuristics technique is quite objective and detailed but requires a high degree of observers’ expertise both in cognitive science and in virtual reality devices. It is worth to notice that the other ones are more focused on statistic aspects and the evaluation context is far from VR performance measurement, therefore these three techniques are adopted.

3.3 The identification of the experimental test environment The next step is the identification of the experimental test environment where evaluation takes place, and the sample of final users submitted to the post-hoc questionnaire, interviews and heuristics evaluation. As we are exploring the performance of VR technologies in DR activities, the environment consists of the virtual room equipped with the chosen technologies combination, while the final users are the actors that are involved in DR working sessions.

3.4 The correction of the metrics values The values of metrics have to be estimated in order to successively compare the results of the different analysis. The Post-Hoc Questionnaire Witmer provides a useful guideline in order to establish which questions must be submitted to the users, which information is necessary for classifying the different sample users typology and the degree of VR technology expertise and how to assign the score to the users answers in order to evaluate the usability and presence provided by the VR system. The Likert scale [19] is chosen in order to assign different values to users answers. It allows the evaluation of users stimuli with respect to some designated attributes that reflect usability and presence metrics. The figure below shows the questionnaire format (figure 3). Questions are strictly connected to the predefined metrics; they are related to the user satisfaction, the working session efficiency and effectiveness, the involvement and immersion in product scene, the achieved interaction with virtual models, etc.


Figure 3: Example of one of the used post-hoc

questionnaire Each metrics value is determined by assigning a value from 1 to 5. It has been chosen to propose closed answers to compare the results of different experimental tests. The Interview technique has been adopted to deepen some aspects that may emerge during the filling of questionnaire. The interview aims at specifying the closed answers once a meaningful activity has been carried out. The Likert scale is also used to assign a value from 1 to 5 to the different judgments (1 for negative and 5 for positive, intermediate values correspond to different degrees of satisfaction) Thanks to the Heuristics Evaluation it is possible to avoid users subjectivity. It can be observed that the user is influenced by the company’s expectations and his/her answers are filtered by a general positive attitude to every proposed technology. The heuristics evaluation consists of three main steps: • Briefing session, in which VR experts identify the

heuristics (usability and presence metrics); • Evaluation session where each expert individually assesses

the VR performance and elaborates a judgement; • Debriefing session where the experts meet together to

elaborate only one judgement by assigning a value from 1 to 5 to the metrics according to the Likert scale.

In all cases the values have to be properly weighted in order to consider the importance of design intent oriented metrics and the application context where the VR technology is used. In the first case, if we call vmi the value assigned to the ith metrics during the analysis, the correction of value is

determined by multiplying it with the design intent corrective factor Cdi:

Cdi =(1+wdia+wdic+wdie) (1)

Where wdia is the weight of the “actions” related to design intent for the specific metric. Its value varies from 0 to 0,2. If the metric is completely independent from “actions”, wdia assumes the value zero; otherwise it assumes a value in the mentioned interval. The analyst decides the right value depending on how many “actions” are related to the metric. Similarly wdic and wdie vary from 0 to 0,2 and they are linked, respectively, to “concrete” design intent product features and “evocative” design intent product features. Hence the final corrected value for design intent constraints vmid is:

vmid = vmi x Cdi (2)

The application context where the VR technology will be used implies further corrections. We assume the following parameters in order to characterise the context: the size of company, the developed product typology, the skill of operators and, finally the number of operators involved in product development. These parameters are related to a scale of the importance (low, medium, high) that measures their impact factor on the specific metric; the weight of the four corrective values (wci) varies from 0 to 0,2. Hence, the context corrective factor Cc can be calculated as follows:

Cc =(1+wc1+wc2+wc3+wc4) (3)

The corrected value due to context constraints (vmic) and assigned to the ith metrics vmi is:

vmic = vmi x Cci (4)

Where: Cci =(1+wc1i+wc2i+wc3i+wc4i) (5)

When the analysis of a specific context is performed a matrix, as shown in table 3, has to be fulfilled adopting the right weights. Table 3. Weights in order to correct metrics in according

to the context

Size of company Product typology

Skill of operators

No. of operators

Input devices customization wc11 wc21 wc31 wc41 Variety of input information wc12 wc22 wc32 wc42 Variety of output information wc13 wc23 wc33 wc43 Software reuse and integration

wc14 wc24 wc34 wc44

…………….. ………………. …………….. ……….. ………… Nth metric wc1n wc2n wc3n wc4n


For instance, we consider the parameter product typology and the sound feedback. If company produces chairs the importance is low and the wc2 value is 0, on the other hand if company produces washing machines the importance is high and the wc2 value is 0,2. At the end the final corrected value (vmicorr) of ith metric takes into account both the mentioned constraints:

vmicorr = (vmid + vmic)/2 (6)

The scale of metrics and weights values have been defined thanks to the analysis of specific questionnaires filled by the chief of technical departments of 5 five different Italian companies (Teuco-Guzzini S.p.A., Indesit Company, S.p.A., Biesse S.p.A., Poltrona Frau S.p.A., Fiat S.p.A.). Such questionnaires highlight the importance of the different factors for each company viewpoint.

3.5 The experimental results evaluation The post-hoc questionnaire and the interview techniques experimental data is immediately stored by calculating the total score either of users answers or of users judgments. In the case of heuristics evaluation it is necessary to define a method for the observation of users behaviours in virtual environment. Two different analysis methods to allow the VR experts to observe the DR sessions and to elaborate a judgment have been identified: the diary study [20], and the Video Interaction Analysis (VIA) described by Jordan [21]. The diary study is adopted to elicit information about design cycle activities in terms of both qualitative and quantitative data (date, time, documents format, VP users, etc.). In order not to be intrusive by interviewing the VP system’s users, the diary study provides a format that can be immediately filled in real time by researchers observing the activities or by users themselves. VIA is chosen in order to capture the user interaction, the experiences in the virtual environments, and the users interface usability. A video camera positioned in the virtual room can capture the dynamics of work session and the complexity of interactions, can record the human activities such as talks, nonverbal interactions, the use of artifacts and technologies and the immediate workplace context available for repeated viewing and analysis. Therefore, the real usability, ease of use and users satisfaction can be assessed. In tables 4, 5 and 6 are reported the used diary study operative information.

Table 4. DR information in the diary study

TYPE OF INFORMATION RECORDED INFORMATION

Number of session Integer number Date of session Free text Project name Free text VR users and company’s departments involved Free text

Type of performed activity Free text Means of representation adopted to communicate the design intent Free text

Virtual prototype formats Free text

Format files checkbox Files names checkbox Number of physical prototypes realized Integer number

Number of virtual prototypes manipulations (shape modifications, aesthetic properties change, alternative product scenes analysed, etc.)

Integer number

Time of the working session number

Table 5. Analysis of the main chunked stages of the DR session

Number of session Integer number

Total time for the session hours

Chunked stages Measure units

Presentation time percentage (%) Discussion time percentage (%) Requests or queries time percentage (%) Solution or performed actions time percentage (%)

Table 6. Synthesized data collected via diary study

method

Project name Free text

Data collected Measure units

Number of design reviews number Number of aesthetic design reviews number Number of engineering design reviews number Number of physical prototypes number Number of company’s departments involved number Total time of design reviews in the virtual environments hours

Number of different format files employed during design reviews

number

Number of users disappointed requests number Number of hand-made sketches used to communication the design intent

number

4 THE APPROACH EXPERIMENTATION The proposed method has been applied on two different industrial test cases. They do not represent a statistical sample but give a preliminary verification of method effectiveness. The analyzed companies realize products that are a compendium of aesthetic and technological features. They rely on the emotional values of brand identity and on the sensorial properties of product shapes to stimulate consumers’ response. Their goal is the reduction of physical prototypes number needed for DR activities and design iterations, in order to shorten time to market and reduce costs. Both companies retain strategic the design intent communication and preservation. On the other hand, the application context is different both in terms of company size and product types, as described below.

4.1 Wellness product design test case The first case study regards Teuco Guzzini s.p.a., an Italian medium sized company that designs and produces wellness products, such as bathtubs, mini-pools, whirlpools, etc. Its


products integrate complex free-form shapes with advanced therapeutic and comfort functions. CAD (ProEngineer), CAM (Tebis) and CAE (Plastic Insight by MoldFlow) are the software packages used in the technical department that is composed by 14 design engineers, 6 process engineers and 3 project managers. Famous external stylists, such as Ron Arad, complete the design team in order to realize the product concept. The company realizes about 3 new product lines and 15 product customizations in a year. Time to market is unsatisfactory: 10 months for products of average complexity. The requirements of company management, in this context, can be resumed as follows: the improvement of communication between stylists and internal product design team, the minimization of physical prototypes, the visualization and manipulation of full scale virtual prototypes and their navigation into different realistic scenes, the interaction with the virtual models in order to detect design errors, validate assembly/disassembly, compare different performed simulations. VR technology has been considered a possible valid solution. The choice of the VR system that better answers the company expectation has been determined by using the benchmarking method that authors presented in a previous work [4]. The budget limitations pushed towards a technically valid solution but not the optimal one. The implemented system (figure 4) consists of: • a large volume projection-based display consisting of a

DLP projector by ProjectionDesign with high resolution, 1400x1050, and high luminescence that rear projected on a diffuser plate (3x2,5 m);

• the display is made of a special Fresnel lens in order to intensify the main light source. The special form of prisms allows the display to be more efficient at collecting and directing the light rays in order to improve the image projected quality;

• a stereo sound system is chosen for the sound feedback; • Flamingo by Rhinoceros is the current software toolkit; the

final system has not yet been defined (Showcase by Autodesk is under investigation);

• the tracking devices have not been integrated due to the fact that the company staff is waiting for final evaluation results of VR benefits before upgrading the implemented system in according to the method’s suggestions.

Figure 4. The VR system implemented in Teuco Guzzini

The company design team uses the VR system mainly during DR activities. Full-scale virtual prototypes are generated as photorealistic images with and without different product scenes to evaluate the aesthetic impact of design solution. Simulations of manufacturing processes are analyzed interfacing CAE system with the visualization tool. In the virtual environment, company’s staff evaluates different product solutions by modifying different taps, colors and materials on the same 3D model, sketching and annotating design ideas on the photorealistic images, and, finally, comparing different product alternatives developed in different working sessions. The virtual environment is also used to display predefined products presentations. The VR performance assessment has been conducted during the development of two different products (a bathtub and a shower stall). In the following tables 7 and 8 the key elements of the studied process are resumed Table 7. Key elements of the analysis for DR of shower

stall

PRODUCT

Shower stall

DURATION OF STUDY

3 months

NUMBER OF DESIGN REVIEWS VR-BASED

5

NUMBER OF PARTICIPANTS From 4 to 7 (the project manager, the chief designer and the manufacturing process responsible were ever present; the stylist ws present at three reviews)

MEAN DURATION OF DR

4 hours

ANALYSIS TECHNIQUES post-hoc questionnaire, interview and heuristic evaluation (diary study)

Table 8. Key elements of the analysis for DR of bathtub


PRODUCT

Bathub

DURATION OF STUDY

4 months


6

NUMBER OF PARTICIPANTS From 5 to 7 (the project manager, the chief designer and the manufacturing process responsible were ever present; the stylist ws present at three reviews)

MEAN DURATION OF DR

4 hours

ANALYSIS TECHNIQUES post-hoc questionnaire, interview and heuristic evaluation (VIA)

In the same period the virtual reality room has been monitored through video cameras in order to register all generic activities outside DR. We neglect to describe all analysis results. The heuristics evaluation results are particularly meaningful (table 9, see annex 1); in the same table the corrective values fixed for this specific analysis are reported. The results show how the VR system usability has high performance in DRs while the presence is inadequately provided. The total score is 4,05. The relative value is thus 0,69 that is obtained by the ratio between the total score and the highest admissible score (5,9) 4,05/5,9. It is very similar to that calculated [22] by the benchmarking method [4] (0,7) that was used for determining the most suitable VR technology. In [22] was established that the ideal value should be 0,81 (obtained by balance between technical requirements and financial budget) and it is coherent with the present results that highlight the low performance in some aspects. In fact, the difference between the values of the measured metrics and the ideal values (table 9), which can be calculated by multiplying the maximum value (5) with the related weights, highlights that the current system should be modified by choosing solutions for improving the haptic feedback, the control and navigation support, the variety of input information and the multisensorial modalities involvement. Making interviews and submitting questionnaires to the users also obtain similar scores.

4.2 Household appliances test case The second case study involved Indesit Company, an Italian medium-large sized company leader in household appliances design and production. The design context is similar to the first case study. A larger time to market, a more complex organization that requires many DRs and a continuous product innovation are the main differences. VR technology has been adopted in order to approach two different design situations: • the aesthetic DRs when designers and product committee

meet together to identify the best design solution, and they evaluate different product alternatives. Full scale virtual prototypes are visualized in different environments in accord with the type of the company brand, of appliance context (kitchens and bathroom) and of the specific buyer.

• during technical meetings for visualization of technological solutions, verification of whole dimensions, simulation of functional parts movements, assembly/disassembly operations and interference inspection.

The virtual environment of Indesit Company (figure 5) consists of: • a large volume display (4x3 m), retro-projected; • two advanced BARCO Galaxy 6 Classic+ projectors that

can provide stereoscopic imaging; • a stereo sound system composed by two loudspeakers FBT

StudioPRO9; • a commercial software toolkit for the manipulation and

navigation of the virtual environment, in particular Alias Studio Tool and Showcase by Autodesk for the creation of photorealistic images and the interaction with aesthetic virtual prototypes;

• VisMock-Up of UGS for engineering design tasks.

Figure 5. The VR system implemented in Indesit

Company The evaluation technique has been applied to measure the performance of the chosen VR technologies. The research objective is achieved by post-hoc questionnaires, interviews and heuristics evaluation techniques observing DRs via Diary Study and VIA methods. The analysis context is resumed in table 11.

Table 11: Key elements of the analysis for DR of washing machine

PRODUCT

Washing machine

DURATION OF STUDY

6 months


9

NUMBER OF PARTICIPANTS From 3 to 12 (the chief designer wa ever present; the stylist was present at four reviews)

MEAN DURATION OF DR 6 hours

ANALYSIS TECHNIQUES post-hoc questionnaire, interview and heuristic evaluation (VIA)


Post-hoc questionnaires have been submitted to all users of virtual environment while only project leaders have been interviewed for their high practice and experience of design problem-solving. VR experts coming from the University have performed heuristics evaluations. Diary Study and VIA techniques have been adopted in order to facilitate the interpretation of users interaction with virtual prototypes in different design situations and the evaluation of technologies performance. By interpreting the questionnaire score for the closed questions given to the users, it is possible to deduce that the VR technology has higher performance for aesthetic DRs than for engineering DRs (table 12).

Table 12. The post-hoc questionnaire resulting score highlights the best performance of the VR system for

aesthetic working session

Post-hoc questionnaire resulting score from multiple analysed sessions

Type of working session

Usability metrics score

(corrected score)

(weighted average)

Presence metrics

(weighted average)

TOTAL

Aesthetic Design Reviews (ADR)

3,8 (4,35) 3,9 (4,4) 7,7 (8,75)

Engineering Design Review (EDR)

3,2 (3,65) 3,5 (3,95) 6,7 (7,6)

Design Review activity ((ADR+EDR)/2)

3,5 (4) 3,7 (4,175)

Interview and heuristics evaluation techniques allow giving an interpretation to this difference. In particular the second one allows specifying the weight of each metric on the whole score. This is used for determining possible VR system upgrades. The resulting value of the heuristics evaluation corresponds to that obtained both in questionnaire analysis (table 13, see annex 2). In this case the correction weights are not detailed but they have been estimated as in the previous case study; the different applicative context generated different correction values. For the usability metrics, the main critical VR system features are related to the haptic feedback, the variety of input information, the variety of output information and the ease of use metrics. While the control and navigation support, the natural engagement and the interactivity metrics are critical factors related to presence metrics. This is due both to the difficulty to parametrically modify the shape in accordance with the users’ requests during the working sessions and to navigate the scene in a natural manner. Users have to continuously ask to the working session manager to execute specified operations to better interpret the virtual prototype. On the other side, presence metrics values, such as intuitiveness, multi sensorial modalities involvement and

natural engagement, are higher than those measured in the first test case. This is mainly due to the provided stereoscopic image viewing and the lower sense of scale that improve visual perception. The results are interesting if compared with the first case study, in fact the analyzed systems are similar in terms of HW and SW solutions, and, even if the applicative context is different, the critical technological factors for a better design intent communication are common.

4.3 Results discussion The empirical studies examined the applicability of metrics-based approach in practical industrial context. The results demonstrate how the method can support evaluation of VR performance for DRs. However, many issues are worth exploring further, in particular three aspects have to be treated in depth: the DR process schematization, the correlation between metrics and design intent and, finally, the use of protocol analysis techniques. The DR process is a form of reasoning that interests different actors in a common space. DR involves both functional and aesthetical reasoning. The four steps (presentation, interrogation, discussion and evaluation) adopted for DR schematization have been effectively carried out during a DR session. Virtual prototypes allow the realistic visualization and simulation of product behavior, but 2D sketches still remain irreplaceable in order to represent ideas and concepts. In both case studies, the discussion of emerging new ideas have been supported by sketches, but it was evident the difficulty to merge the digital model with such schematic signs. This caused a waste of time for clarification of intentions. The studies of DRs revealed the need for computational tools to support sketching directly on 3D model. Hence, in several cases the interpretative stage has been critical and this has influenced the evaluation phase. The difficult representation of emerging design alternatives generated poor evaluations. In two cases it has been necessary to stop the DR process in order to make 3D models of proposed idea. The stylist draws symbols and configurations when thinking about different functional design concerns. He manipulates the ideally visualized representations to evaluate the consequences. These manipulations are simple transformations but the process becomes complex and difficult to understand. In conclusion, the interactivity degree of used systems and the manipulation mechanism of virtual models are still far from the ideal tool for supporting DR. Anyway the adopted solutions, into the specific industrial context, provide a first step towards this system. The second key point of the approach is the metrics definition. The proposed classification and the differentiation related to the scope of evaluation seem to be robust. Usability and presence metrics provided interesting outcomes in both case studies. Anyway, the correlation of them with design intent features needs further investigation in order to achieve a complete qualification. A larger amount of experimental data is necessary to validate relations and corrective weights. Metrics typologies should be extended when new systems as Augmented Reality technologies will be tested.


Bochenek et al. [13] provided a similar assessment of prototyping means during design reviews, using metrics as the number of design errors found, the time to track errors down, and the time to discover solutions. They specifically analyze the VR displays performance and highlight that the monoscopic monitor-based systems perform best if compared with head-mounted displays and stereoscopic screen. Two of them are similar to those implemented in the illustrated case studies. In these years the imaging technologies have gained a meaningful improvement in terms of performance. Different results derive both from the technological improvements as the sense of immersivity is concretized in a more realistic way and from the metrics used to evaluate presence, considered in our work as the ability of a medium to immerse users in the virtual environment. Thanks to the comparison of the two different technologies on the same kind of process the achieved result can be considered valid. The used protocol analysis techniques showed a good usability. The actors of DRs easily accepted them since their simplicity and the low intrusivity (interviews and questionnaires need few minutes to be accomplished). In order to avoid the sample bias of DRs participants, in both case studies the chief of technical office together with researchers have pre-selected the design team. In this way it has been involved people with different levels of skill, personality and intrinsic curiosity. However, expert designers were ever present during DRs sessions in order to monitor activities and readdress discussion. In future experimentation will be useful to classify the actors in more detail before the test. The designers’ profile will be a further variable of the analysis.

5 CONCLUSIONS AND FUTURE WORK Companies, especially SMEs, require dedicated techniques and tools to objectively evaluate performance of implemented Virtual Reality technologies. Since they can be widely used in DR activities, this has been considered as the most interesting process to be analyzed. The communication of design intent has been determined as the most important parameter that has to be monitored in DR. This paper described a new metrics-based technique of evaluation, in order to face this problem. Two different classes of metrics have been studied for evaluating VR technology performance. In both cases they have been qualified to highlight their correlation with design intent. During DRs metrics measurement has been performed by using protocol analysis techniques (interviews, questionnaires and heuristics evaluations). The metrics, related to the concepts of usability and presence, allowed establishing the degree of VR technology performance and the main criticalities in order to propose a possible system improvement. The approach application on two different case studies showed a preliminary experimentation of the proposed technique. Although the experimental data is numerically limited, results seem promising. However, the robustness of the approach should be more extensively investigated. Currently the research activity is focused on new data collection by monitoring the activities within the two VR rooms. Thanks to the new data will be possible to optimize the metrics and the correlation between them and the design intent. The future work will be also

focused on method validation through a more extended number of industrial cases (different VR systems, use of Augmented Reality systems, different company typologies, etc.).

6 REFERENCES [1] Jayaram, S., Vance, J., Gadh, R., Jayaram, U.,

Srinivasan, H., 2001 “Assessment of VR technology and its applications to engineering problems”, Journal of Computing and Information Science in Engineering, Vol. 1, No. 1, pp.: 72-83

[2] Zimmermann, P., 2008 “Virtual Reality Aided Design: a survey of the use of VR in automotive industry”, Product Engineering: Tools and Methods based on Virtual Reality, Talaba D. and Amaditis, A. (eds.), Springer Verlag, pp. 277-296

[3] Rooks, B., 1999 “The reality of Virtual Reality”, Assembly Automation, Vol.19, No. 3, pp. 203-208

[4] Mengoni, M., Germani, M., 2006 “Integration of Virtual Reality Technologies in Industrial Design Processes: a Structured Approach”, Research in Interactive Design, Proceedings of Virtual Concept 2006, Vol. 2, Cancun 29 November - 2 December, Springer Press.

[5] Verlinden, J., Horvat, I., 2007, “A critical systems position on augmented prototyping systems for industrial design”, Proceedings of the ASME 2007 International Design Engineering Technical Conferences & Computer Information in Engineering Conference, September 4-7, Las Vegas, Nevada, USA.

[6] Fu, M. C., East, E. W., 1999 “Virtual design review”, Computer Aided Civil and Infrastructure Engineering, Vol. 14, No. 1, pp. 25–35

[7] Knopfle, C., Voss, G., 2000 “An intuitive VR user interface for Design Review”, Proceedings of the working conference on Advanced Visual Interfaces AVI 2000, Palermo (Italy), pp. 98-101.

[8] Raposo, A., Corseuil, E., Wagner, G., Dos Santos, I., Gattas, M., 2006 “Towards the use of CAD models in VR applications”, Proceedings of the 2006 ACM international conference on Virtual Reality continuum and its applications, Hong Kong, pp. 67-74.

[9] Burtnyk, N., Khan, A., Fitzmaurice, G., Kurtenbach, G., 2006 “ShowMotion: camera motion based 3D design review”, Proceedings of the 2006 symposium on Interactive 3D graphics and games, pp. 167-174.

[10] Bochenek, G.M., Ragusa, J.M., Malone, L., 2001 “Integrating virtual 3D display systems into product design reviews: some insights from empirical testing”, International Journal of Technology Management, Vol, 21, No. 3/4, 2001, pp. 340-352.

[11] Tsang, M., Fitzmaurice, G., Kurtenbach, G., Khan, A., Buxton, B., 2002 “Boom Chameleon: simultaneous capture of 3D viewpoint, voice and gesture annotations on a spatially-aware display”, Proceedings of the 15th


annual ACM symposium on User interface software and technology, Paris (France), pp. 111-120.

[12] Daily, M., Howard, M., Jerald, J., Craig, L., Martin, K., Mc Innes, D., Tinker, P., 2000 “Distributed design review in virtual environments”, Proceedings of the third international conference on Collaborative Virtual Environments, San Francisco (USA), pp. 57-63.

[13] Gabbard, J., Hix D., Swan E., 1999 “User-centered design and evaluation of virtual environments”, IEEE Computer Graphics and Application, Vol. 19, No. 6, pp. 78-91

[14] Witmer, B., Singer, M., 1998 “Measuring presence in virtual environments: A presence questionnaire”, Presence, Vol. 7, No.3, pp. 225-240.

[15] Stevens, B., Jerrams-Smith, J., Heathcote, D., Callear, D., 2002 “Putting the Virtual into Reality: assessing object-presence with projection-augmented models", Presence, Vol. 11, No. 1, pp. 79-92.

[16] Lombard, M., Ditton, T., 1997 “At the heart of it all: the concept of presence”, Journal of Computer- Mediated Communication, Vol.3, No. 2, online http://jcmc.indiana.edu/vol3/issue2/lombard.html

[17] Nosper, A., Behr, K-M., Hartmann, T., Vorderer, P., 2005 “Exploring the relationship between the usability and the sense of spatial presence perceived by the user”, Proceedings of Presence 2005, pp.24-33

[18] Sutcliffe, A., 2005 “Presence, memory and interaction in virtual environments”, International Journal of Human-Computer Studies, Vol. 62, No. 3, pp. 307–327.

[19] Likert, R., Hayes, S., 1957 “Some Applications of Behavioural Research”, Unesco, Paris.

[20] Dorst, K., Cross, N., 2001 “Creativity in the design process: co-evolution of problem-solution”, Design Studies, Vol. 22, No. 5, pp. 425-437.

[21] Jordan, B., Henderson, A., 1995 “Interaction analysis: foundations and practice”, The Journal of the Learning Science, Vol. 4(1), pp. 39-103.

[22] Mengoni, M., Bordegoni, M., Germani, M., “A Method for the Selection and Evaluation of VR Technologies Combination in Product Development Processes”, Proceedings of XVI ADM – XIX INGEGRAF International Conference, Perugia 6-8 June 2007


ANNEX 1

Table 9. Heuristics evaluation results for Teuco case study

VR PERFORMANCE METRICS Wdia Wdic Wdie

Size of

compa

ny

Product

typology

Skill of

operat

No. of

operat

RESULTING

(and corrected)

SCORE

(average) for

metrics

DIFFERENCE

WITH MAX

ADMISSIBLE

VALUE

RESULTIN

G SCORE

(average)

for metrics

class

FINAL

SCORE

Input devices customization 0 0,2 0,1 0,1 0 0,2 0,1 2 (2,7) 4,05 Variety of input information 0,2 0,2 0,2 0,1 0,2 0,2 0,1 2 (3,2) 4,8 Variety of output information 0,2 0,1 0 0,2 0,1 0,2 0,1 5 (7,25) 0 Software reuse and integration 0 0 0 0,2 0,1 0 0 2 (2,3) 3,45 Imaging resolution 0 0 0 0 0,1 0 0 5 (5,25) 0 Haptic feedback 0 0,2 0,1 0 0,2 0 0 0 6,25 Sound feedback 0,1 0 0,2 0 0,1 0 0 3,5 (4,2) 1,8 Motion feedback 0,2 0,1 0,1 0 0,1 0 0,1 2,5 (3,25) 3,25 Background noise 0 0 0 0 0 0 0 5 0 Concentration 0 0 0 0 0 0,2 0,1 5 (5,75) 0 Ease of learning 0 0 0 0 0 0,2 0 4,5 (4,95) 0,55 Achievement of design tasks 0,2 0,2 0 0,1 0,1 0 0 4 (5,2) 1,3 Tasks result quality 0 0 0 0,1 0,2 0 0 4 (4,6) 1,15 Detecting errors ability 0,2 0 0 0,2 0 0,1 0 4 (5) 1,25 Errors remained 0 0 0 0,2 0 0,1 0 2 (2,3) 3,45 Mental load 0 0 0 0 0 0,2 0,1 5 (5,75) 0 Centrality of the user 0,2 0 0 0 0,1 0,2 0,1 5 (6,5) 0 User motivation 0 0 0 0,1 0 0,2 0 4 (4,6) 1,15 Barrier free 0,2 0 0,1 0,2 0 0,1 0 5 (6,5) 0 Latency 0 0 0 0 0 0 0 4 1

Usa

bilit

y m

etric

s

Easy-to-use 0,1 0,1 0 0,2 0 0,2 0,1 4 (4,4) 1,1

3,7 (4,4)

Degree of movements 0,2 0,1 0 0 0 0 0 5 (5,75) 0 Control and navigation support 0,1 0,1 0,1 0 0,1 0,1 0,2 1 (1,35) 5,4 Intuitiveness 0 0 0 0 0 0,2 0 4 (4,4) 1,1 Events prediction 0 0 0 0 0 0 0 2 3 Natural engagement 0,2 0 0,1 0 0 0,2 0 3 (3,75) 2,5 Environment image complexity 0,1 0 0 0 0,2 0 0,1 5 (6) 0 Multisensorial modalities

involvement 0,1 0,1 0,2 0 0,2 0,1 0,1 2 (2,8) 4,2

Interactivity 0,1 0,1 0,1 0 0,1 0 0 2 (2,4) 3,6 Unconsciousness of the VR

devices 0 0 0 0 0,1 0,2 0 5 (5,75) 0

Discriminating attention 0 0 0 0 0 0 0 4 1 Realism of the scene and of the

interpreted information 0,1 0 0,1 0 0,1 0,2 0,1 2 (2,6) 3,9

Meaningfulness of the experience 0 0 0 0 0 0,1 0 3 (3,15) 2,1

Pres

ence

met

rics

Bewilderment 0 0 0 0 0 0 0 4 1

3,2 (3,7)

3,45

(4,05)


ANNEX 2

Table 13. The heuristics evaluation results for aesthetic and engineering DRs VR PERFORMANCE METRICS

RESULTING SCORE

(average) for metrics

RESULTING SCORE (and

corrected) (average) for metrics

class

RESULTING

SCORE

input devices customization 3 variety of input information 3 variety of output information 3 software reuse and integration 3.5 imaging resolution 5 haptic feedback 0 Sound feedback 3.5 motion feedback 3.5 background noise 3 concentration 4 ease of learning 4 achievement of design tasks 4 tasks result quality 3 Detecting errors ability 4 Errors remained 3,5 mental load 3.5 Centrality of the user 2 User motivation 4 barrier free 3.5 Latency 4 Easy-to-use 3

Usa

bilit

y m

etric

s

3,4 (3,9)

degree of movements 4 Control and navigation support 2 Intuitiveness 3 Events prediction 3.5 Natural engagement 2.8 Environment image complexity 4.5 multisensorial modalities involvement 4 Interactivity 3 unconsciousness of the VR devices 4 Discriminating attention 3.5 realism of the scene and of the

interpreted information 2

meaningfulness of the experience 4 bewilderment 4

Pres

ence

met

rics

3,6 (4,1)

3,5 (4)

Metrics-Based Approach for VR Technology Evaluation in Styling Product Design

Documents

Transcript of Metrics-Based Approach for VR Technology Evaluation in Styling Product Design