In times of shortened product life cycles and increased product complexity it has become necessary to define, design and test the features of a forthcoming product prior to creation of any physical prototypes. Especially, this is true for the development of electronics and telecommunication products. Virtual prototyping is a novel approach, which aims at provide a full-digital front-end for product development. Realistic and comprehensive simulation of various features of these products sets specific requirements for the used virtual prototyping system. We are building an experimental system to study and develop virtual prototyping technology applicable for the development in the area of telecommunications and consumer electronics. The presented virtual prototyping system will enable the integration of multiple discipline simulation models and provides realistic rendering of simulated product features. The system augments visual rendering of product models, that is based on three-dimensional stereoscopic imagery, by integrating it with auditory and haptic rendering.
Shortened product life cycles, increasing product complexity and tightening global competition, where innovation and dynamics are the key requirements for success, set new challenges for the development of new electronics and telecommunication products. It is necessary to be able to define, design and test all the features of a future product prior to buidling any physical prototypes. A promising approach to meet these requirements is virtual prototyping technology. The aim is to enable full-digital front-end for product development, which should increase innovation of product concepts, shorten product development cycles, reduce costs and improve the accuracy and quality of the development to meet the needs of the customer and market.
Next-generation electronics and telecommunication products represent technology-rich and highly complex products. Comprehensive, realistic and accurate simulation of their features sets specific and demanding requirements for the applied virtual prototyping system. For instance, the system should enable integration of simulation models of various engineering disciplines, and to provide adequate rendering capabilites to render the simulated features. Fulfilling these requirements is becoming more and more important, because right balancing of the product's form, user-interface and functions is an important competatitive tool in the tightening product markets. For instance, comfort and ease of use of increasingly complex products have become a top priority in consumer electronics markets.
In this paper we present a virtual prototyping system, which is targeted mainly to be applied to the development of small-sized, usually hand-held, electronics and telecommunication products. Products of interest include, for instance, cellular phones, mobile communicators, PDAs and GPS navigators. In order to study and develop virtual prototyping technology applicable to the mentioned products domain, we are building a research environment, that enables integration of various simulation models and provides efficient rendering techniques for simulated product features.
Virtual prototyping can be regarded as a digital design approach based on the simulatable computer models of a product or product concept. These digital models are referred here after to as virtual prototypes. A virtual prototype simulates product features with a degree of functional realism comparable at least to a physical prototype [1]. Simulatable features include product's visual outlook, sound characteristics, functionality of user interface, functionality and behaviour of the product, and tactile and force feedback present in user-to-product interactions. Virtual prototypes can be used instead or in combination with physical prototypes. Virtual prototyping technology is based on integrated application of advanced modelling, multiple discipline simulation, interactive user interface and virtual reality techniques.
Products in the area of electronics and telecommunication are typically multi-technology products, whose development incorporates several engineering technologies that employ their own modelling languages as well as simulation techniques. These products comprise embedded computer systems consisting of hardware and software subsystems. Software engineers make use of graphical languages to define and design software subsystems, and programming languages, such as C or C++, to implement them. Electronical engineers use, for instance, executable VHDL (VHSIC Hardware Description Language) [2] models to produce designs of hardware subsystems. Mechanical engineers work with CAD (Computer-Aided Design) models to specify functionality of mechanical assemblies.
Nowadays, various phases of the product development process are mainly computer aided. For instance, CASE (Computer-Aided Software Engineering) tools have been used in software development and EDA (Electronic Design Automation) tools has been utilised in electronics product development. Unfortunately, the current computer-aided development tools usually lack inter-tool communication and operation capabilities.
Virtual prototyping has to add value compared to traditional approaches if it is to be of real value in the development of the products in question. The following benefits are generally presented for it. Virtual prototypes can be used to:
A virtual prototyping system is a development environment, which includes the necessary tools to create, build, test, modify, use and maintain virtual prototypes. These tools provide the following basic functions: modelling, simulation, rendering, and control. Figure 1 depicts a classification of the basic functions and their interrelationships.
Modelling of virtual prototype may start by defining the geometry of the product. Then, surface properties, such as colour, material, and texture are added to the geometric model. Modelling of the functionality of mechanical assemblies is usually necessary, especially in products whose physical user interface includes buttons and other controls. The reason for this is that it is increasingly important to a product to have a convenient and easy-to-operate user interface. Modelling of the product's behaviour and functionality comprises, for instance, modelling of the user interface, and the functions implemented by embedded software and hardware.
Ultimately, virtual prototyping is accomplished by executing computer models of virtual prototypes through iterative dynamic simulations prior to making any physical prototypes. Kinematics simulation is used to apply the laws of the physics to a geometric model to analyse the motion of the mechanical components of the product. Kinematics simulation is usually integrated to the visualisation model.
A novel form of simulation within virtual prototyping is haptic simulation [4], which means that the virtual prototype produces tactile and force feedback for the user through a specific interface device called a haptic display. The user is able to touch and feel the texture of the virtual prototype and the force feedback occurring in the simulated user-to-product interactions. Providing haptic feedback is necessary, since conventional rendering techniques may not provide the user with sufficient information about the geometry and textures of the physical user interface, and further, about manipulability of it. Simulation of texture involves simulation of surface friction, and many studies have already demonstrated the importance of it [5]. Without friction simulation, the resulting surfaces feel like oiled glass. The problem is that the user's probe tends to slip off convex surfaces and into concave ones. Therefore, it is necessary to provide surface friction simulation for various materials used in hand-held products, such as plastics and rubber.
Electronics and telecommunication products contain various components whose functionality and/or behaviour can be simulated:
Simulation of these and other components implies that appropriate simulation tools are applicable with and integratable to the used virtual prototyping system. For instance, simulation of certain hardware components, such as ASICs (Application Specifid Integrated Circuits), would require integration of a selected VHDL simulation tool to the virtual prototyping system.
Visual rendering involves creating as realistic view as possible of the virtual prototype. In most cases, it is not possible to achieve real-time photorealistic imaging with a medium-sized graphical workstation, but appropriately simplified visual rendering has to be used. Realism of 3D images can be augmented by using a stereoscopic viewing system, that provides the user with a realistic feeling of depth. A head-tracked stereo display produces high-quality virtual reality imagery of the virtual prototype.
In its simplest form, auditory rendering is achieved by generating sound effects that simulate the auditory features of the product. To provide more realistic auditory rendering, the user could be immersed into a 3D virtual sound environment produced using a 3D sound generator and renderer. Accurate rendering of simulated audio characteristics is highly beneficial for products like mobile phones. Accurate acoustic simulation and high-quality auditory rendering could be used to analyse, for instance, how the form of the mobile phone or location of the speaker affects the sound quality.
Haptic rendering means that the user is provided with a tactile and force feedback about the user-to-part interactions with the geometric model of the virtual prototype. The sensing of force and tactile feedback is tightly coupled to the visual sensing and one's spatial sense [5]. Therefore, haptic feedback usefully enhances the capabilities of the user to directly interact with the virtual model of a product's physical user interface. This provides more efficient means to test physical user interfaces in earlier stages of the development.
The realism achieved in the rendering of simulated product features depends not only of the applied rendering systems and their quality, but also of the consistency of the clues produced by various rendering systems. Therefore, careful synchronisation and integration of applied rendering techniques are essential for a virtual prototyping system.
The user's control and interactions are arranged through various output and input interface devices. Typical examples of the former include visual display devices (e.g., monitor, stereo classes, HDMs), sound cards with loudspeakers, and haptic displays. The latter include, beside conventional keyboard and 2D mouse, 3D mouse, various tracking devices, haptic control devices and microphones. To achieve effective simulation of user-to-product interaction in virtual prototyping of electronics and telecommunication products, it is necessary to apply all the above mentioned interface devices.
Virtual Reality Prototyping (VRP) project [6] at VTT Electronics aims at developing a virtual prototyping system called VRP Environment for electronics and telecommunication products. The VRP Environment is used as a research environment to study, develop and demonstrate virtual prototyping technology for these products. Figure 2 shows a part of the implemented system with the main interface devices. In the creation of the environment, the emphasis has been on the using of existing software and hardware components. Therefore, the effort could be focused on making an effective integration of the selected components, and filling the gaps with new software only where required.
In the current virtual prototyping system, virtual prototypes consist of the following component models that are illustrated in Figure 3:
The generic structure of virtual prototypes and the main steps in the process of creating a virtual prototype are shown as a block diagram in Figure 3. Creation and building of a virtual prototype imply the following main steps:
Selected parts of the logical behaviour and functionality of the product (see virtual prototype logical object model in Figure 3) are specified using an object-oriented formalism similar to one provided in the ROOM method [9][10]. Different components of a target product, such as hardware or software components, are represented in this model as virtual components. The object-oriented formalism can be used to describe the hierarchical structure of virtual components, their behaviour, functionality, and inter-object communication. The formalism, virtual components, a supporting graphical model editor and simulation engine are all written in Java [11]. The Java language and the Java Virtual Machine (Java VM) provided several advanced features, such as object reflection, dynamic loading of classes and multithreading, that suited well to implement various mechanisms of the formalism and simulation engine.
If more efficient performance for simulation or specific simulation techniques is required, e.g., VHDL-based simulation of hardware components, existing simulation tools are used (see External SW/HW Simulation Models in Figure 3). They are connected to the overall environment using appropriate integration interfaces and platforms. We are also experimenting in using Java as a Universal Description Language (UDL): One language for describing both software and hardware.
It is also possible to integrate existing hardware or software to a virtual prototype. This approach can be utilised especially in cases where a next-generation product is developed using a current generation product as a starting-point. The utilisation of existing technology compresses the development cycle of the new product considerably. To experiment with this kind of approach, we have connected a GSM cellular phone to our virtual prototyping system, which enables us to make actual calls to GSM network using virtual prototypes of next-generation mobile phones [8].
Since virtual prototyping process is an iterative prototyping process, the cycle to create, build, and modify successive prototype versions should be as short as possible. One way to accomplish this requirement, is to employ reusable components as much as possible. There are several artifacts in the current environment that can be handled as reusable components:
To support shared reuse of various virtual prototype components, we implemented a server called WebCompo that is based on a HTTP server [13] and can be accessed using any WWW browser supporting frames and JavaScript. WebCompo supports management, browsing, viewing and use of shared components by providing a set of applications, which are based on Java, JavaScript, CGI programming and HTML.
Figure 4 illustrates how the execution of virtual prototypes is distributed among several workstations: Visual and auditory rendering, and kinematics simulation are performed in a SGI Indigo 2 workstation: Virtual acoustics simulator is run in a Pentium PC workstation, as well as haptic rendering in a Pentium PC. Logical model simulations, hardware and software simulations are executed either in Sun SPARCStations or Pentium PCs.
As mentioned above, a haptic simulation model is generated from the geometric model represented using the OpenInventor format. The OpenInventor model is parsed resulting a node tree (scene graph), which consists of OpenInventor nodes. Haptic simulation model can be thought to be composed of set of virtual objects, each of which, in turn, can be thought to be composed of several virtual planes. Haptic simulation/rendering is based on the plane and probe and point-to-point spring techniques [5] provided by the UNC-CH Force-Feedback Library [14].
The user of a virtual prototype, either person participating in actual development of the product or the end user of the product, interacts with the virtual prototype though various user interface devices (Figure 5):
Visual feedback is provided for the user using a conventional monitor. As a visual renderer we use SGI's OpenInventor. Realism of the visual feedback has been increased by providing the user with a sensation that the 3D image of the virtual prototype is floating in front of the monitor screen. The sensation is achieved using the CrystalEyes stereoscopic system. It is necessary to change the 3D image according to the user's viewing direction in order to avoid any distortions. For that reason, the Logitech 3D Head Tracker has been integrated to the visual feedback system to track the position and orientation of the user's head. The visual rendering system software uses this 3D tracker data to re-calculate the 3D image of the virtual prototype.
Simulated sounds of the product are rendered either through stereo headphones or loudspeakers. At the moment, the auditory rendering includes merely generation of sound effects utilising pre-defined sound files. In near future, our virtual prototyping system will include auditory rendering hardware and software needed to generate 3D virtual sound environments.
By using the PHANToM haptic interface device, the user is able both to directly touch and feel virtual objects and manipulate them as if they were physical objects. The haptic display can be used to simulate the user's interactions with the physical user interface of the product, such as controls. PHANToM is a one finger system and is feasible to be utilised with small-sized virtual prototypes and in point-to-point interactions. The CyberGlove system will be integrated to the haptic system to achieve more accurate and realistic haptic simulation.
The research environment includes also the Polhemus FASTRAK tracking device based on electromagnetic fields. The device consists of four tiny receivers, which can be attached to the objects to be tracked, and of one transmitter. A possible application with virtual prototypes would be to attach a receiver to a rapid prototype manufactured from plastic using stereolitography, and to update the stereoscopic 3D image according to the user's manipulation of the rapid prototype.
Virtual prototyping is a novel approach which attempts to produce products with better accuracy and quality in shorten period of time. It will provide earlier detection of design flaws which otherwise would be expensive to correct in the later stages of development. The full-function virtual prototyping system for electronics and telecommunication products will let the developers to virtually manufacture products and test them with the end users without actually building any physical prototypes.
In order to study and develop virtual prototyping technology for the above mentioned products and to demonstrate its possibilities, we are creating an experimental virtual prototyping system. Fundamental to the effort is the investigation of effective integration of various simulation models, and comprehensive and realistic rendering techniques for simulated product features. A key aspect in the development of rendering techniques is to augment visual rendering of virtual prototypes with a haptic interface which allows the user to receive tactile and force feedback from the direct manipulation of a virtual prototype. In short, advanced user interfaces based on virtual reality techniques make virtual prototypes intuitively understandable.
The building of the presented virtual prototyping system implies solving of many demanding problems. To demonstrate the possibilities to utilise it in the development of electronics and telecommunication products, the key objectives are to:
We have also launched other related research projects together with our research partners. For instance, one such project aims at developing Internet/intranet -based distributed virtual prototyping techniques. The projects support the development of both the virtual prototyping system, and products and services based on virtual prototyping technology.
The authors would like to acknowledge the members of the research group at VTT Electronics for their comments and assistance in this work. In particular, we would thank Tommi Anttila, Rauli Kaksonen, Harri Kyllönen, Tarja Leinonen, and Marko Palola for their work on parts of the research environment.
| [1] | Haug, E.J., Kuhl, J.G., Tsai, F.F. "Virtual Prototyping for Mechanical System Concurrent Engineering." In Haug E.J. (ed.) Concurrent Engineering: Tools and Technologies for Mechanical System Design. NATO ASI Series Series F: Computer and Systems Sciences. vol. 108, 1993, p. 851-879. Springer-Verlag, Berlin Heidelberg. |
| [2] | Lipsett, R., Schaefer, C., and Ussery, C. "VHDL: Hardware Description and Design", Kluwer Academic Publishers 1989. |
| [3] | Maxfield, J., Fernando, T., and Dew, P. "A Distributed Virtual Environment for Concurrent Engineering", Proc. of the IEEE Virtual Reality Annual International Symbosium (VRAIS'95), Research Triangle Park, North Carolina, West Virginia, March 1995. 9 p. |
| [4] | Burdea, G. "Force and Touch Feedback for Virtual Reality", John Wiley & Sons Inc., 1996, 339 p. |
| [5] | Mark, W.R., Randolph, S.C., Finch, M., Van Verth, J.M., and Taylor, M.R. "Adding Force Feedback to Graphics Systems: Issues and Solutions", Proc. of SIGGRAPH 96, New Orleans, Lousiana, August 4-9, 1996, 6 p |
| [6] | VRP project at VTT Electronics, http://www.ele.vtt.fi/projects/vrp/vrp.html |
| [7] | SGI OpenInventor, http://www.sgi.com/Technology/Inven or.html |
| [8] | Kerttula, M., Pulli, P., and Similä, J."Product Development, Marketing and Mobility Support in Virtual Enterprice", Proceedings of CE'96, the Third International Workshop and Conference on Concurrent Engineering, November 7-8, 1996, 11 p. |
| [9] | Selic, B. Gullekson, G. and Ward, P. "Real-Time Object-Oriented Modeling", John Wiley & Sons Inc., 1994, 560 p. |
| [10] | ROOM, http://www.objectime.on.ca |
| [11] | Java, http://java.sun.com |
| [12] | VRML 2.0, http://vrml.sgi.com/worlds/index.html |
| [13] | Network Working Group. HTTP 1.0 specification. WWW: http://www.w3.org/pub/WWW/Protocols/ |
| [14] | Mark, W.R., Randolph, S.C., Finch, M., and Van Verth, J.M. "UNC-CH Force Feedback Library", Technical Document, University of North Carolina at Chapel Hill Department of Computer Science. 1996, 67 p. |
| [15] | Massie, T., and Salisbury, J. "The PHANToM Haptic Interface: A Device for Probing Virtual Objects", Proc. of the ASME Winter Annual Meeting, Symbosium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Chicago, November 1994, 7 p. |
| [16] | PHANToM Haptic Device, http:://www.senseable.com/ |
| [17] | CyberGlove System, http://users.quake.net:80/~virtex/products.html |
| [18] | POLHEMUS FASTRAK, http://www.polhemus.com/trackers/ trakds.html |
| [19] | CrystalEyes Stereoscopic System, http://www.infolane.com/infolane/ster og/ |