Abstract
This paper describes a robust input method that allows an immersed virtual reality user to perform common input tasks such as keyboard data entry and telephone dialing. The method is based on the use of the standard telephone keypad which, in this instance, is reversed, placed upright and operated by the fingers from the reverse side. The keypad, in the numeric input mode, is operated by the immersed user by referring to the 3D representation of the keypad in the virtual environment and actuating the spatially corresponding keys on the reverse mounted keypad. The basic functionality of the keypad is expanded through the use of key chording whereby the key chords are viewed as binary symbols (binagrams) within the virtual environment. This permits alphanumeric and other forms of data entry. This input method is further enhanced by using a technique for creating macro binagrams that are used for such tasks as speed dialing a telephone or word input. The general method of using binagrams for data representation and input (binagraphy) is being developed for its application to standard telecommunications as well as virtual reality. A six-degree-of-freedom handgrip, called a binagrip, with a telephone-type keypad located on its reverse side allows not only data entry but also complete navigation control within a virtual environment.
Introduction
Although the World Wide Web is one of the fastest growing means of communication in history, its growth could explode as further technological advances are made. These advances must make access to the Web much easier than it is today. While the introduction of graphical Web browsers has made it easier to surf the Web, one still needs a computer and has to jump through various hoops to access specific sources of information. The introduction of high-speed fiber optics into the home promises interactive access to the Web through cable TV. PCs as we know them today with their standard keyboards and mouse controls would become an obstacle to easy interaction with information on the Web. Languages such as VRML will allow Web sites to build 3-D worlds for users to explore. To navigate through these 3-D worlds in an arbitrary way one needs some type of 6 degree-of-freedom device that will allow complete control of one's position and orientation. Current devices for doing this, such as a spaceball or a mouse, have significant limitations. The spaceball can be frustrating to use because it is difficult to move in a single direction without inadvertently rotating or moving in an unwanted direction. Similarly, it is difficult to rotate about a particular axis without inadvertently moving in some direction or rotating about some other axis. The mouse was not designed for navigating in a 3-D space and thus has to be adapted in some way for such a use. This is typically done by using a combination of button pressings and cursor location on the screen to indicate which of the 6 degrees-of-freedom is to be varied. This technique is far from intuitive and also makes it easy to couple two types of motion inadvertently.
This paper describes a new device, called a binagrip, that allows the user to control all six degrees-of-freedom independently. In addition, a keypad is built into the binagrip at the natural location of the fingers. This allows one to interact and communicate with virtual environments without the need of a computer keyboard. The keypad has the layout of a standard telephone keypad. Key chording, the pressing of more than one key simultaneously, can be used to enter alphanumeric data or perform the task of a function key such as selecting an object in a virtual environment. The key chords can be viewed as binary symbols, called binagrams, that can be located on (as if pasted on) objects in the virtual world. Pressing the chord representing the binagram associated with a particular object will select that object. For example, selecting the receiver of a phone will lift the receiver, produce a dial tone, and wait for a telephone number to be entered on the keypad
The Binagrip
A photo of the first binagrip prototype is shown in Figure 1. Three spring-loaded gimbals are mounted at right-angles to each other about a central point. This allows the user to rotate the binagrip about the roll, pitch, and yaw axes independently, or in combination. Suppose that the binagrip is attached to the viewpoint. As long as the user rotates the binagrip through some yaw angle, for example, the viewpoint will continually rotate in that direction. When the user returns the binagrip to its initial neutral position the rotation will stop. This is the same intuitive behavior one would expect from say steering a car or boat. A rocker-arm lever located at the thumb position is used to move forward and backward in the viewpoint's z-direction, i.e. the direction in which the viewpoint is looking. A separate spring-loaded, 4-position switch is used to move in the ±x and ±y directions. For example, holding the switch in the up position will move the viewpoint up, while holding the switch to the left will move the viewpoint to the left
Figure 1 The original binagrip prototype
A telephone-type keypad is located on the binagrip's reverse side. A microcontroller and a telephone dialing circuit are located in the base. The device is connected by cable to both an external speaker phone and to the computer that generates the virtual environment. A driver for the binagrip has been written for Sense8's WorldToolKit. The binagrip can be attached to the user's viewpoint or it can be attached to any object in the virtual world. It can therefore be used as a 3-D mouse or to interact with 3-D objects.
The binagrip is connected to the host computer through a standard asynchronous serial port. When the host computer sends an ASCII "T" to the binagrip through this serial line, the binagrip responds by sending six bytes of data to the host computer. The first four bytes contain four values between 0 and 255 that represent the positions of four potentiometers that measure the three angles (roll, pitch, and yaw) and the z-direction rocker arm. These four values are derived from the four potentiometer voltages that are converted to digital values using an A/D converter built into the microcontroller within the binagrip. The last two bytes sent to the host computer represent a 16-bit integer in which each bit is associated with one of the twelve key switches or with one of the four positions of the ±x±y switch. These six bytes of raw data are converted to changes in position and orientation by the WorldToolKit sensor driver that makes this information, together with the key pressings, available to the application program.
Several ergonomical improvements have been incorporated into more recent models of the binagrip. The latest model, shown in Figure 2, is molded to fit easily into the user's hand. The z-direction and x-direction movements have been combined into a single thumb-controlled, self-centering lever that can be moved forward or backward and left or right. These rocker-arm movements are recorded by two potentiometers. A third rocker-arm potentiometer controlled by the index and middle fingers is used for moving up and down in the y-direction. The entire binagrip can be rotated in the roll, pitch, and yaw directions as with the original prototype.
Figure 2 The latest binagrip model
The newer binagrip shown in Figure 2 has only 4 keys compared to the 12 keys on the original prototype shown in Figure 1. These four keys can be used as function keys for a variety of control operations. Again both single and multiple key pressings can be detected. A separate 12-key keypad has been developed for simultaneous use by the left hand while the right hand is controlling the binagrip. This left-handed keypad can be used for dialing phones or other operations where alphanumeric information is required. This can be done using the binagraphy coding scheme described in the next section.
Binagraphy
The binagrip's telephone keypad is split into two parts with eight key switches on the user's left and the remaining four key switches on the right. Note that in the photograph of the virtual environment in Figure 3, there is a binagram on the receiver of the virtual telephone. The binagram is a rectangle made of black and white squares. A person immersed in this virtual environment would dial the actual telephone by first selecting the telephone by pressing the top and bottom key switches on the right side of the control device. This key chord is detected when the user releases all keys. This key combination corresponds to the binagram on the virtual receiver. The telephone number is then dialed by viewing the virtual telephone and pressing the spatially corresponding key switches on the actual keypad - the upper left key switch of the virtual telephone corresponds to the upper left switch of the actual keypad and so on.
Figure 3 A phone in a virtual world
The right side of Figure 4 shows how the 10 digits 0-9 can be indicated on a linear 4-key keypad by using multiple key pressings of contiguous keys. Thus, for example, the four keys on the binagrip shown in Figure 2 could be used to dial a phone. The left side of Figure 4 shows how the 26 letters of the alphabet can be encoded on a 4 x 2 keypad by using a maximum of three contiguous multiple key pressings. The binagrams shown in Figure 4 for encoding characters can be displayed in the virtual environment when characters need to be typed on the keypad. Pressing non-contiguous keys can be used to control various functions such as lifting a phone receiver or switching the binagrip mode. For example, in one mode the binagrip might be attached to the viewpoint while in another mode it might be attached to an object in the virtual environment.
As another example, a pair of binagrams (start_definition and end_definition) can be used to associate a sequence of operations with a new binagram. In this way the new binagram is defined in terms of other, previously defined, binagrams. This programming method can be used for any binagraphic device such as the telephone for speed dialing.
Binagraphics as a symbol system was developed based on an analysis of the contiguous black square patterns in the 4 x 3 matrix of black and white squares where a given black square is meant to be synonymous to a binary 1 and signify a key switch in its depressed state. The 4 x 3 matrix corresponds to the key-switch layout of the basic AT&T telephone keypad. A linear binary array of length n has n(n+1)/2 contiguous patterns of 1's. Therefore an array of length 4 has ten contiguous patterns, as shown in Figure 4. This is significant because it suggests that just one column of the 4 x 3 matrix can serve as the basis for assigning the 10 decimal digits of an alphanumeric system.
If the third column of the matrix is to be used for the 10 digits there obviously remains a 4 x 2 matrix which can be assigned symbols. An analysis of this matrix shows, by counting, that there are 34 contiguous patterns of three or less black squares. This is ample for assigning the 26 letters of the alphabet as shown in Figure 4.
Figure 4 Encoding the digits 0-9 with four keys and the letters A-Z with eight
Conclusion
The binagrip has been designed to provide a convenient human-machine interface for navigating and communicating in virtual environments. Because the hands and fingers are in constant contact with the appropriate switches and controls the user can keep his or her eyes on the screen when operating the binagrip. This will contribute to the sense of immersion and is particularly important when using head-mounted displays
Selecting an object within a virtual environment is a common operation in many applications. While the binagrip can be used to move a 3-D cursor to any location and then click a key to select an object, a different paradigm is possible with the binagrip. Each object to be selected can have a unique binagram (e.g. a 4-bit pattern) associated with it. This binagram could actually appear on the object. Simply pressing the four keys on the binagrip in the binagram pattern would select the object. This could be a much faster operation than first moving the cursor to the object before pressing a key