MagGetz: Customizable Passive Tangible Controllers On and Around Conventional Mobile Devices

Sungjae Hwang, Myungwook Ahn, Kwang-yun Wohn Graduate School of Culture Technology

Korea Advanced Institute of Science and Technology {best, chapi, wohn}@kaist.ac.kr

ABSTRACT This paper proposes user-customizable passive control widgets, called MagGetz, which enable tangible interaction on and around mobile devices without requiring power or wireless connections. This is achieved by tracking and ana-lyzing the magnetic field generated by controllers attached on and around the device through a single magnetometer, which is commonly integrated in smartphones today. The proposed method provides users with a broader interaction area, customizable input layouts, richer physical clues, and higher input expressiveness without the need for hardware modifications. We have presented a software toolkit and several applications using MagGetz.

Author Keywords Tangible User Interface (TUI), Around Device Interaction (ADI), pressure input, magnetometer, mobile device.

ACM Classification Keywords H.5.2 [Information interfaces and presentation]: User Inter-faces. – Input devices and strategies.

INTRODUCTION Physical objects in real life provide users with intuitive tactile clues through its physical artifacts so that users can conduct low-attention and vision-free interactions. Howev-er, flat touchscreens, commonly used on conventional mo-bile devices, have several interaction issues. First, users’ fat fingers [19] frequently occlude the content of interest. Sec-ond, the virtual controls lack tactile clues and require visual attention. Third, the touchscreen device merely detects touch coordination and the binary touch state (i.e., touch and release). Based on these issues, researchers have since long attempted to increase this input space by adding other types of tangible objects. These pioneering methods in-clude the usage of a capacitive foot print [4, 10, 15, 21, 22], camera markers [2, 11], palettes that embedded conductive

layers [5, 20], and magnetic augmentations [1, 6, 12, 13, 14, 17, 18]. However, some approaches do not alleviate the occlusion problem (e.g., 4, 10, 15, 21), some need addi-tional special hardware (e.g., 1, 2, 5, 17, 20, 21), or some do not have any tactile clues (e.g., 6, 12, 13, 14, 18).

In this paper, we have introduced a set of tangible control widgets to provide users with richer tactile clues, higher input expressiveness, and a wider interaction area without the need for hardware modifications (Figure 1). The goal of this research is to show how the magnetically driven tangi-ble controllers can be successfully integrated for conven-tional mobile devices. The rest of the paper introduces the MagGetz, with the technical novelty and benefits of this approach. The second part of this paper describes the pos-sible applications using the MagGetz and indicates future avenues of our research.

RELATED WORK Here, we present separately in the next three sections the previous work on each of the three related fields.

Tangible Interactions for Mobile Devices Tangible user interfaces (TUIs) provide physical forms of digital information and computation, facilitating the direct manipulation of bits [8]. However, touchscreen devices do not support this beneficial tactile clue. With this background, the

Permission to make digital or hard copies of all or part of this work for person-al or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permit-ted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. UIST'13, October 08 - 11 2013, St Andrews, United Kingdom Copyright 2013 ACM 978-1-4503-2268-3/13/10…$15.00. http://dx.doi.org/10.1145/2501988.2501991

Figure 1: MagGetz enables tangible interaction on and around mobile devices without requiring pow-

er or wireless connections

research community has proposed a variety of methods to enable tangible interaction for touchscreen devices. The first approach relies on capacitive markers [4, 10, 15, 21, 22]. Yu et al. [21] suggested a technique that senses and tracks objects on a capacitive touchscreen by simulating finger touches us-ing passive materials and active modulation circuits embed-ded inside tangible objects. Later, they introduced a passive controller, called a Clip-on gadget [22], which clips on the bezel to reduce screen occlusion. Similarly, Kratz et al. [15] presented a passive rotary knob controller that works with a capacitive touchscreen. Chan et al. [4] introduced tangible blocks that allow an underlying capacitive touchscreen to identity different blocks or stacking heights. Jansen et al. [10] explored the tangible remote controllers for a wall-size dis-play. They explored the benefits of using tangible sliders on a table as a remote controller by comparing a virtual and tangi-ble slider. These studies enabled tangible interactions on capacitive multi-touch devices without requiring any hard-ware modifications. However, the interaction area is con-

fined within the touchscreen, which results in occlusion problems. In addition, the passive widgets are invisible to the touchscreen when it is untouched, and the active widg-ets need circuit hardware and a power.

The second approach of enabling tangible interaction for mobile devices is using cameras [2, 11]. Portico [2] is a port-able system for enabling tangible interaction on and around table computers. In that study, the authors used two cameras on small foldable arms to recognize a variety of physical objects placed on and around the tablet. In addition, Kato et al. [11] introduced a tangible joystick that realizes continu-ous analog input by tracking a marker at the top end of a controller with the device’s inertial camera. This approach successfully extends its interaction area with richer tactile clues. However, the cameras do not work properly in the dark environment. Moreover, the cameras and foldable arms are very bulky [2], or the controller’s location is fixed and limited with regard to the location of camera [11].

Figure 3: The location of a magnet and the visualization of magnetism for each control element. The side view of push button (a), toggle switch (b), slider (c), and joystick (d). The top view of the rotational wheel (e).

Type

Magnet orienta-

tion (to the

ground)

Magnet size

(mm)

Magnetic strength

(uT)

Max. working distance

(mm)

Input expres-

sive-ness

Button (Small)

Linear

Horizon-tal

2ø, 5T 300 100 Analog

Button (Large)

5ø, 3T 500 100 Analog

Toggle (Small)

5ø, 3T 500 150 Binary

Toggle (Large)

8ø, 5T 800 200 Binary

Slider Vertical 8ø, 3T 600 200 Analog

Key lock

Curved Vertical 2ø, 5T 300 120 Binary

Wheel 120 Analog

Joystick2D

planeHorizon-

tal

10ø, 5T 1500 400 Analog

Suction Stick

5ø, 3T 500 100 Analog

Table 1: Comparison of MagGetz control widgets.

Push Buttons Toggles Sliders

Joystick Suction sticks

Key-lockingdials

Wheels

(a)

(b) (c)

Figure 2: Three types of MagGetz controllers: (a) linear movement type, (b) curved movement

type, and (c) 2D plane movement type.

Magnetically Driven Input for Mobile Devices Magnets are very cheap, small, and robust, do not require power, and their positions and orientations can be easily detected wirelessly using magnetometers embedded in modern phones [3]. Moreover, they do not easily degrade over time. For these reasons, a number of input techniques using magnetism have been introduced, attempting to ex-pand the interacting surface beyond the physical bounds of devices [6, 12, 13, 14, 18]. Harrison and Hudson [6] de-scribed an input interaction around the small device through a finger-attached magnet. Similarly, Ketabdar et al. [12] proposed a gestural interaction for playing music through a small magnet worn on the finger. They used a magnetometer embedded in typical smartphones for inter-action with music-related applications. Besides these, mag-netic-based interfaces have been successfully applied to many tasks such as hand-writing [13], turning pages and zooming [14] or gestural authenticating [18]. These ap-proaches definitely improve usability by expanding the interaction beyond the bounds of the device screen. How-ever, the main limitation is that they do not provide users with tactile clues.

More recently, Ashbrook et al. [1] presented a subtle input with a magnetically tracked finger ring. They introduced a unique input method that makes a selection by twisting the ring and clicks by sliding it along the finger. Liang et al. [17] introduced GaussSense, which is a back-of-device sensing technique for enabling input on an arbitrary surface using a stylus. While these approaches provide users with tactile clues through their tangible tokens (e.g., ring [1] and stylus [17]), they need additional sensing hardware (e.g., a wrist-worn tracking bracelet [1] and a magnetic sensor grid [17]). Besides these approaches, magnets are successfully used in more con-strained ways. For instance, Mobile ActDresses [9] uses the specific spatial properties of a magnet attached onto slip-on device covers to customize the visual theme. Moreover, many devices in the market already have an automatic wake and sleep feature triggered by a magnet (e.g., Nexus7 and iPad2).

Customizable physical Interfaces Unlike virtual controls on screens, physical controllers provide richer tactile clues through their rigid shapes and predetermined layouts. However, the physicality of these

interfaces makes them static, limiting the customization of the interfaces they can support. To address this issue, re-searchers have attempted to support the customization of physical interfaces [5, 7, 16, 20]. Phidgets (physical widg-ets) [5] constitute a general toolkit of physical user inter-face components. The toolkit provides a developer with a collection of controls, sensors, and actuators to design and customize physical interfaces. Similarly, Calder [16] and d.tools [7] are toolkits for supporting product designers in the process of developing rapid prototypes of their ideas. Later, Villar et al. [20] introduced a malleable control structure made up of a set of controls that users can freely arrange. In their study, the individual controls can be ac-tively added, removed, and rearranged on a flexible sub-strate that embeds conductive layers and custom-designed coaxial connectors.

While these studies support a rapid prototyping and de-ployment of custom interfaces, they need wires or wireless connections with electric powers. Moreover, they do not support the real-time adaptation of the physical layout.

THE DESIGN OF MAGGETZ Building on previous works, we take advantage of magnets and suggest unpowered, low-cost, and user-configurable magnetic tangible controllers on and around mobile devices using them. Here, we describe our system, which compris-es both hardware and software.

Hardware Design MagGetz constitutes a set of tangible control widgets made from traditional physical input elements (~15 US$) covered with an acrylic case. In order to track the physical move-ment of control widgets, we attached a permanent magnet on the appropriate position of each widget (see Figure 3). We then covered it with an acrylic case that we designed using computer-aided design (CAD) and attached a 3M restickable adhesive tape (R100) underneath the case in order for the magnet to be individually attached, detached, and rearranged on various surfaces (e.g., desks, back of mobile phones, or phone holders). Basically, we have im-plemented three different types of MagGetz based on the magnetic movement: linear type (e.g., push buttons, toggle switches, and sliders), curve type (e.g., rotational wheels, and key-locking dials), and two-dimensional (2D) plane

Figure 4: Magnetic traces for different control widgets: (a) a button, (b) a wheel, and (c) a joystick. The combination of multiple widgets: (d) two buttons, and (e) three toggle switches.

type (e.g., a joystick, and suction-cup sticks). The control widgets of each type are shown in Figure 2. These controls act as input blocks that can be combined to create a custom control environment. For instance, the user can attach three different control widgets surrounding the device as shown in Figure 1. To make a noticeable difference in the magnet-ic field when the user changes the states of controllers (e.g., pushing the button, rotating the wheel, or turning the toggle switch on and off), we chose an appropriate permanent magnet and embedded it on a place where it moves as much as possible (see Figure 3). We used Neodymium coin magnets with 3~5mm in thickness and 2~10mm in diame-ter for different widgets (300 ~1500uT).

The comparison of MagGetz control widgets is shown in Table 1. Referring to this, the working distance varies de-pending on how we arrange the magnet: The distance that the magnetic pulse reaches increased when the base side of the magnet is heading for the sensor (e.g., the wheel’s working distance is much longer than the small button’s). The work-ing distance is the maximum distance between the sensor of the device and the controller at which they work properly. With this setup, the system could measure a change in the magnetic field according to the state of each controller.

Software Design We developed a general toolkit on Android 4.0.4. (SHV-E160S, Samsung Galaxy Note) using Java. We also im-plemented a three-dimensional (3D) Visualizer using OpenGL library for developers to examine a change in the magnetic field. Our software used an inertial three-axis magnetometer that senses the magnetic field strength along with x, y, and z directions at a sampling frequency of 100 Hz. To mitigate sampling noise, we used a fixed size buff-er; the movement vector from the last reading to its prede-cessor is computed and normalized. This process generates a 3D vector that reflects the variations of the magnetic field around the mobile device.

Figure 4 shows the magnetic traces for different control widgets that a three-axis magnetometer sensed. These trac-es are: (a) a straight line when a small button was pressed, (b) a curved line when a wheel was rotated, and (c) a circu-lar plane when a joystick was controlled (all directional points and four right-angled directions from center). As shown in this visualization, the magnetism around the con-trol widget differs according to its physical movement. Basically, our system relies on this phenomenon and de-termines the current states of widgets by comparing a cur-rent magnetic value with the reference magnetic values stored in the calibration procedure. In order to determine the status of widgets, tangible widgets have to be static with respect to the device. For the calibration, our system stores a trace of magnetic values according to the change of the widgets’ states (e.g., pressing a button, or rotating a wheel).

In the case of linear type, the system stores two end-points (e.g., when a button is fully pressed and released) rather than all points at a regular interval distance (see Figure 4a). It then infers the state of the widget by using a ratio for the distances between the current point and the two end-points. To do so, the software projects a new point vector onto the vector between two end points and calculates relative posi-tion within the vector between two end points, which de-notes the state of a widget. For instance, pressure level l applied on a button widget when we have a new input x is computed using the following formula:

(1)

where p1 is a last reference point (e.g., a point when the button is fully pressed), p0 is a first reference point (e.g., a point when the button is not pressed), and x’ is a vector that new point x projected onto p0 p1.

In the case of the curved type, the system captures and stores reference points at a regular interval distance, which is different from the linear type (see Figure 4b). Our system records every magnetic point that changes more than 10uT. For instance, the system could capture 30 points for a wheel while it rotates. Later, the system infers the state of the wheel by finding the nearest one among 30 points to a current point and then considering it as the state of the wheel. The level of rotation l for a wheel is calculated us-ing the following formula:

(2)

where pi is ith reference point from starting point p0 (e.g., minimum rotation of a wheel), k is a sequence of the refer-ence point closest to a new magnetic input point.

For the 2D plane type, our system stores a center point (e.g., the default state of a joystick), four right-angled directional points (e.g., east, west, south, and north), and other direc-tional points (e.g., the arc between east and north) at a reg-ular interval as reference points (see Figure 4c). In order to minimize errors when the circle is distorted, the system infers the state of the joystick by finding a vector closest to a new point among vectors between a center point and di-rectional reference points. The pressure level l on a joystick when we have a new input x is calculated using the follow-ing formula:

(3)

where pc is a center point, pk is a vector that is closest to a new input point x among vectors between a center point pc and directional reference points (p0~pn), pk

n is the last point of vector pk starting from pk

0 (= pc), and x’k is a vector that new point x projected onto pc pk

n. The direction of a joy-stick can be obtained by applying k of pk to equation (2).

Besides the inference of a single widget, our system can track the multiple widgets simultaneously. To do so, our system needs to store magnetic traces for all widgets. In particular, the system stores a point when all widgets are deactivated and points when each widget is activated. Fig-ure 4(d) shows an example of a magnetic trace that two widgets generate (button a, and button b). In order to cali-brate them, for instance, a user needs to attach them around the device and tell the type of widgets to the system. The user then stores three conditions of the widgets by clicking a virtual button on the screen after changing the states of widgets one by one: a condition when the button a and b are released, a condition when the button a is fully pressed while the button b is released, and the button b is fully pressed while the button a is released (needs N+1 condi-tions for using N widgets). In this case, the trace of two buttons generates a 2D-filled square plane and the pressure level applied on button a (la) and the pressure level for but-ton b (lb) are calculated by the following equation:

(4)

where x is a new magnetic point according to the user’s input, pa is a reference point of button a with maximum pressure, pb is a maximum point of button b, and p0 is a reference point of button a and b with minimum pressure. By using the Gauss elimination method, our system can infer the states of each widget’s pressure level (la, and lb) simultaneously. This equation can also be applied to a hex-ahedron graph that three linear-type widgets generate, as shown in Figure 4(e). Our system can individually track the states of control widgets that generate up to three dimen-sional shapes (e.g., a set of three pressure-sensitive buttons, a set of a joystick and a pressure-sensitive button). For in-stance, our system does not work properly for a set of four sliders because magnetic traces of them are overlapped each other.

MAGGETZ APPLICATIONS In this section, we introduce a series of applications using the MagGetz toolkit and their potential interactions.

Control Cover for Mobile Devices If the controllers are fixed on the mobile device, physical movement of the device does not affect the relative posi-tion to the sensor. We have built a series of mobile phone holders that contain some unpowered control widgets. Fig-ure 5(a) shows a game control pad that has a joystick and two buttons. Figure 5(b) shows a cover for taking a photo that consists of a pressure-sensitive button and a wheel.

This setup enables users to use a halfway push with their thumbs and to adjust an aperture with their index fingers. In this way, a user can take a photo through pressure-sensitive interaction, and no occlusions result from this process.

User Customizable Controllers One of the features of MagGetz is a customization of con-trols. We have implemented some applications under the user-customized controllers: a music player and a drawer. Figure 5(c) shows a music player with a toggle switch for controlling the power and a slider for adjusting the volume. Besides this custom setup, the user can rearrange and re-place the toggle switch and slider. For instance, the user can change a slider to a wheel for controlling the volume and rearrange the toggle switch on the back of the device. Another possible application is a drawer, as shown in Fig-ure 5(d). In this case, we designed software that allows the user to control the thickness of a pen through the wheel and to change the color of the pen through the slider.

MagGetz Authentication Similar to the previous work that uses magnetic gestures for authentication [18], we proposed an application that uses magnetic control widgets for the authentication pur-pose. Figure 5(e) shows an example setup that consists of two wheels and a key lock. With this setup, the user places his/her mobile device on the slot and unlocks the system by adjusting the wheels and key locks at a right angle.

Figure 5: Various forms of MagGetz Applications: (a) joystick pad cover, (b) photo-taking cover, (c)

music player, (d) drawer, (e) MagGetz for authenti-cation, and (f) MagGetz number pad

Multiple Widgets with Binary States We can use more than three controllers simultaneously through the binary input control widgets. We have imple-mented a simple calculator application with a physical number pad that contains nine binary buttons, as shown in Figure 5(f). In this case, the system stores nine reference points for nine keys that are pressed and compares them with an incoming value.

DISCUSSION AND CONCLUSION We have presented a MagGetz toolkit and various applica-tions using it, which provide users with a wider interaction area, higher input expressiveness, and a customizable layout of richer physical clues. MagGetz do not need power, a wire-less connection, or the hardware modifications of mobile devices in order to operate. We believe that MagGetz toolkit can be successfully integrated for conventional mobile de-vices. On the contrary, MagGetz do have limitations: First, the magnetism involved might damage the objects (e.g., magnetic stripes on credit cards and magnetic hard disks.) when the objects are very close to the magnets. However, magnets do not affect flash memories on modern mobile devices or IC chips on credit cards. Second, the magnetome-ter we used occasionally reports a large offset in the data or a stuck pointing in one direction. In this case, user should wave the device in figure eight to get back on track. This technical glitch, however, will be undoubtedly solved as sen-sor technology improves. For future work, we plan to design different shapes and forms of input control widgets (e.g., deformable, stretchable, or elastic properties) and consider using magnetic shields to enhance performance.

REFERENCES 1. Ashbrook, D., Baudisch, P., White., S., Nenya: subtle

and eyes-free mobile input with a magnetically-tracked finger ring. In Proc. of CHI, (2011), 2043-2046.

2. Avrahami, D., Wobbrock, J. O., Izadi, S., Portico: tan-gible interaction on and around a tablet. In Proc. of UIST, (2011), 347-356.

3. Bianchi, A., Oakley, I., Designing Tangible Magnetic Appcessories, In Proc. of TEI, (2013), 255-258.

4. Chan, L., Müller, S., Roudaut, A., Baudisch, P., Cap-Stones and ZebraWidgets: sensing stacks of building blocks, dials and sliders on capacitive touch screens. In Proc. of CHI, (2012), 2189-2192.

5. Greenberg, S., Chester F., Phidgets: easy development of physical interfaces through physical widgets. In Proc. of UIST, (2001), 209-218.

6. Harrison, C., Hudson, S. E., Abracadabra: wireless, high-precision, and unpowered finger input for very small mobile devices. In Proc. of UIST, (2009), 121-124.

7. Hartmann. B., Klemmer. S. R., Bernstein. M., Mehta. N., d.tools: Visually Prototyping Physical UIs through Statecharts. In Proc. of UIST, (2005).

8. Ishii, H., Tangible bits: Beyond pixels. In Proc. of TEI, (2008).

9. Jacobsson, M., Fernaeus, Y., Nylander, S., Mobile ActDresses: programming mobile devices by accesso-rizing. In Proc. of CHI EA, (2012), 1071-1074.

10. Jansen, Y., Dragicevic, P., Fekete, J. D., Tangible re-mote controllers for wall-size displays. In Proc. of CHI, (2012), 2865-2874.

11. Kato, H., Kato, T., A camera-based tangible controller for cellular phones. In Proc. of MobileHCI, (2009).

12. Ketabdar, H., Haji-Abolhassani, A., Yüksel, K.A., Kehribar, I., Ercil, A., Digital Music Performance for Mobile Devices Based on Magnetic Interaction, IEEE Transactions on Instrumentation and Measurement, (2011), vol.60, no.11, 3524-3533.

13. Ketabdar, H., Roshandel, M., Yüksel, K.A., MagiWrite: towards touchless digit entry using 3D space around mo-bile devices. In Proc. of MobileHCI, (2010), 443-446.

14. Ketabdar, H., Yüksel, K., A., Roshandel, M., MagiTact: interaction with mobile devices based on compass (magnetic) sensor. In Proc. of IUI, (2010), 413-414.

15. Kratz, S., Westermann, T., Rohs, M., Essl, G., Cap-Widgets: tangile widgets versus multi-touch controls on mobile devices. In Proc. of CHI EA, (2011), 1351-1356.

16. Lee, J. C., Avrahami, D., Hudson, S. E., Forlizzi, J., Dietz, P. H., Leigh, D., The calder toolkit: wired and wireless components for rapidly prototyping interactive devices. In Proc. of DIS, (2004), 167-175.

17. Liang, R.H., Cheng, K.Y., Su, C.H., Weng, C.T., Chen, B.Y., Yang, D.N., GaussSense: Attachable stylus sens-ing using magnetic sensor grid. In Proc. of UIST, (2012), 319-326.

18. Shirazi, A.S., Moghadam, P., Ketabdar, H., Schmidt, A., Assessing the vulnerability of magnetic gestural authen-tication to video-based shoulder surfing attacks. In Proc. of CHI, (2012), 2045-2048.

19. Siek, K. A., Rogers, Y., Connelly, K.H. Fat Finger Wor-ries: How Older and Younger Users Physically Interact with PDAs. In Proc. of INTERACT, (2005), 267-280.

20. Villar, N., Gellersen, H., A Malleable Control Structure for Softwired User Interfaces. In Proc. of TEI, (2007), 49-56.

21. Yu, N., Chan, L., Tsai, S., Hsiao, I., Tsai, D., Hsiao, F., Cheng, L., Chen, M., Huang, P., Hung, Y. TUIC: ena-bling tangible interaction on capacitive multi-touch dis-plays. In Proc. of CHI, (2011), 2995-3004.

22. Yu, N., Tsai, S., Hsiao, I., Tsai, D., Lee, M., Chen, M., Hung, Y. Clip-on Gadgets: expanding multi-touch in-teraction area with unpowered tactile controls. In Proc. of UIST, (2011), 367-372.