Genie: Input Retargeting on the Web through Command Reverse Engineering

Amanda Swearngin, Andrew J. Ko, James Fogarty

2

Culture1 and gender2 affect usability, appeal, and trustworthiness of a website.

1. Reinecke, et. al., ToCHI ’11 2. Simon, et. al. , SIGMIS ’01 3

People with motor disabilities may need to use alternate devices to interact with the web1.

1. Findlater, et al., UIST ’11 4

5

Blind people frequently use screen readers to navigate and operate web interfaces1.

1. Bigham, et. al., IUI ‘09

6

Pixel-Based AnalysisPrefab2

PAX3

Personalized InterfacesSUPPLE3

Interface Adaptation: 3 prior approaches

1. Gajos, et. al., UIST ‘07 2. Dixon, et. al., CHI ’10 3. Chang, et. al., UIST ’11 4. Eagan, et. al., UIST ‘11

Runtime Toolkit ModificationScotty4

7

8

Program Analysis

Genie

9

Enables new interactions on the web throughprogram analysis and input retargeting.

Automatic Speech Input

10

Navigate a website.

GenieInterface

Automatic Speech Input

11

Use your voice to play a game.

Overview

Program Analysis & Input Retargeting

System Details

Demonstration

12

Program Analysis

13

Invoke

Input retargeting

Detect

Program Analysis

Filter

DiscoverProperties

Describe

Original Web Interface

StartPause

Command Objects

.

.

.

onClick = function(e) {login();

}

Command Metadata

Genie UI

Input Retargeting

14

Input (click)

Voice

Keyboard

Input n

.

.

.

Invoke

Input retargeting

Detect

Program Analysis

Filter

DiscoverProperties

Describe

Genie UI

Original Web Interface

onClick = function() {startGame();

}

Dynamic Command Objects

15

Command Object

Properties: enabled, visible, inputBehaviors: invokeLabels

Periodically refreshed

Invoke

Input retargeting

Detect

Program Analysis

Filter

DiscoverProperties

Describe

.

.

.

StartPause

Command Objects

Overview

Program Analysis & Input Retargeting

System Details

Demonstration

16

Detecting CommandsWhat is a command?

Event

button.onClick = function(evt) {startGame();

}

“click”

Interactive ElementOR

17Invoke

Input retargeting

Detect

Program Analysis

Filter

DiscoverProperties

Describe

Detecting Commands

Event RegistrationDOM mutation

Genie

Command 2Command 1

Command Objects

.

.

.

18

Invoke

Input retargeting

Detect

Program Analysis

Filter

DiscoverProperties

Describe

Filtering Commands

click

mouseDown

focus

blur

error load

19

Invoke

Input retargeting

Detect

Program Analysis

Filter

DiscoverProperties

Describe

Filter

Discovering Properties

enabled onClick = function handleStartGame(e) {var startBtn = $(”#startBtn”); if(!startBtn.attr(“disabled”)) {

startBtn.attr(“disabled”), true);startGame();

}}

20

Invoke

Input retargeting

Detect

Program Analysis

DiscoverProperties

Describe

Discovering Properties

21

Filter

Invoke

Input retargeting

Detect

Program Analysis

DiscoverProperties

Describe

enabledvisible CSS

DOM Attributes

Discovering Properties

onKeyDown = function keyDownHandler(e) {if (e.keyCode == 13) {

submitOnEnter(); }

}22

Filter

Invoke

Input retargeting

Detect

Program Analysis

DiscoverProperties

Describe

enabledvisibleinput

Discovering Properties

Genie UI

Update Properties

Every 1s

23

Filter

Invoke

Input retargeting

Detect

Program Analysis

DiscoverProperties

Describe

onClick = function handleStartGame(e) {var startBtn = $(”#startBtn”); if(!startBtn.attr(“disabled”)) {

startBtn.attr(“disabled”), false);

// start the game if any lives remainingif(livesRemaining >= 0) {

startGame(); }

}}

onClick = function handleStartGame(e) {var startBtn = $(”#startBtn”); if(!startBtn.attr(“disabled”)) {

startBtn.attr(“disabled”), false);

// start the game if any lives remainingif(livesRemaining >= 0) {

startGame(); }

}}

DiscoverProperties

Describing Commands

24

Filter

Invoke

Input retargeting

Detect

Program Analysis

Describe

Invoking Commands

“Start”

mouseDownclick

mouseUp

Invoke

Genie UI Command

25

Invoke

Input retargeting

DiscoverProperties

Filter

Detect

Program Analysis

Describe

Demonstration

26

4 example applications3 types of input

Built with Genie API• HTML/CSS• Translation layer• Small amount of code

Audio CommandLine Keyboard

Gaze Gesture Touch

.

.

.

Possible Input Types

Keyboard Shortcuts

27

Type Shift+r to ”Try Again”No shortcut for“New Game”

Keyboard Shortcuts

28

Missing keyboard shortcuts

Type ”33”Type “Ctrl+d”Type “11”Type “Enter”

User types “paperReview”

Creating a meeting Creating a To-Do list

Command-Line Input

29Type commands here. Genie adds To-Do items.

Keyboard Mouse Input

30Type cell number to move mouse

Keyboard Mouse Input

31Type cell number to place a node.

Add node – clickAdd edge –drag

32

Genie

Enables new interactions on the web throughprogram analysis and input retargeting.

Amanda Swearngin amaswea@cs.washington.eduAndrew J. Ko ajko@uw.eduJames Fogarty jfogarty@cs.washington.edu

GenieKey Takeways

Genie enables interacting with the webin new ways through input retargeting

Genie enables discovering and monitoringinteractive commands through program analysis