Improving pen-based interfaces (was Re: scribble update)

Russ Nelson writes:
> Carl Worth writes:
> > Modifiers seem like one of the most useful mappings for buttons ...
>
> 'Course, you then can't use the buttons as global "Run this program"
> buttons like the Palm does. Then again, the Palm doesn't run X.
> Hehe.

:)

Seriously though, I think this is an area where a pen/stroke interface
can be extremely powerful. From most of what I've seen with most PDAs,
(this is speaking from extremely limited experience as I had never
owned anything like this until my iPAQ), the pen-based input has
typically been used for two purposes:

        1) Single-character input, (ala graffiti)
        2) Pointing at on-screen buttons and icons

With (1), the stylus is intended to replace the keyboard, which it
does quite poorly. With (2), the stylus replaces the mouse, which it
does very well indeed! I've long thought that point-and click
interfaces are not ideal when using a desktop computer, (I almost
always prefer a complete set of keybindings). But with a handheld
device, pointing at things on the screen seems very natural.

Now, the area in which I haven't seen the stylus used much is:

        3) Stroke-based command input

This is where pen-based computing could be very powerful. Anyone who
has used the Mentor Graphics CAD suite for more than a short time
certainly appreciates how quickly complex commands can be described
with simple strokes. If you haven't used it -- the entire (complex!)
program was usable with almost nothing but a three button mouse. One
button was used in standard ways for things like selection. The middle
button was used for strokes, (a 'Z' shaped stroke might zoom, 'C'
would copy the selected objects, etc.) This turns out to be
surprisingly efficient as you don't have to lose time switching
between keyboard and mouse.

Getting back to the original message, the way I would like to fire up
common programs is not with a single button press, (which would be
horribly limiting with the iPAQs ~10 buttons), but rather by holding a
button while making a pen-based stroke in the root context or some
such, ('X' for xterm, 'M' for an MP3 player, etc.) This allows for a
virtually unlimited number of programs to be started very
efficiently. (Of course, drop-down menus are still good for those who
prefer them -- and as a way of viewing available programs or reminding
what strokes are mapped to which programs).

Similarly, I think that a great mode of operation for applications
would be to get away from "toolbars", (they waste precious screen real
estate and are very limited in the number of commands they can make
available) and move to stroke-based command input.

In fact, it's painless to enable all programs to accept stroke-based
command input. As long as the application provides a complete set of
keybindings, (which all Good Programs do anyway), the standard strokes
for those keys can simply be entered, (perhaps using buttons for
modifiers). Voila! Suddenly all programs become as powerful as Mentor
Graphics without any rewriting of the code. Additionally, the same
application is easy to use with a stylus, (using strokes to generate
command characters), or with a keyboard, (entering the command
characters directly).

The only thing missing to do all of this right now with the iPAQ,
(sorry I'm on the wrong list), is:

        1) Map some buttons to some modifiers, (Jim says this will be
           in the next release).

        2) Allow strokes on top of application windows. (Not strictly
           necessary, but best for seamless integration). Mentor
           Graphics used a different mouse button to distinguish
           command-strokes from standard mouse input. We don't have
           that luxury, but a transparent xscribble that could be
           easily raised/lowered would be a good start. Maybe I'll
           look into how hard it would be to get Keith's render
           extension into the X server to allow for transparency.
           This is new territory for me -- I've never peeked deep
           into X -- any words or warning or suggestions?

Well, apologies for the long post. Hopefully that wasn't all
meaningless and/or obvious.

-Carl
Received on Wed Sep 06 2000 - 07:49:36 EDT