OnScreenKeyboardMorph: Smalltalk keyboard on a phone

OnScreenKeyboardMorph: Smalltalk keyboard on a phone

Part of a series: #squeak-phone

Back in October 2020, I built an on-screen keyboard for Squeak Smalltalk. It ended up being about 230 lines of code (!) in total.

I’ve been using Smalltalk as the primary UI for my experimental cellphone software stack, and I needed a way to type input. Using VNC to develop on the phone works fine, but to work on the device itself - both for day-to-day phone tasks as well as developing the system - I need a proper on-device keyboard.

This video shows the keyboard in action. As you can see, it’s not perfect! At one point I tapped ctrl while typing, leading to an unexpected pop-up menu appearing.

But the basic idea is sound. (Aside: why aren’t there any Android keyboards that are laid out like a PC keyboard, with punctuation in the right place, etc.?)

The usual shortcuts like Alt-P for “print it”, Alt-D for “do it” and Alt-I for “inspect it” all work for evaluating snippets of Smalltalk interactively.

The keyboard integrates with my previously-written touchscreen support code, with the red/blue/yellow modifier buttons affecting touches, making them into simulated left-/middle-/right-clicks, and with keyboard focus changes auto-popping-up and -down the keyboard.

Simulating mouse clicks is a temporary state of affairs that lets me use the touchscreen reasonably naturally, making use of context menus and so on, without having to make invasive changes to the input pipeline of Squeak’s Morphic.

How it works

Class OnScreenKeyboardMorph synthesizes keyboard events and injects them into the Morphic world.

OnScreenKeyboardMorph >> emitKeystroke: aCharacter
    | evt |
    evt := KeyboardEvent new
        setType: #keystroke
        buttons: modifiers
        position: 0@0
        keyValue: aCharacter asciiValue
        hand: ActiveHand
        stamp: Time millisecondClockValue.
    self resetModifiers.
    ActiveHand handleEvent: evt.

You can have arbitrarily many keyboards instantiated, but there’s a global one living in a flap at the bottom of the screen. That’s the blue tab at the bottom left of the screen you can see in the video.

OnScreenKeyboardMorph class >> rebuildFlap
    | f k |
    self flap ifNotNil: [:old |
        Flaps removeFlapTab: old keepInList: false.
        ActiveWorld reformulateUpdatingMenus].

    k := self new.
    f := FlapTab new referent: k beSticky.
    f setName: self flapId edge: #bottom color: Color blue lighter.
    k beFlap: true.

    Flaps addGlobalFlap: f.
    ActiveWorld addGlobalFlaps.
    ActiveWorld reformulateUpdatingMenus.

I had already written code to read from /dev/input/eventN, tracking each multitouch contact separately. A subclass of HandMorph overrides newKeyboardFocus: to pop the keyboard up and down as the keyboard focus comes and goes:

LinuxInputHandMorph >> newKeyboardFocus: aMorphOrNil
    aMorphOrNil
        ifNil: [OnScreenKeyboardMorph hideFlap]
        ifNotNil: [(OnScreenKeyboardMorph future: 200) raiseFlap].
    ^ super newKeyboardFocus: aMorphOrNil.

Each key is represented by an OnScreenKeyMorph. It tracks keyboard shift state by registering as a dependent of the OnScreenKeyboardMorph:

OnScreenKeyMorph >> keyboard: anOnScreenKeyboardMorph
    keyboard := anOnScreenKeyboardMorph.
    keyboard addDependent: self.

OnScreenKeyMorph >> update: aParameter
    aParameter = #keyboardShiftState ifTrue: [^ self recomputeLabel].
    super update: aParameter.

The important part, of course, is the mouse event handler for OnScreenKeyMorph:

OnScreenKeyMorph >> mouseDown: evt
    selector
        ifNil: [keyboard emitKeystroke: self activeLabel first]
        ifNotNil: [keyboard perform: selector]

The only other interesting part is the keyboard initialization routine, which builds the layout:

OnScreenKeyboardMorph >> initialize
    super initialize.
    self borderWidth: 0.
    self extent: 180 points @ 120 points.
    self layoutPolicy: TableLayout new.
    self wantsHaloFromClick: false.

    font := (TextStyle named: #Roboto) fontOfPointSize: 18.
    modifiers := 0.
    stickyModifiers := 0.
    buttons := Dictionary new.

    {
        '`~ 1! 2@ 3# 4$ 5% 6^ 7& 8* 9( 0) -_ =+' substrings, {'<-' -> #backspace}.
        {'->' -> #tab}, 'qQ wW eE rR tT yY uU iI oO pP [{ ]}' substrings.
        {'esc' -> #escape}, 'aA sS dD fF gG hH jJ kK lL ;: ''" \|' substrings, {#cr}.
        {#shift}, 'zZ xX cC vV bB nN mM ,< .> /?' substrings.
        {#ctrl. #space. #alt}.
        {'|<-' -> #home. '<' -> #arrowLeft. 'v' -> #arrowDown. '^' -> #arrowUp. '>' -> #arrowRight. '->|' -> #end.}.
    } do: [:r | self addMorphBack: (self makeRow: (r collect: [:item | self makeButton: item]))].
    self addMorphBack:
        (self makeRow:
            {
                OnScreenMouseButtonMorph new bit: 4.
                OnScreenMouseButtonMorph new bit: 2.
                OnScreenMouseButtonMorph new bit: 1.
            }).

The rest is book-keeping and simple delegation methods, like this:

OnScreenKeyboardMorph >> shift
    self toggleButton: 8

OnScreenKeyboardMorph >> space
    self emitKeystroke: Character space

and so on.

I haven’t released the whole package yet, because it’s all very much in flux, but if you’re interested, you can take a look at the changeset for the Keyboard-related code here: LinuxIO-Morphic-KeyboardMorph.st