Thoughts 'n Stuff

Painting Strategy in MeeGo's Virtual Keyboard

I've never been happy with the conclusion in the influential blog post "Qt Graphics and Performance — The Cost of Convenience" by Gunnar Sletta: If you want performance, downgrade QGraphicsView to a mere canvas with a single QGraphicsItem. It defeats the whole purpose of decomposing the problem into many small QGraphicsItems and is therefore entirely counter-intuitive. One might be quick to ask what Qt Graphics View is good for (and one might find the answer here), but instead I would like to present an alternate solution to Gunnar's which reaches the same performance but embraces the very nature of Qt Graphics View.

Flashback

Gunnar took the example of a virtual keyboard (which doesn't really do anything, it just sits there and paints itself). And as it happens, I've been working on a real virtual keyboard. At one point, our painting strategy was exactly following the advice of Gunnar, but once new requirements piled up, it became increasingly harder to deal with that painting strategy.

The new requirements were custom key sizes, flexible layouts, custom font sizes for each label, custom font colours, custom key background and custom key labels. And all that dynamically during runtime please.

In MeeGo 1.3, applications will be able to make us of that. For example, when writing an email, the application can temporarily turn the enter key on the virtual keyboard into a "Send" button. Should the user enter an invalid email address, the application might choose to disable the "Send" button until the mistake has been fixed.

Reinventing QGraphicsItems - or not?

As the required logic for keys becomes more complex, so does the data structure representing keys. The paint method we used in the single keyboard item was about to become a real mess. Especially after I added geometry to keys, I knew that I was, in fact, reimplementing QGraphicsItems. But I could not reuse them because of performance reasons! Or was it possible that Gunnar's blog post pointed into the wrong direction?

So I thought about my dilemma and figured out that the actual constraint is rather the amount of paint event calls per frame, and eventually, how much screen area needs updating. With the single QGraphicsItem approach, every change triggers a scheduled repaint for the whole keyboard - even if it's only the label position in one key.

I then realized that only pressed or customized keys would need to be actively painted, and there we had it: An alternate painting strategy that blits the idle keyboard view into a single QGraphicsItem but lets active keys paint themselves as overlays. The rest of our team was sceptical about the idea, so Viacheslav Sobolev - a colleague of mine during my 11 months in Helsinki - had to first improve the performance benchmarks before the team would accept the fact that indeed we managed to keep the same performance, and even improve it for Qt's raster engine.

The Strategy

Even if the general idea of overlay items is very simple, I summarized the following steps that one needs to take care of:

• Aggressive caching - only update when absolutely needed. Analyse what changes and encapsulate change into QGraphicsItems.
• Keys keep changing. Therefore, model keys as QGraphicsItems. Qt Graphics View will not propagate paint events to hidden items, so we hide all keys by default.
• Allow the keyboard item to query a pixmap version from each key. Therefore, the keys act as painting delegates for the keyboard. However, they can also actively paint themselves when activated.
• Paint the idle state of the keyboard into a single QGraphicsItem and avoid calling update on it. Use QGraphicsItem::DeviceCoordinateCache as its caching strategy. The idle state itself can be described as the keyboard's background together will all keys in idle state (that is, neither pressed nor customized).
• Activate keys if pressed or customized, which lets themselves paint as overlays over the static keyboard item.
• Caveat#1: Make sure your implementation can paint the idle state of the keyboard at all times. There's no need to waste more memory with manual caching here - just make sure you can query pixmaps from a key's idle state (even if the key is not idle at that particular moment). The documentation regarding QGraphicsItem caching modes is incorrect.
• Caveat#2: Handle mouse and touch events on the parent keyboard item and propagate state changes to the keys. Most of the keys will usually be inactive, so they cannot receive such events directly.

As a bonus, you can find the basics of the strategy in a modified benchmark of the original blog post. I added two new options "-overlayitems" and "-fullscreen". The latter is useful for N900 where it helps to get rid of system compositor noise.