Maliit Keyboard now with word prediction and error correction (video)

This is a feature that I personally wanted for a long time. Interrupted by FOSDEM, some Wayland research and many other things, I finally managed to get word prediction and error correction beyond prototype quality. The video shows just how amazingly good the Presage word prediction can be, even without extensive training (in fact, for the video we used the minimal language model training that comes with a regular Presage installation). The second part of the video shows how combining Presage with a spellchecker such as Hunspell further improves the provided word candidates.

Presage uses a very scalable approach called text n-grams. There is a lot of research in that area, but language models of contemporary language usage are either well guarded or cannot be freely distributed. Luckily, Presage comes with training tools such as text2ngram. Users can feed arbitrary language corpora to it, though one should be careful to perhaps not mix different languages too much.

Matteo Vescovi, the author of Presage (formerly known as Soothsayer), started the work as part of his master thesis a couple of years ago. The heart of Presage are the different predictors. They can be queried in parallel and the result lists are merged, using probability analysis.

Presage certainly has a lot of potential. It comes with an easy to use C++ API but also provides bindings for C and Python. In fact, it even provides a D-Bus API, which would make it possible to run it as a system service. The user could then benefit from (and train!) the same language models from different applications.

Hunspell probably doesn’t need much introduction. It is used in many Linux desktops. It’s a fine library and comes with many dictionaries. However, one should be aware that Hunspell itself cannot provide word prediction, which is why it wasn’t enough for mobile text input. As a fallback for Presage, it works very well though.

Of course there are a couple of things we could do from here. As Jon mentioned in the video, the virtual keyboard’s word ribbon UI could host word suggestions from other applications, such as the Google search in the browser. For Unity’s dasher input or Gnome Shell’s search, the application names could be shown instead. Or we could hook it up to Bash completion.

PS: Anyone up to package Presage for Fremantle or Nemo? It could be pretty interesting to see the next release of Maliit running on the N900.

Maliit as your remote control (video)

There is seemingly a trend (in ideas) to use your mobile device as a remote control for stationary devices such as your TV.

We thought that experimental support for remote text input should be easy to add to Maliit, as it already comes with a server-client architecture. As soon as client and server can run on different host machines, we’d have network transparency and it should basically just work. Luckily, we use D-Bus as our IPC, and it turns out that D-Bus understands remote host addresses. If you check the README in maliit-framework you will recognize that – with the latest release (0.90.0) – it only takes an additional environment variable, MALIIT_SERVER_ADDRESS, to connect a client (that is, an application that uses the Maliit input context) to the input method server. The scary part is perhaps that one has to disable D-Bus authentication. Therefore it's a good idea to only use this inside a trusted network.

Jon created a video which demonstrates the new feature in a better way, using a laptop that is connected to a projector as a TV replacement. The great thing about this feature is that it requires no changes to the applications, nor to the Maliit input method plug-ins, it’s the framework that handles this internally (or not so internally, now that Qt 5 turned our D-Bus API into a public one).

However, to make Maliit truly useful as a remote control, at least two additional features are required:

• Have a proxy text editor widget on mobile device that allows text interactions such as copy and paste, cursor positioning and rich text formatting.
• Emulate and transmit mouse or touch events, possibly through a dedicated touch area on the touch screen of the mobile device.

The second feature requires an extension to the Maliit protocol, whereas the first is already filed as bug MALIIT#84. After that, I could use my N9 (or my N900, as Maliit also runs there) to remote control my Gnome or Ubuntu Unity desktop, which would work perfectly for my movie watching habits. Kudos goes to Krzesimir and Jon for their work on this feature.

Miniature 0.5 "London 1851" released

From the release notes: "Miniature now supports different languages thanks to a determined community of translators. Thank you for your effort! This is why we are dedicating this release to the first international chess tournament, celebrated in London on 1851.

Miniature 0.5 is being released for MeeGo Harmattan (Nokia N9 & N950) and Maemo (Nokia N900). Thanks to everybody involved in the initial Maemo attempts and the experimental version that was made available after the Miniature 0.4 release."

We also improved usability, compared to the previous release, but there's still a ton of work left.

A bit of history

I started working on Miniature – a chess client for freechess.org – in November 2009, after reading the Call for Contributors. Even though we had a pretty cool P2P feature (based on Telepathy and developed mostly by Dariusz Mikulski), it never quite reached the original goal: playing chess online. Back then I was learning how to create UI's with Qt Graphics View, which was all the rage at the time. Well, we now know that writing real UI's with that technology is a major PITA, but for my pet project, it was just too much. I got lost in the struggle.

For the next 18 months, Miniature was basically dead. Another failed project that started so promising. Quim did not want to give up though. After the N9 announcement, he launched a second Call for Contributors.

Perhaps I responded to his mail because I was embarrased at the idea of people wasting time trying to salvage the working parts of Miniature; there simply wasn't much to salvage! So I started again, this time with a very clear goal: online chess, and online chess only. Let others create the actual UI and whatnot. Focusing on one prominent feature and not having to worry about the UI worked well for me, even though I had to iterate over some architecture ideas until I felt comfortable. Quim in the meantime started to prototype the UI with QML. It was impressive to see his results, a level of polish I could have never achieved with my Qt Graphics View approach. At some point the backend was good enough to be sewn together with the frontend and suddenly we had achieved where I failed before: A touch enabled chess client for the N9 that can play chess online.

Having my own useful application available on the N9, published through OVI store, means a lot to me. I hope others will enjoy Miniature as much as we enjoyed re-creating it the second time around.

Writing QML-based input methods for Maliit

This week I pushed a simple QML-based virtual keyboard to our MeeGo Keyboard repository. It's functional, but don't expect too much. This functionality will most likely never arrive in MeeGo 1.2, sorry folks.

Getting Started

Let's continue with the good parts instead: The required C++ code was kept to an absolute minimum and in fact it's only needed to wrap the QML resources into a Qt/C++ plugin that can be loaded by our framework. If you want to write your own QML-based input method plugin for Maliit, then this is the required interface:

class MyQmlBasedInputMethodPlugin
    : public QObject, public MInputMethodQuickPlugin
{
    Q_OBJECT
    Q_INTERFACES(MInputMethodQuickPlugin MInputMethodPlugin)

public:
    MyQmlBasedInputMethodPlugin();

    //! \reimpl
    virtual QString name() const;
    virtual QString qmlFileName() const;
    //! \reimpl_end
};

The plugin's name is used for a specific GConf key. Once set to that name, the framework will load this plugin as the currently active one:

$ gconftool-2 -s /meegotouch/inputmethods/plugins/onscreen -t string "WhateverNameYouCameUpWith"

qmlFileName returns the (absolute) file name (or URL) to your main QML file. The framework will set up the required QML environment and try to load this file.

QML components can interface with the application through a set of properties and slots, tied to a root context named MInputMethodQuick. It exposes the following read-only Qt properties:

• int screenWidth,
• int screenHeight,
• int appOrientation (0, 90, 180, 270 with 0 being the typical landscape orientation and with 270 being the typical portrait orientation).

Additionally, QML components can invoke the following Qt slots:

• setInputMethodArea(const QRect &area)
• sendPreedit(const QString &text)
• sendCommit(const QString &text)
• pluginSwitchRequired(int switchDirection)

Please refer to the header file of MInputMethodQuick to access the full documentation (sorry, not available in MeeGo API docs).

Also, big thanks to Kimmo Surakka and Miika Silfverberg for all their help with this side project.

Self Compositing

Any QML plugin using MInputMethodQuickPlugin will support self compositing out of the box. That is, instead of letting the system compositor blend the application window and virtual keyboard window together into one, the virtual keyboard bypasses the system compositor and redirects the application's window to itself. This usually improves raw FPS but also cuts down on latency. Try running the MeeGo Input Methods service like so:

$ meego-im-uiserver -use-self-composition

This should result in a noticable speedup for your QML plugin. Self compositing is tricky, as it requires a well-behaving window manager (and believe it or not, mcompositor is the only one available getting it nearly 100% right). Usually an additional parameter combination of -manual-redirection, -bypass-wm-hint and -software will do the trick. For MeeGo 1.3, we want to activate self compositing by default, so it's good to know that your QML-based plugin will support this useful feature for free.

A QML-based virtual keyboard for Fremantle

As part of the exercise I also packaged MeeGo Keyboard Quick for Fremantle - it's available from Maemo's extras-devel repository. Be warned though, this is a rough tech demo and as such, fairly useless. A nice advantage of it is that it allowed me to package a MeeGo Input Methods version for Fremantle that is completely free of any MTF/libmeegotouch components.

How to testdrive:

• Install meego-keyboard-quick (after you convinced yourself that you really know what you do - activating extras-devel can brick your N900!),
• Install meego-im-demos

• Open a terminal, start the MeeGo Input Methods service:

  $ meego-im-uiserver -bypass-wm-hint -software -graphicssystem raster

   

• Open a second terminal, to run the meego-im-demos-plainqt application:

  $ export QT_IM_MODULE=MInputContext$ meego-im-demo-plainqt -fullscreen -software -graphicssystem raster

   

Instead of the test application, one should be able to use it in any other Qt application, as long as the MeeGo Input Methods service is running and the QT_IM_MODULE environment variable is set correctly.

To fight wrong expectations: I won't have the time to polish or to maintain this port, and from my first tests it would need undefined amounts of integration work to get it to run properly in Fremantle, starting with proper GTK+/Hildon support. If you are, however, interested in furthering this port, then please don't hesitate to contact us.

Miniature goes Telepathy

Thanks to the work of Dariusz and Alban we now have P2P via Telepathy for Miniature.

Maybe it's worth to notice that we use Maemo's contact chooser here, and that it blends perfectly with our Qt application (we still have some crashes on our side though). Nice job from the Fremantle team! there!

Logging facility for Miniature

On Maemo 5, log output from your app isn't always accessible to the user. This has created problems for Miniature bug reports (see bug #8124). To solve this, I created a "Game Log" screen which allows to filter the messages (by a given log level, I might want to allow combinations, too). It also has a nice fat "Copy all" button, so that the log output can be quickly attached to a bug report.

Now I "only" need to add useful log information =D

Thanks to Openismus for letting me work on this.

Testdriving the UI Extensions for Mobile on Maemo 5

Reading about Nokia's UI Extensions for Mobile (sources available) I wanted to quickly try it myself. So I looked at the provided examples, and this video is the result of what I came up with (well, of course it is FOSDEM-induced). The provided API allowed me to easily apply my previous Qt knowledge, which is a nice touch. Sources for the example app in the video can be found here and here. Happy hacking!

Update: The UI Extensions for Mobile will not compile on 64bit architectures.

First Miniature .debs sighted!

Thanks to Mathias, Miniature is now suitable for debian packaging. He also uploaded a first test balloon which at least allows you to move pieces around. Feedback is of course more than welcome.

Personally I wish the board's cell size could have been made bigger, but 60x60 pixels is already the upper limit. It is now up to us - the Miniature team - to come up with clever techniques to make the board handling as finger-friendly as possible.

Qt 4.6 for Maemo: It works!

Today at work, David King kindly informed me that there was some new Qt package in extras-devel. This could only mean one thing - I immediately fired up my scratchbox environment and installed the packages, trying to confirm that this new version would run with Qlom. And in fact, it was surprisingly painless. Thanks to autotroll, a simple QT_PATH env variable did all the magic, hooray!

Both of us were impressed with the UI improvements. It's certainly a big step forward regarding the Hildonisation of Qt on Maemo5. The application menues look correct now. Button sizes, colors, animations, etc - it all comes together nicely, finally.

There are still some widgets that need more work, but for a tech preview this is a pleasant surprise.

On another note, the timing for the Miniature project could not have been much better. We immediately switched to Qt 4.6, and it even runs on the N900. It feels good to know that we can stop using hacks and that we can start to do (most) things properly, staying as cross-platform as possible. Needless to say, Quim was happy, too.

Miniature - it moves!

How it begun

When I read Quim's thread about the idea for a better Maemo chess app I knew I wanted to join the project. To me, it's all about the device and the sparkling Hildon UX. I really want a good chess app, for myself! I want to play chess online, everywhere! And I want to analyze games as (OK, maybe after =p) they happen. No more "I'll check this position later" (we all know this rarely happens).

So I finally started last friday. At this point, Quim and Andreas had already created a beautiful, content-rich wiki page. It took a while for me to digest it all, and I added information where appropiate.

Kick-starting the development

Andreas had registered a garage project, but we eventually decided to use gitorious for our repository. Gitorious' UI definitely improved over the recent months, and the possibility to have teams working on a single project - also known as not-so-extreme-dvcs-development - makes gitorious a better choice than github, at the moment.

Saturday night (what better things to do than coding some Qt - my soul will be forever lost) I had a first running example (see screenshot). Currently, Miniature can move between positions, using next/prev menu navigation (we don't need this functionality per se, but it's perhaps a good demonstration that the simple approach I took works).

So no matter the toolkit, no matter the outdated packages or the endless confusion I had with the various Qt repos at gitorious - this project is really fun! Hopefully we get to make a 0.1 release soon.