Maliit Status Update

We’ve been busy! The result is our latest release, Maliit 0.99, with a massive amount of changes. While the framework has mostly seen code cleanups (:~/source/maliit/fw$ git diff 0.94.0..0.99.0 --stat: 389 files changed, 3612 insertions(+), 30081 deletions(-)), we’ve been adding tons of features to the reference plugins (:~/source/maliit/plugins$ git diff 0.94.0..0.99.0 --stat: 324 files changed, 32499 insertions(+), 12799 deletions(-)).

After (finally!) getting official Debian and Ubuntu packages, the next big step is our Maliit 1.0 release, which we expect sometime later this year.

Maliit 0.99 works with applications using GTK+2/3 or Qt4/5, though Qt4 still works best. Maliit’s framework and reference plugins strictly require Qt5 as build dependency, allowing us to get rid of Qt4 and Xlib legacy code. Maliit can run on Wayland but at this point only EFL applications support Wayland input methods. For Qt5 applications wanting to use Maliit together with Wayland, a Qt5 platform plugin for Wayland input methods would be needed.

Who is using Maliit?

The best Maliit showcase is still the Nokia N9. No other device got anywhere near to the demonstrated input method integration level. The haptic feedback of its virtual keyboard remains unchallenged. I think it’s a major success for our old Harmattan text input method team that we see Maliit being used in other projects. It was hard work to get to this point and yet it feels as if this was only the beginning of Maliit as an open-source project.

We have the friendly folks beind the OLPC project who were one of the first to go with Maliit. Few of us will probably make the new XO their primary device and therefore, won’t see the results of the collaboration. But from the very beginning, they worked with us in the open, provided official Fedora packages and helped us test-driving the then-new styling engine of Maliit Keyboard. The XO is one of the use-cases where Maliit has to run without a compositing window manager, which made them an early adopter of our windowed mode.

Plasma Active is another project that — for its size — invested significant development time improving and testing Maliit’s reference plugins. They’ve been (and hopefully remain!) a huge help whenever we faced problems with QML. I wish we could find a sponsor however to work on improving their input method integration, which is the “invisible” and challenging part that easily gets forgotten.

Ubuntu Touch is choosing Maliit as well. Canonical sponsored significant work in Maliit Keyboard, especially in terms of Qt5 support and replacing QGraphicsView with QtQuick2.

SailfishOS is happy user of Maliit, too. They are the main consumers of our 0.80 branches that still offer full support for Qt4 and X11. We always considered 0.80 our latest “stable” release branch, so it’s good to know that at least someone is also using it.

OpenEmbedded has integrated Maliit into their meta layer. This makes it easier to build custom Linux distros for embedded with Maliit on board. My understanding of OpenEmbedded is still limited at this point so I cannot explain all the real benefits this collaboration provides. Luckily there’s the always friendly Samuel Stritzel who’s following our official mailing list, waiting to answer all your OpenEmbedded related questions.

Gnome3. Just kidding. But once Gnome3 properly integrates with Wayland input methods, we can just plug in Maliit, too. Of course to be of any interest for Gnome3, we need to demonstrate working CJK input methods too. Luckily, we got the opportunity to just do that: IBus working with Wayland input methods. If things pan out I’ll be talking about this at GNOME.Asia.

I would like to get the chance to improve input method support for Firefox and Chrome/Chromium. There is one nasty bug around password fields that’s been quite a blocker for us. Some day, we’ll hopefully be able to convince the browser guys to just “do the right thing”.

And then there’s yours project of course. You just didn’t tell me about it yet ;–)

Architecture

There is an influx of new developers showing interest in Maliit which is why I feel that reiterating the basic architecture behind Maliit doesn’t hurt.

Maliit is split into three parts: framework, reference plugins and input method modules (GTK+, Qt4, Qt5).

The framework provides maliit-server, a process that can load reference plugins (such as VKBs) and allows applications to connect to it through input method modules. The input method modules are UI toolkit specific and are loaded dynamically by the applications. For each UI toolkit Maliit wishes to support, a separate input method module is required.

Input method modules and framework are designed as client-server architecture, with applications behaving as clients. The default IPC is QDBus but it can be replaced by other mechanisms, for instance the Wayland IPC.

In a follow-up blog post I will provide an outlook of where Maliit goes next and which parts of it we might have to rethink, now that the leviathan of Linux DEs is quickly moving towards Wayland.

People

Those are the people who have contributed to Maliit, but you wouldn’t know if you only checked commit messages:

• Łukasz Zemczak, Thomas Möricke and Kevin Wright: You guys have been great. Would work together again.
• Iain Lane, Michał Zajac, Peter Robinson: Without you, we still wouldn’t have proper Debian/(K)Ubuntu/Fedora packaging. Thanks!
• Samuel Stritzel: Thanks for bringing Maliit to OpenEmbedded! Hopefully we’ll see more use of your work soon.
• Simon Schampijer, Daniel Drake, Gary Martin, Jennifer Amaya and Martin Langhoff: Thanks for the opportunity to work on OLPC. I know most of us are not in the target audience which makes it hard for you guys to motivate others to help you. Your work is important, please never give up.
• Pekka Vuorela: I am just glad to have you back in the team. Maliit just isn’t the same without you.
• Aaron Seigo, Marco Martin: Thanks for all your enthusiasm about Maliit and Plasma Active. Your motivation is infectious.
• Jan Arne Petersen, Krzesimir Nowak: I am impressed by your motivation and dedication that you still have for input methods, even though none of us ever planned to become an “input method expert”.

Get ready for Qt Contributor Summit!

With only roughly over five weeks to go, QtCS 2012 is approaching fast on the event horizon (pun absolutely intended). During the TiZen Developer Conference, Quim Gil and I, with help from Robin Burchell, found time to work on a draft, the result being visible in the QtDev wiki. Don’t be scared by the amount of pre-scheduled sessions, we will try to keep it as unconference-y as possible. However this allows us to properly prepare selected sessions.

Please keep in mind that QtCS 2012 is invite-only.

Session presenters, prepare!

If you are a maintainer of a Qt Essentials module, then you will have to drive one of the pre-scheduled sessions.

It might be a good idea to start writing a report about what happened in your area over the last couple of months, and where you plan to take your module for the Qt 5.1 release. Attendees of your session will probably expect you to talk about Qt 5 migration and how it will affect code using your module. On top of that, the most urgent bugs might get discussed, along with new feature requests, so please be prepared.

If you find a topic that isn’t showing up in the program yet, consider to get in touch with other interested folks (the Qt development mailing list or #qt IRC channel are good places to ask). With sufficient feedback, you should have everything it needs to drive the session yourself!

Plenary sessions

The first day will start with the State of the Union session about Qt 5 and Qt 5.1, summarizing where we are and where do we want to go with the Qt project.

The second day is about betting on Qt Quick, covering topics such as theming, platform integration and cross-platform. We want to invite the maintainers and a few vendors/users/contributors to expose their thoughts and needs with the hope of bringing a common understanding.

For the last day, we have a special surprise, as the topic will be HTML5 & the web. I am aware that this will raise a lot of controversy among the community, but it is an area where we need to find answers. With all the latest interest in web technologies, the answer can no longer be “We don’t do web”. That’s what killed the dinosaurs, remember? Then again, Qt already has better answers then that, so it’s about time for a fierce but honest panel discussion, including the Qt WebKit maintainers and other users/contributors driving this area.

Hack'n'Tell

Of course we also want to you to feel entertained during the summit, which is why we want to try out Hack'n'Tell sessions.

The rules are simple: Show us a cool hack involving Qt. You got 5 minutes. No slides.

I realize that preparing cool hacks takes some time, which is the original intent for this blog post: to announce the Call for Cute Hacks. For the first day at least, it would be great to have a couple of presenters before the summit starts. For the other days, I kind of hope that the summit itself will spark great ideas and demos that need to be shown immediately.

I hope I got you excited a bit, so let’s make QtCS 2012 an event to remember!

Designing an input method system for Wayland

As mentioned before, Jon and I met with Kristian Høgsberg and Peter Hutterer during FOSDEM 2012. The discussion quickly revolved around Wayland and input methods. Kristian and I basically picked up from where we left after the Qt Contributor Summit last year in Berlin.

Right now, there is no explicit input method support in Wayland, and it’s the responsibility of the applications to provide such. But we came to the conclusion that moving it into Wayland provides several benefits.

That is why I spent some time during February to write down an initial design proposal, using the two years of experience from working in the application framework team for the Nokia N9. It has gotten longer than expected, but hopefully it is structured well enough to still be accessible.

From the introduction:

Currently, Wayland has no explicit text input method support and relies entirely on toolkit-specific or platform-specific solutions. This is awkward for platforms that support multiple toolkits. The resulting fragmentation and re-implementation would make it hard to attract developers of high-quality text input methods. These developers would generally prefer a unified architecture that lets them ignore application-specific or platform-specific details. This would let them simultaneously target all platforms that support Wayland.

We believe this goal can be reached by adding two new Wayland protocols, of which only one requires adaptation by toolkit providers.

The idea is not new, XIM has been around since the dawn of time. Because of the protocol’s daunting complexity, it apparently never became popular among application toolkit providers. That is why I decided to split up the envisioned Wayland input method protocol up into two:

• The Wayland text protocol, to be used between the Wayland compositor and its clients. It tries to keep input method knowledge to a minimum and therefore can be used for any kind of (hardware) input that results in text input.
• The Wayland input method protocol, to be used between the Wayland compositor and the input method system, which can either run as a stand-alone process or inside the compositor process.

In the proposal, I explain my ideal solution where applications and input method system are completely separated from each other and where the compositor acts as a mediator between them. I continue with alternative solutions that are feasible but have several drawbacks. Towards the end, there is a comparison matrix that features all discussed solutions.

There is an important aspect though: The alternative solutions provide an iterative implementation approach to reach the ideal solution. The obvious idea here is that already in early stages, we get testable prototypes. The solutions with a compositor-hosted input method system for instance allow others to think about window management policies that take input methods into account, whereas I could continue my iterations towards the ideal solution.

Anyhow, the next steps are now to discuss the proposal in a wider public (feedback from developers of exotic input methods especially welcome!), possibly also at upcoming conferences, and to come up with a prototype implementation so that we can iron out protocol bugs.

I would probably target Qt 5 as the first application toolkit to support the Wayland text protocol, followed by GTK+ and perhaps Enlightment. One also needs to take the popular desktop shells into account, such as Gnome Shell, Unity and Plasma Active.

Maliit could fill the role of a reference input method system, but we should have at least one more implementation, at some stage (IBus comes to mind, naturally).

Looks as if this could keep me busy for the better part of 2012, but it would be great if we could get at least the Wayland text protocol into Wayland 1.0. It’ll require some luck though.

Maliit as your remote control (video)

There is seemingly a trend (in ideas) to use your mobile device as a remote control for stationary devices such as your TV.

We thought that experimental support for remote text input should be easy to add to Maliit, as it already comes with a server-client architecture. As soon as client and server can run on different host machines, we’d have network transparency and it should basically just work. Luckily, we use D-Bus as our IPC, and it turns out that D-Bus understands remote host addresses. If you check the README in maliit-framework you will recognize that – with the latest release (0.90.0) – it only takes an additional environment variable, MALIIT_SERVER_ADDRESS, to connect a client (that is, an application that uses the Maliit input context) to the input method server. The scary part is perhaps that one has to disable D-Bus authentication. Therefore it's a good idea to only use this inside a trusted network.

Jon created a video which demonstrates the new feature in a better way, using a laptop that is connected to a projector as a TV replacement. The great thing about this feature is that it requires no changes to the applications, nor to the Maliit input method plug-ins, it’s the framework that handles this internally (or not so internally, now that Qt 5 turned our D-Bus API into a public one).

However, to make Maliit truly useful as a remote control, at least two additional features are required:

• Have a proxy text editor widget on mobile device that allows text interactions such as copy and paste, cursor positioning and rich text formatting.
• Emulate and transmit mouse or touch events, possibly through a dedicated touch area on the touch screen of the mobile device.

The second feature requires an extension to the Maliit protocol, whereas the first is already filed as bug MALIIT#84. After that, I could use my N9 (or my N900, as Maliit also runs there) to remote control my Gnome or Ubuntu Unity desktop, which would work perfectly for my movie watching habits. Kudos goes to Krzesimir and Jon for their work on this feature.

How we enable others to write 3rd party plugins with Maliit

We finally published a video about Maliit - an input method framework including a virtual keyboard - and 3rd party plugins. Kudos goes to Jon for making time for that.

This video highlights one of Maliit's key features: pluggable input methods which come with their very own user interfaces. The Chinese input methods show how Maliit offers support for composed characters. The video is proof that 3rd party development for Maliit (open-source and proprietary) is not only possible but also happening.

maliit.org states that "it should be easy to customize existing input methods or develop powerful new input methods, whether for profit, research or fun", we actually mean it.

The harder question is of course how to motivate others to actually get started on input method development with Maliit. For that, we have a multipronged strategy:

1. Provide sufficiently polished reference plugins that can show off Maliit capabilities but also serve as inspiration for new plugins (hence the BSD license for reference plugins). Our reference plugins are currently using Qt/C++ (Maliit Keyboard) and QML (Nemo Keyboard). We also have PySide support, but no one contributed a reference plugin yet. This gives choice to interested input method developers, and we think that's important. The reference plugins serve another role when it comes to designing new API: They become our testbed, allowing us to verify our API proposals.

2. Ship Maliit with a bunch of example plugins and example applications. None of them try to be complete. They are all self-contained though and usually show one feature at a time. This can be tedious to maintain, but we believe that examples need to stay small and focused, otherwise developers won't look at them.

3. Documentation that is easy to consume. Our documentation is not as concise and clear as we'd like it to be, but it's slowly improving. We also experiment with videos that can serve as an introduction to more in-depth (text) documentation.

4. Packages for most common Linux distributions. This one seems obvious, but sadly, it's quite a lot of work for us to keep up with it (and we already use automated services such as Launchpad and OpenSuse Build Service). In the hope to attract dedicated packagers we wrote down some packaging guidelines

5. An architecture that had 3rd party plugins and multiple toolkit support in mind from the start. The plugin developer facing API needs to be easy to use and clearly documented. This will be the focus of the upcoming 0.9x series.

We will demo Maliit @ FOSDEM 2012, hope to see you there!

Better GTK+ support in Maliit

So far, using Maliit's virtual keyboard in GTK+ applications required fetching and compiling a GTK+ input method brigde yourself. Not any more. With the latest release, GTK+ applications should just work out of the box, thanks to Jon's integration efforts. Right at the same time, Łukasz was looking into using Maliit together with GTK+ applications on his Ubuntu desktop. He did a great job testing Jon's improvement and also contributed patches to properly update GTK+'s input method module cache. When compared to the Qt support, the gap in terms of supported features is quite large. We would like to further improve the GTK+ support and contributions are certainly welcome.

Real users, real feedback

We released Maliit 0.80.7 on Friday. Over these last days, I am doubly proud about our project. Not only did the N9's virtual keyboard get astonishing reviews across the board, but what's even better: We managed to keep this software open-source. In our communities, there will always be those who focus too much on technical aspects. I remember the technical struggles we had even within MeeGo! But now we get feedback from real users who couldn't care less about what Qt or MeeGo Touch is, and to be honest, that's a refreshing change.

Being here at Qt's Developer Days 2011, it feels great to get such feedback directly, from first-time users of the Nokia N9. Especially the fine haptic feedback and the keyboard's accuracy gets noticed.

I also had the possibility to see a Japanese input method — running on the N9 and powered by Maliit. Seeing how well this plugin already integrates with the platform, I feel that our architecture yet again has been justified. I am looking forward to see more Maliit plugins, and more platforms using Maliit!

Using C++ enums in QML

When mapping Qt/C++ API's to QML, or, to put it more precisely, making a Qt/C++ API available to QML, road bumps are to be expected. One such bump is the mapping of C++ enums.

If you happen to create enums inside a QObject, then it will be exported to QML via the Q_ENUMS helper:

SomeEnumsWrapper
    : public QObject
{
    Q_OBJECT
    Q_ENUMS(SomeState)
public:
    enum SomeState {
        BeginState,        // Remember that in QML, enum values must start
        IntermediateState, // with a capital letter!
        EndState
    };
};

You will still need to declare this class as an abstract type for QML to be able to use enums from it (put in your main function for example):

qmlRegisterUncreatableType<SomeEnumsWrapper>("com.mydomain.myproject", 1, 0,
                                             "SomeEnums", "This exports SomeState enums to QML");

Now in QML, the enums can be accessed as '''SomeEnums.BeginState'''. Note how the enum is accessed through the exported type name, not an instance.

But what if you've put your enums into a dedicated C++ namespace? Then the same mechanism can be used. Let's start with the namespace:

namespace SomeEnums {
    enum SomeState {
        BeginState,
        IntermediateState,
        EndState
    };
}

We can re-use the idea of wrapping enums in a QObject type, with one tiny change:

SomeEnumsWrapper
    : public QObject
{
    Q_OBJECT
    Q_ENUMS(SomeState)
public:
    enum SomeState {
        BeginState = SomeEnums::BeginState, // Keeps enum values in sync!
        IntermediateState = SomeEnums::IntermediateState,
        EndState = SomeEnums::EndState
    };
};

The process of forwarding all your enums through this mechanism can be tedious, but being able to use enums properly in QML properly will improve the readability and maintainability of your QML code.

For a fully working example check Maliit's Qt Quick support

Brave New World

Today, we managed to get out the first release of Maliit as an independent project. As far as possible without API break, we started to use our own Maliit naming convention for installation paths, library names, etc.

During that cleanup, Jan Arne did a tremendous job improving our build infrastructure. Instead of Qt's .prf files, Maliit now uses pkg-config everywhere. Plugin developers don't have to think about install paths - they can simply read it from the .pc files, for instance:

pkg-config --variable pluginsdir maliit-plugins-0.80

The version number at the end indicates another nice improvement: versioned libraries and plugin interfaces, which means several Maliit versions can be installed in parallel (although you never want to run more than one server per session).

To install the Maliit framework into a custom directory (say, $HOME/install), simply use:

qmake -r M_IM_PREFIX=$HOME/install

Afterwards, set your PKG_CONFIG_PATH to $HOME/install/lib/pkgconfig and compile the Maliit plugins. Quite honestly, it has never been easier to develop input methods for Maliit than with this release.

Input methods and Wayland in Qt5

I was attending the Qt Contributors' Summit 2011. During the key note, it was promised that everything is up for discussion so I took my chance to discuss about improving input methods support for Qt5.

After some initial discussions with Kristian Høgsberg (Wayland, of course) and Jørgen Lind (who works on Qt Lighhouse), I also addressed Wayland. It became clear that one needs some kind of input method interface directly in Wayland. Kristian immediately started with a small prototype, in order to explain better how a Wayland compositor can provide a much better window management policy than what we currently have with Maliit and X11.

I think the session itself was really successful. I was surprised at the strong interest in this topic.

It became apparent that we should do something about Qt's input context API. For instance, add more input methods hints, come up with a better interface that describes the focus widget, preedit handling, orientation support and so on.

Now we only need to agree on how to make it happen :-)