Maliit Status Update

We’ve been busy! The result is our latest release, Maliit 0.99, with a massive amount of changes. While the framework has mostly seen code cleanups (:~/source/maliit/fw$ git diff 0.94.0..0.99.0 --stat: 389 files changed, 3612 insertions(+), 30081 deletions(-)), we’ve been adding tons of features to the reference plugins (:~/source/maliit/plugins$ git diff 0.94.0..0.99.0 --stat: 324 files changed, 32499 insertions(+), 12799 deletions(-)).

After (finally!) getting official Debian and Ubuntu packages, the next big step is our Maliit 1.0 release, which we expect sometime later this year.

Maliit 0.99 works with applications using GTK+2/3 or Qt4/5, though Qt4 still works best. Maliit’s framework and reference plugins strictly require Qt5 as build dependency, allowing us to get rid of Qt4 and Xlib legacy code. Maliit can run on Wayland but at this point only EFL applications support Wayland input methods. For Qt5 applications wanting to use Maliit together with Wayland, a Qt5 platform plugin for Wayland input methods would be needed.

Who is using Maliit?

The best Maliit showcase is still the Nokia N9. No other device got anywhere near to the demonstrated input method integration level. The haptic feedback of its virtual keyboard remains unchallenged. I think it’s a major success for our old Harmattan text input method team that we see Maliit being used in other projects. It was hard work to get to this point and yet it feels as if this was only the beginning of Maliit as an open-source project.

We have the friendly folks beind the OLPC project who were one of the first to go with Maliit. Few of us will probably make the new XO their primary device and therefore, won’t see the results of the collaboration. But from the very beginning, they worked with us in the open, provided official Fedora packages and helped us test-driving the then-new styling engine of Maliit Keyboard. The XO is one of the use-cases where Maliit has to run without a compositing window manager, which made them an early adopter of our windowed mode.

Plasma Active is another project that — for its size — invested significant development time improving and testing Maliit’s reference plugins. They’ve been (and hopefully remain!) a huge help whenever we faced problems with QML. I wish we could find a sponsor however to work on improving their input method integration, which is the “invisible” and challenging part that easily gets forgotten.

Ubuntu Touch is choosing Maliit as well. Canonical sponsored significant work in Maliit Keyboard, especially in terms of Qt5 support and replacing QGraphicsView with QtQuick2.

SailfishOS is happy user of Maliit, too. They are the main consumers of our 0.80 branches that still offer full support for Qt4 and X11. We always considered 0.80 our latest “stable” release branch, so it’s good to know that at least someone is also using it.

OpenEmbedded has integrated Maliit into their meta layer. This makes it easier to build custom Linux distros for embedded with Maliit on board. My understanding of OpenEmbedded is still limited at this point so I cannot explain all the real benefits this collaboration provides. Luckily there’s the always friendly Samuel Stritzel who’s following our official mailing list, waiting to answer all your OpenEmbedded related questions.

Gnome3. Just kidding. But once Gnome3 properly integrates with Wayland input methods, we can just plug in Maliit, too. Of course to be of any interest for Gnome3, we need to demonstrate working CJK input methods too. Luckily, we got the opportunity to just do that: IBus working with Wayland input methods. If things pan out I’ll be talking about this at GNOME.Asia.

I would like to get the chance to improve input method support for Firefox and Chrome/Chromium. There is one nasty bug around password fields that’s been quite a blocker for us. Some day, we’ll hopefully be able to convince the browser guys to just “do the right thing”.

And then there’s yours project of course. You just didn’t tell me about it yet ;–)

Architecture

There is an influx of new developers showing interest in Maliit which is why I feel that reiterating the basic architecture behind Maliit doesn’t hurt.

Maliit is split into three parts: framework, reference plugins and input method modules (GTK+, Qt4, Qt5).

The framework provides maliit-server, a process that can load reference plugins (such as VKBs) and allows applications to connect to it through input method modules. The input method modules are UI toolkit specific and are loaded dynamically by the applications. For each UI toolkit Maliit wishes to support, a separate input method module is required.

Input method modules and framework are designed as client-server architecture, with applications behaving as clients. The default IPC is QDBus but it can be replaced by other mechanisms, for instance the Wayland IPC.

In a follow-up blog post I will provide an outlook of where Maliit goes next and which parts of it we might have to rethink, now that the leviathan of Linux DEs is quickly moving towards Wayland.

People

Those are the people who have contributed to Maliit, but you wouldn’t know if you only checked commit messages:

• Łukasz Zemczak, Thomas Möricke and Kevin Wright: You guys have been great. Would work together again.
• Iain Lane, Michał Zajac, Peter Robinson: Without you, we still wouldn’t have proper Debian/(K)Ubuntu/Fedora packaging. Thanks!
• Samuel Stritzel: Thanks for bringing Maliit to OpenEmbedded! Hopefully we’ll see more use of your work soon.
• Simon Schampijer, Daniel Drake, Gary Martin, Jennifer Amaya and Martin Langhoff: Thanks for the opportunity to work on OLPC. I know most of us are not in the target audience which makes it hard for you guys to motivate others to help you. Your work is important, please never give up.
• Pekka Vuorela: I am just glad to have you back in the team. Maliit just isn’t the same without you.
• Aaron Seigo, Marco Martin: Thanks for all your enthusiasm about Maliit and Plasma Active. Your motivation is infectious.
• Jan Arne Petersen, Krzesimir Nowak: I am impressed by your motivation and dedication that you still have for input methods, even though none of us ever planned to become an “input method expert”.

Concrete problems and generic solutions

For the official Gnome3 virtual keyboard, one can notice how the space bar is strangely sized, how the regular keys are unpleasantly square and how the enter and backspace keys are way too small. They all use the same graphical asset. For the Gnome3 styled Maliit Keyboad, the letters d-j, x-n, together with the space bar and its neighboring keys, form a neatly aligned center area, with a staircase pattern to each side of the keyboard. If you look in the last row, you will notice that the space bar is just a few pixels too large. This happens because the only non-pixel-perfect elements in the Maliit Keyboard are stretched keys and spacer elements, and I somehow still compute their width incorrectly. From a designer perspective, that “pixel bug” in the layout destroys the immersion of an otherwise good design. Pixel perfectness — whilst having a poor reputation among us developers perhaps — is actually quite a challenge to get right!

Maliit Keyboard is a reimplementation of the MeeGo Keyboard that was shipped on the Nokia N9. It doesn’t share any code with it, but it does reuse the ideas and the XML layout files. The format for the layout files evolved from programmer-friendly to designer-friendly over time, which I think was a good change. Instead of using the complex and pretty generic styling system of libmeegotouch (which was intended to fit the needs of any mobile application), I use a concrete style class that reads a styling profile folder, looks into into the two INI files (one for the main keyboard area, one for the extended keys) and fills a model with style attributes. In the given context of virtual keyboard styling, the concrete solution offers the same flexibility as the generic solution but comes with a fraction of the implementation complexity and remains simple to extend. This is because the concrete solution fits the intended use-case perfectly, whereas a generic solution will always acrue overhead.

The trade-off, of course, is that for a new use-case, I’d probably have to write a new style class and create new style attributes. It is very unlikely, however, that the styling requirements of a fairly specific “application” would fit the the styling requirements of a regular application, so why burden myself with a generic enough abstraction that I will never use? Inversely, why try to make a generic solution fit a specialised problem? Jokingly, that’s precisely what I call Java programming: Instead of solving the problem at hand, write a framework that can solve all similar problems. I have started to grow a strong aversion against that, not because I like reinventing the wheel (I don’t), but because I think that understanding the concepts behind an idea is far more valuable than clinching to an implementation of it.

For the Maliit Keyboard, I took the concepts of the libmeegotouch styling system and mixed my understanding of CSS with the XML layout files. The result feels pretty clean and easy to use, but that’s because I based the implementation on well-understood concepts, not because of the implementation itself. Who knows, if I keep myself busy with implementing virtual keyboards for just a bit longer, I should eventually be able to write a pretty decent book about VKB design patterns ;–)

Getting rejected as GSoC organization

Last week I submitted an application on behalf of the Maliit project to participate in this year’s Google Summer of Code, as a mentoring organization. Today I got the rejection mail in my inbox.

Maybe I am too European to understand this, but what I find disappointing is the lack of any reason given in that e-mail, other than that too many applications were received. I mean, how am I supposed to learn from that? How will I know whether a project like Maliit has any chance at all, or whether I should even try to submit a project next year? Even for the list of accepted organizations, such as KDE or GNOME (of which both are pretty much doing the same stuff, so why do we need both here?) no reasons are given of why they were accepted.

Having had a bad experience with opaque decision making in the MeeGo project, this tastes just as bad. Sure, not giving reasons so that you don’t have to argue about them later on has apparently become a standard procedure, but I am not willing to accept such behaviour. After all, I also spend time in writing the proposal, and I find it disrespectful to only receive an empty, canned response.

If you are on the other side of a similar process, be it for a conference or because you work in the HR department of a company, please do think about whether it’s really too much to ask for the extra five minutes it takes to provide the reasoning behind your decision, and whether you would like to be treated the same way as you treat others.

Google can certainly do less evil than that.

PS: This blog post about the rejected WordPress project mentions a follow-up meeting, but I was unable to find any concrete information on the GSoC website.

PPS: On the list of accepted organizations, you’ll find entries such as “Name: Pidgin, Finch, and libpurple. Tags: pidgin, finch, libpurple, instant messaging, im, networking, gtk, glib, c, amazing, awesome, a+++++ would do business with again, space mice, team edward” (emphasis mine) – is that supposed to be taken seriously? Or shall I conclude that the submissions are not thoroughly reviewed?

Update: There are some who try hard to misread what I wrote. I think I should clarify what a successful application as a GSoC organization means.

You will:

• spend a lot of time mentoring students during your typical holiday and travel season;
• not get paid for that (the students will); (update: mentors get 500USD, thanks to lucianafujii for pointing that out)
• receive a GSoC T-Shirt;
• be invited to the Googleplex later the year (you will usually have to take extra holiday for that though, so there goes more of your spare time, if you accept the invite);
• increase the visibility of your project.

Heavily simplified, GSoC consists of one entity providing the money, another entity providing time and a third entity consuming both. There’s also a big deal of knowledge transfer happening between the entities, but the flow is less clear than for time and money.

However, my point is: I would have been part of the time spending entity and without that entity, GSoC simply does not happen. So think again before you essentially reduce me to a loudly complaining free-rider who should STFU and get over himself.

Designing an input method system for Wayland

As mentioned before, Jon and I met with Kristian Høgsberg and Peter Hutterer during FOSDEM 2012. The discussion quickly revolved around Wayland and input methods. Kristian and I basically picked up from where we left after the Qt Contributor Summit last year in Berlin.

Right now, there is no explicit input method support in Wayland, and it’s the responsibility of the applications to provide such. But we came to the conclusion that moving it into Wayland provides several benefits.

That is why I spent some time during February to write down an initial design proposal, using the two years of experience from working in the application framework team for the Nokia N9. It has gotten longer than expected, but hopefully it is structured well enough to still be accessible.

From the introduction:

Currently, Wayland has no explicit text input method support and relies entirely on toolkit-specific or platform-specific solutions. This is awkward for platforms that support multiple toolkits. The resulting fragmentation and re-implementation would make it hard to attract developers of high-quality text input methods. These developers would generally prefer a unified architecture that lets them ignore application-specific or platform-specific details. This would let them simultaneously target all platforms that support Wayland.

We believe this goal can be reached by adding two new Wayland protocols, of which only one requires adaptation by toolkit providers.

The idea is not new, XIM has been around since the dawn of time. Because of the protocol’s daunting complexity, it apparently never became popular among application toolkit providers. That is why I decided to split up the envisioned Wayland input method protocol up into two:

• The Wayland text protocol, to be used between the Wayland compositor and its clients. It tries to keep input method knowledge to a minimum and therefore can be used for any kind of (hardware) input that results in text input.
• The Wayland input method protocol, to be used between the Wayland compositor and the input method system, which can either run as a stand-alone process or inside the compositor process.

In the proposal, I explain my ideal solution where applications and input method system are completely separated from each other and where the compositor acts as a mediator between them. I continue with alternative solutions that are feasible but have several drawbacks. Towards the end, there is a comparison matrix that features all discussed solutions.

There is an important aspect though: The alternative solutions provide an iterative implementation approach to reach the ideal solution. The obvious idea here is that already in early stages, we get testable prototypes. The solutions with a compositor-hosted input method system for instance allow others to think about window management policies that take input methods into account, whereas I could continue my iterations towards the ideal solution.

Anyhow, the next steps are now to discuss the proposal in a wider public (feedback from developers of exotic input methods especially welcome!), possibly also at upcoming conferences, and to come up with a prototype implementation so that we can iron out protocol bugs.

I would probably target Qt 5 as the first application toolkit to support the Wayland text protocol, followed by GTK+ and perhaps Enlightment. One also needs to take the popular desktop shells into account, such as Gnome Shell, Unity and Plasma Active.

Maliit could fill the role of a reference input method system, but we should have at least one more implementation, at some stage (IBus comes to mind, naturally).

Looks as if this could keep me busy for the better part of 2012, but it would be great if we could get at least the Wayland text protocol into Wayland 1.0. It’ll require some luck though.

Maliit as your remote control (video)

There is seemingly a trend (in ideas) to use your mobile device as a remote control for stationary devices such as your TV.

We thought that experimental support for remote text input should be easy to add to Maliit, as it already comes with a server-client architecture. As soon as client and server can run on different host machines, we’d have network transparency and it should basically just work. Luckily, we use D-Bus as our IPC, and it turns out that D-Bus understands remote host addresses. If you check the README in maliit-framework you will recognize that – with the latest release (0.90.0) – it only takes an additional environment variable, MALIIT_SERVER_ADDRESS, to connect a client (that is, an application that uses the Maliit input context) to the input method server. The scary part is perhaps that one has to disable D-Bus authentication. Therefore it's a good idea to only use this inside a trusted network.

Jon created a video which demonstrates the new feature in a better way, using a laptop that is connected to a projector as a TV replacement. The great thing about this feature is that it requires no changes to the applications, nor to the Maliit input method plug-ins, it’s the framework that handles this internally (or not so internally, now that Qt 5 turned our D-Bus API into a public one).

However, to make Maliit truly useful as a remote control, at least two additional features are required:

• Have a proxy text editor widget on mobile device that allows text interactions such as copy and paste, cursor positioning and rich text formatting.
• Emulate and transmit mouse or touch events, possibly through a dedicated touch area on the touch screen of the mobile device.

The second feature requires an extension to the Maliit protocol, whereas the first is already filed as bug MALIIT#84. After that, I could use my N9 (or my N900, as Maliit also runs there) to remote control my Gnome or Ubuntu Unity desktop, which would work perfectly for my movie watching habits. Kudos goes to Krzesimir and Jon for their work on this feature.

Is MeeGo Keyboard conquering GNOME3?

You'd probably like me more if I had spend my time testing the upcoming release of MeeGo Tablet UX on this shiny new WeTab (I ♥ Openismus), but Jon couldn't resist to test-drive GNOME3 first. Well, and since we like our virtual keyboard so much, we just had to make it work of course.

Jon also wrote about the features shown in the video.

Impressions from GUADEC 2010

Arrived in Amsterdam on Jul 22. First person I bumped into was Marc-Andre, fun. Already saw more of that city in one afternoon than I managed during the Maemo Desktop Summit in 2009.

Enjoyed trip to House of Bols on monday, Holland House Improved FTW (you won't find it on their website). Free shots got me squiffy. Bought passion liquor (let's hope Jon is trustworthy).

Met the trainees later the day and went to the Hague. Nice city, not as crazy as Amsterdam. Liked the sea side + grote markt.

GUADEC itself was great. Should have looked into the user-centric-design workshop I guess, but missed the one on monday, and the other overlaped with a talk, oh well. ClutterSmith looks fun. Compared to how you'd have to use Qt's graphics view + state machine framework, to get something similar, I prefer the tool-based approach that hides all the boring code from me, but allows me to focus on the transition logic.

MX and St are thinking about a remerge, that's a good thing. In overall, GNOME Shell looks a lot more promising now.

Let's see whether we can make GUADEC 2011 in Berlin as good as this one.

Audio File Preview

I am probably the last guy who discovered the preview feature for audio files in GNOME. Nevertheless I have a strong opinion about it, perhaps because of the way I made my first contact with it.

I have auto-login enabled for my GNOME session, which means the mouse pointer starts in the center of the screen. Now I also have a cluttered desktop and some of the files were actually audio files. Totally by random, one of them made to the center of the desktop. Wow - what a nasty surprise next time I started up the machine: My box autoplays music!?

Of course it doesn't, I only triggered the audio file preview by hovering the mouse pointer over the file's icon for a few seconds. There are several things I dislike about that:

• Audio file preview behaves differently than any other preview (granted, how would you make a thumbnail out of an audio file?)
• It will play the whole file if you don't move the mouse. How is that different from opening the file now?
• It is only activated if you use the mouse. Just selecting it with the keyboard won't do it.
• (It doesn't check whether I really moved the mouse pointer over the icon, it just checks whether the mouse pointer has been there for a definite amount of time.)

As it is, this feature isn't easily discovered. If you discover it, chances are good it will confuse you. Solutions? Or is it only annoying to me?

Learning GTK+

During the last weeks I started to learn about the GTK+. I mostly followed the official tutorial but soon tried to walk on my own legs. Perhaps that was not always the best choice, as I ran into dead ends later (I skipped some chapters which probably were important, but their title did not give it away). Requiring sequential reading is so 1999 ...

Now why is the GTK+ awesome? Simple, when using the GTK+ you almost forget you write in portable macro assembler. Pointers are annotated with a nice semantic (reference to an object, comparable to what languages with OO support see in variables), and memory management is usually none of your concern.

The next feature I was impressed with is type-checking since it gives you valuable hints whether you got the semantics of your pointers right. Granted, most of it happens at runtime, and it does not come without costs. But how else would you do it in C?

Also, I do like the approach of the GTK+ to not enforce a pixel-based layout. The GTK+ will negotiate the appropiate window size for your widgets, and it will re-negotiate it once you remove/hide widgets from your layout. This all comes for free together with the object hierarchy you create when you define the ownership relation of your widgets.

However, as soon as you start writing custom widgets it all gets a bit hairy. The good news are that for one, this part is well-documented on the web, and also that once you created a custom widget it can be used just as easily as the built-in widget types. Should that not be enough to encourage you to go the extra steps necessary then simply take it as a valuable lesson about "Implementing Object Orientation from Scratch".

And then I came across GObject properties. It all looked cumbersome, for little gain. At first I feared I would even have to pack everything into GValues before I can access properties, but luckily there is g_object_set() (and not g_object_set_property(), as you might have guessed). Now if properties could be made to react to signals directly ...

Overall, I like the GTK+ so far. Some things could be more obvious, some other things you just have to accept as god-given, and the rest seems dead-simple. If you are undecided which GUI toolkit to learn then perhaps the long list of available language bindings (quality may vary) is convincing to you. Learn once, use everywhere :-)