Maliit Status Update

We’ve been busy! The result is our latest release, Maliit 0.99, with a massive amount of changes. While the framework has mostly seen code cleanups (:~/source/maliit/fw$ git diff 0.94.0..0.99.0 --stat: 389 files changed, 3612 insertions(+), 30081 deletions(-)), we’ve been adding tons of features to the reference plugins (:~/source/maliit/plugins$ git diff 0.94.0..0.99.0 --stat: 324 files changed, 32499 insertions(+), 12799 deletions(-)).

After (finally!) getting official Debian and Ubuntu packages, the next big step is our Maliit 1.0 release, which we expect sometime later this year.

Maliit 0.99 works with applications using GTK+2/3 or Qt4/5, though Qt4 still works best. Maliit’s framework and reference plugins strictly require Qt5 as build dependency, allowing us to get rid of Qt4 and Xlib legacy code. Maliit can run on Wayland but at this point only EFL applications support Wayland input methods. For Qt5 applications wanting to use Maliit together with Wayland, a Qt5 platform plugin for Wayland input methods would be needed.

Who is using Maliit?

The best Maliit showcase is still the Nokia N9. No other device got anywhere near to the demonstrated input method integration level. The haptic feedback of its virtual keyboard remains unchallenged. I think it’s a major success for our old Harmattan text input method team that we see Maliit being used in other projects. It was hard work to get to this point and yet it feels as if this was only the beginning of Maliit as an open-source project.

We have the friendly folks beind the OLPC project who were one of the first to go with Maliit. Few of us will probably make the new XO their primary device and therefore, won’t see the results of the collaboration. But from the very beginning, they worked with us in the open, provided official Fedora packages and helped us test-driving the then-new styling engine of Maliit Keyboard. The XO is one of the use-cases where Maliit has to run without a compositing window manager, which made them an early adopter of our windowed mode.

Plasma Active is another project that — for its size — invested significant development time improving and testing Maliit’s reference plugins. They’ve been (and hopefully remain!) a huge help whenever we faced problems with QML. I wish we could find a sponsor however to work on improving their input method integration, which is the “invisible” and challenging part that easily gets forgotten.

Ubuntu Touch is choosing Maliit as well. Canonical sponsored significant work in Maliit Keyboard, especially in terms of Qt5 support and replacing QGraphicsView with QtQuick2.

SailfishOS is happy user of Maliit, too. They are the main consumers of our 0.80 branches that still offer full support for Qt4 and X11. We always considered 0.80 our latest “stable” release branch, so it’s good to know that at least someone is also using it.

OpenEmbedded has integrated Maliit into their meta layer. This makes it easier to build custom Linux distros for embedded with Maliit on board. My understanding of OpenEmbedded is still limited at this point so I cannot explain all the real benefits this collaboration provides. Luckily there’s the always friendly Samuel Stritzel who’s following our official mailing list, waiting to answer all your OpenEmbedded related questions.

Gnome3. Just kidding. But once Gnome3 properly integrates with Wayland input methods, we can just plug in Maliit, too. Of course to be of any interest for Gnome3, we need to demonstrate working CJK input methods too. Luckily, we got the opportunity to just do that: IBus working with Wayland input methods. If things pan out I’ll be talking about this at GNOME.Asia.

I would like to get the chance to improve input method support for Firefox and Chrome/Chromium. There is one nasty bug around password fields that’s been quite a blocker for us. Some day, we’ll hopefully be able to convince the browser guys to just “do the right thing”.

And then there’s yours project of course. You just didn’t tell me about it yet ;–)

Architecture

There is an influx of new developers showing interest in Maliit which is why I feel that reiterating the basic architecture behind Maliit doesn’t hurt.

Maliit is split into three parts: framework, reference plugins and input method modules (GTK+, Qt4, Qt5).

The framework provides maliit-server, a process that can load reference plugins (such as VKBs) and allows applications to connect to it through input method modules. The input method modules are UI toolkit specific and are loaded dynamically by the applications. For each UI toolkit Maliit wishes to support, a separate input method module is required.

Input method modules and framework are designed as client-server architecture, with applications behaving as clients. The default IPC is QDBus but it can be replaced by other mechanisms, for instance the Wayland IPC.

In a follow-up blog post I will provide an outlook of where Maliit goes next and which parts of it we might have to rethink, now that the leviathan of Linux DEs is quickly moving towards Wayland.

People

Those are the people who have contributed to Maliit, but you wouldn’t know if you only checked commit messages:

• Łukasz Zemczak, Thomas Möricke and Kevin Wright: You guys have been great. Would work together again.
• Iain Lane, Michał Zajac, Peter Robinson: Without you, we still wouldn’t have proper Debian/(K)Ubuntu/Fedora packaging. Thanks!
• Samuel Stritzel: Thanks for bringing Maliit to OpenEmbedded! Hopefully we’ll see more use of your work soon.
• Simon Schampijer, Daniel Drake, Gary Martin, Jennifer Amaya and Martin Langhoff: Thanks for the opportunity to work on OLPC. I know most of us are not in the target audience which makes it hard for you guys to motivate others to help you. Your work is important, please never give up.
• Pekka Vuorela: I am just glad to have you back in the team. Maliit just isn’t the same without you.
• Aaron Seigo, Marco Martin: Thanks for all your enthusiasm about Maliit and Plasma Active. Your motivation is infectious.
• Jan Arne Petersen, Krzesimir Nowak: I am impressed by your motivation and dedication that you still have for input methods, even though none of us ever planned to become an “input method expert”.

FOSDEM 2013

I am back from FOSDEM and I am always surprised how quickly it’s all over. Would you believe that I only got to attend two talks again, one of them being my own about Wayland input methods?

Friday Feb 1st

I arrived Friday evening, this time by train. Since I bought my first BahnCard ever in August last year, I am slowly convinced now that flying is just a huge waste of time and should be the last possibility to consider, not the first. Even though I spend much more time on the train than I would on the airplane I still travel more relaxed. Others claim to be super-productive on trains. I am happy enough if I get to braindump my thoughts into my notebook (the dead-tree variant) and then read a book (more dead-tree) to forget about work for once.

Another trick for better conferencing is to arrive (or leave) 1-2 days earlier (later), as this helps catching up with sleep and allows to process thoughts or just do some random stuff not related to conferencing, say, going to a museum or gallery. It allows me to be reckless to myself during the conference without needing one week to recover.

But back to that Friday evening: I was very happy that Quim could make it to our dinner at the Kabash. I knew he had a busy schedule for the next days and that I probably wouldn’t be able to meet him in Berlin. I think he enjoys working for Wikimedia even though he now has to pay for phone calls and 3G. We also had people from Igalia, Digia and Intel OTC joining. We enjoyed great food and nice conversations, then headed off to the beer event. This one has become crazily crowded. It’s nigh impossible to find people you know. Meeting new people is also a challenge. The entire concept of what should be a simple meet&greet is lost these days.

Saturday Feb 2nd

Jens and I arrived around 10am at the campus on Saturday, probably a first for both of us to be so early. It was so empty you could think they’d cancelled FOSDEM but that quickly changed of course. I had zero slides done by then and no motivation to do them either. I went to Dave’s talk and it was nice to see him and Mardy working on a relevant project again. As expected, GOA is dead in the water by now and work will most likely continue with accounts-sso. I realized that I would have to recycle an older talk for some slides, something I loathe to do.

Somewhere in between I had to meet some OpenEmbedded guys to debug Maliit on their custom hardware, but we didn’t get much further than realizing that dlopen() didn’t work properly.

Funniest thing that happened that Saturday was probably rasterman, timj and thiago trashing-talking about the kernel when they could have told each other how much their toolkits suck!

Biggest reveal of the day, however, was that Rob Taylor toured with the Darkness in his crazier years.

Jens was unlucky with his hands-on DLNA talk, but I blame it on the room (Lameere). I remember that my talk there last year was crap, too. None of his prepared demos worked and people started to leave the room. I felt bad for him because I the was one who pushed him into giving the talk. Mark took over the second half and surprisingly, all of his dLeyna demos worked. He got applause (and rightfully so) for each of them.

I only had two hours left by then for my talk (slides). Meeting more people all the time didn’t exactly help with my preparations. I was close to call Luc (organizer of the Xorg DevRoom) and ask him to cancel my talk. I am glad I didn’t because it turned out that worrying about my talk for the whole day was the best way to get back into the topic. You have to understand that Wayland input methods is a yesteryears project for me, all the hard thoughtwork as been done already and I have been mostly working on new projects since August last year. I think the talk was well received by the audience (at least the room was packed). However, I cheated by simplifying and ignoring the actual complexity of input methods a bit.

It was already getting late and by the time we arrived in Brussels center it was about time to head to the Gnome party. Same place at last year, but by around 11pm we had occupied the whole building. I didn’t even know half the guys. Gut feeling tells me this is a side effect of the UX hackfest that happend before, but I would like to believe that the Gnome community is growing again. It was great to see Gnome legends such as Federico or Alp attending, I think both of whom I had last (and first) seen in 2008.

Sunday Feb 3rd

I thought I’d be able to attend a couple of talks on Sunday, but I got stuck talking to people interested in actually working open-source DLNA stacks. I got invited to join the Xorg dinner in the evening, with half of the people being BSD guys. Xorg? BSD? I was fearing for my sanity. Some other embarassment might better be left unmentioned in this blog but the food and wine at least was great! Afterwards we ended up at Delirium again (where else …). At 2am we had to move to the Absinthe bar right next to Delirum. Wiser men than me (such as krh and Rob Clark, who now works for Red Hat!) left before, however. And so we ended up with more beer and several shots of Absinthe until the wee morning hours. Jon made the mistake at that point to ask me about my honest opinion (I actually don’t know why he tagged along with the Xorg hackers for so long), he might regret that by now. Wrong time, wrong place. Happens.

Monday Feb 4th

I had to checkout at 11am and get my train at 2:30pm. How I managed to get up by 11:15am I still don’t know. But luckily I was able to meet with Kat, Dave, Martin & Tobias for some waffle & chocolate breakfast. This was a much better FOSDEM ending than last year.

Tobias chose the same ICE train as me. Poor him, he had no idea what was coming for him: He had to endure my satirical reality talk from Brussels to Cologne. Both of us had to change trains then, with him heading off to Hamburg and me returning to Berlin. I wish I could have made it back a couple hours earlier now because I missed Quim’s presentation in the Wikimedia office.

Exhausted but happy, I finally arrived at home on Monday night.

Travel plans for September

Planning my travels for September made me yearn for a travel assistant — it’s impressive how much time one can waste with crappy booking websites and clueless hotel staff. On the other hand, doing it myself allowed me to shuffle things around until I was happy with the plan. Not sure my imagined travel assistant would have had that much patience ;–)

Anyway, tomorrow I’ll be attending the 2nd Automotive Linux Summit (Sep 19-20) and the schedule is packed with interesting talks. Sadly, there won’t be any time for me to visit Warwick Castle again, which is just 15 minutes away from the venue. They say it’s for kids, but the Warwick Castle daily shows are well worth a visit, even for grown-up kids.

The Automotive Linux Summit clashes with the X Developer Conference (Sep 19-21) in Nuremberg, which means I can only attend XDC on Friday (and perhaps the beer hike on Saturday). Now, I haven’t suddenly become a X hacker over night, but I want to know how the progress in Wayland is perceived and whether there’s some good advice for our Wayland input method work. It would have been a great opportunity for me to learn more about EGL and new trends in OpenGL, but I’ll have to leave that to my colleague Jan Arne Petersen, who will attend the whole conference.

After staying in Nuremberg for the weekend, I will spend a few days in Munich (not for the Wiesn) before I head home to Berlin.

Thanks to Openismus for sending me to ALS and XDC!

Concrete problems and generic solutions

For the Maliit Keyboard, we need the ability to style it for different form factors. As our focus is mostly on mobile devices (where the form factor is known), we got away with pixel-based styling that — while being resolution dependent — makes it easy to place and size the buttons in exactly the way needed. It’s what designers call pixel perfect, and the difference can be very noticable. Let’s take a look at the comparing screenshot, with the Gnome3 virtual keyboard on top (taken from the 3.2 release notes) and the Gnome3-styled Maliit Keyboard at the bottom (video). The former uses a generic layout engine and generic styling, whereas the latter allows to style pixel-perfect layouts.

For the official Gnome3 virtual keyboard, one can notice how the space bar is strangely sized, how the regular keys are unpleasantly square and how the enter and backspace keys are way too small. They all use the same graphical asset. For the Gnome3 styled Maliit Keyboad, the letters d-j, x-n, together with the space bar and its neighboring keys, form a neatly aligned center area, with a staircase pattern to each side of the keyboard. If you look in the last row, you will notice that the space bar is just a few pixels too large. This happens because the only non-pixel-perfect elements in the Maliit Keyboard are stretched keys and spacer elements, and I somehow still compute their width incorrectly. From a designer perspective, that “pixel bug” in the layout destroys the immersion of an otherwise good design. Pixel perfectness — whilst having a poor reputation among us developers perhaps — is actually quite a challenge to get right!

Maliit Keyboard is a reimplementation of the MeeGo Keyboard that was shipped on the Nokia N9. It doesn’t share any code with it, but it does reuse the ideas and the XML layout files. The format for the layout files evolved from programmer-friendly to designer-friendly over time, which I think was a good change. Instead of using the complex and pretty generic styling system of libmeegotouch (which was intended to fit the needs of any mobile application), I use a concrete style class that reads a styling profile folder, looks into into the two INI files (one for the main keyboard area, one for the extended keys) and fills a model with style attributes. In the given context of virtual keyboard styling, the concrete solution offers the same flexibility as the generic solution but comes with a fraction of the implementation complexity and remains simple to extend. This is because the concrete solution fits the intended use-case perfectly, whereas a generic solution will always acrue overhead.

The trade-off, of course, is that for a new use-case, I’d probably have to write a new style class and create new style attributes. It is very unlikely, however, that the styling requirements of a fairly specific “application” would fit the the styling requirements of a regular application, so why burden myself with a generic enough abstraction that I will never use? Inversely, why try to make a generic solution fit a specialised problem? Jokingly, that’s precisely what I call Java programming: Instead of solving the problem at hand, write a framework that can solve all similar problems. I have started to grow a strong aversion against that, not because I like reinventing the wheel (I don’t), but because I think that understanding the concepts behind an idea is far more valuable than clinching to an implementation of it.

For the Maliit Keyboard, I took the concepts of the libmeegotouch styling system and mixed my understanding of CSS with the XML layout files. The result feels pretty clean and easy to use, but that’s because I based the implementation on well-understood concepts, not because of the implementation itself. Who knows, if I keep myself busy with implementing virtual keyboards for just a bit longer, I should eventually be able to write a pretty decent book about VKB design patterns ;–)

Get ready for Qt Contributor Summit!

With only roughly over five weeks to go, QtCS 2012 is approaching fast on the event horizon (pun absolutely intended). During the TiZen Developer Conference, Quim Gil and I, with help from Robin Burchell, found time to work on a draft, the result being visible in the QtDev wiki. Don’t be scared by the amount of pre-scheduled sessions, we will try to keep it as unconference-y as possible. However this allows us to properly prepare selected sessions.

Please keep in mind that QtCS 2012 is invite-only.

Session presenters, prepare!

If you are a maintainer of a Qt Essentials module, then you will have to drive one of the pre-scheduled sessions.

It might be a good idea to start writing a report about what happened in your area over the last couple of months, and where you plan to take your module for the Qt 5.1 release. Attendees of your session will probably expect you to talk about Qt 5 migration and how it will affect code using your module. On top of that, the most urgent bugs might get discussed, along with new feature requests, so please be prepared.

If you find a topic that isn’t showing up in the program yet, consider to get in touch with other interested folks (the Qt development mailing list or #qt IRC channel are good places to ask). With sufficient feedback, you should have everything it needs to drive the session yourself!

Plenary sessions

The first day will start with the State of the Union session about Qt 5 and Qt 5.1, summarizing where we are and where do we want to go with the Qt project.

The second day is about betting on Qt Quick, covering topics such as theming, platform integration and cross-platform. We want to invite the maintainers and a few vendors/users/contributors to expose their thoughts and needs with the hope of bringing a common understanding.

For the last day, we have a special surprise, as the topic will be HTML5 & the web. I am aware that this will raise a lot of controversy among the community, but it is an area where we need to find answers. With all the latest interest in web technologies, the answer can no longer be “We don’t do web”. That’s what killed the dinosaurs, remember? Then again, Qt already has better answers then that, so it’s about time for a fierce but honest panel discussion, including the Qt WebKit maintainers and other users/contributors driving this area.

Hack'n'Tell

Of course we also want to you to feel entertained during the summit, which is why we want to try out Hack'n'Tell sessions.

The rules are simple: Show us a cool hack involving Qt. You got 5 minutes. No slides.

I realize that preparing cool hacks takes some time, which is the original intent for this blog post: to announce the Call for Cute Hacks. For the first day at least, it would be great to have a couple of presenters before the summit starts. For the other days, I kind of hope that the summit itself will spark great ideas and demos that need to be shown immediately.

I hope I got you excited a bit, so let’s make QtCS 2012 an event to remember!

Trip to California, May 2012

In anticipation of two busy conferences in the San Francisco Bay Area, the TiZen Developer Conference 2012 and the Ubuntu Developer Summit for the Quetzal release, I prepared myself by arriving one week earlier. This avoids the stress of a long distance flight immediately before conferences start, but also helps to adjust yourself to the environment (e.g., get used to public transport, find your ways around the area, get used to the food, etc.).

My flight to California was somewhat jinxed. Instead of getting a direct flight to SFO from an European airport, I had to change over in Miami. So instead of 12 flight hours, I was already up to 17 hours. Then the flight got delayed, and I of course missed the connection flight. Being rescheduled to the next flight meant another 5 hours of extra waiting time and arriving at SFO briefly before midnight. Which means no public transport for you. Luckily, I had informed Quim Gil about my additional delays (thanks to free WiFi at Miami airport), and being the good friend that he is, he decided to pick me up from SFO.

On the next day (a warm Sunday), I had time to visit the Computer History Museam in Mountain View. Later that evening, Quim, Henri and I met for a few drinks, the outcome being this Maemo thread. Sadly, the thread itself has seemingly no positive outcome yet.

On Monday, it was time to go to San Francisco, where I spent the next days (mostly) working and preparing my TiZen session for next week. It was great to experience San Francisco from a non-tourist point of view (though I still had to visit Johnny Rockets together with Alberto, Łukasz, Kat and Dave, the milkshakes really are that good). My highlight of the week was perhaps the lunch with the ever friendly Yorba guys on Friday. I saw their awesome new T-shirts, but sadly they’re for employees-only.

On the weekend, I met with Lokesh and we drove down to Santa Cruz (great beach, lots of surfers, too) and from there to Monterey, on the famous 17-Mile Drive. I even dared to (very briefly!) swim in the Pacific Ocean, another first for me. The Pacific made me pay my courage in blood though (nothing too bad, just an annoying cut at my heel).

Then, well rested and all that, it was time for the conferences. Since there was only one keynote for Monday evening at the TiZen Developer Conference, I spent most of the day at UDS. The one thing to notice was probably that UDS had way more developers than TiZen, and to my surprise, even some high profile GNOME developers attended UDS. So much for all the silly fights about Ubuntu not being GNOME etc. I also met Daniel d'Andrader again, which was nice.

For the other two days, it was always a bit of back-and-forth between UDS and TiZen. UDS had the better food (breakfast, hm!), but TiZen probably had the better parties (California Academy of Science comes to mind). On Thursday I started to feel exhausted and on Friday, I was glad UDS was to be over soon. The beach party (without a beach) totally killed it.

The flight back was uneventful. After two weeks in California, with the second one being extremely stressful, I was glad to finally land in Berlin again.

TiZen Developer Conference 2012

The first day started slowly, with only one keynote in the evening. I wasn’t very happy with it, a proper keynote would have focused more on the visions and goals of TiZen, not so much on Linux or Android, or Microsoft.

On the second day, my mind was mostly occupied with my own session, “Challenges of mobile text input”, in the afternoon. It was good that speakers were informed early enough about the whole process, and the presentation templates were available long enough before the event. Even with the cold color scheme, I kind of like the clean TiZen design in the templates (though I still can’t get used to the name “Ti-Zen”).

Two hours or so before my session, I had one technician help me upload my slides to the room’s PC. It’s great to not have to use your own laptop. Instead of worrying until the last minute whether everything will work, you know it just will, and if it doesn’t, it is not your task to fix it. Not having this typical presenter’s crisis helps to focus on the content of your session and also helps to stay calm.

I was quite happy with my presentation, even though I was hoping for more input method developers in the audience. You can find the slides on the Maliit wiki, video (or at least audio) should be available at some later point, too.

On the last day, every attendee could get a TiZen developer device. I was surprised it didn’t have “Intel Inside”, but the ARM Cortex9 with the Mali GPU is interesting hardware nevertheless. Perhaps not too surprising for a HTML5 platform, it also comes with a speedy and actually useful mobile browser from the start. Even if it is uncertain at this point whether TiZen will end being a success, I am looking forward to see actual TiZen consumer devices.

Thanks to the Linux Foundation for sponsoring me, and thanks to Brian Warner again for helping with the organising bits.

Multi word ribbon UI for Maliit (video)

Motivated by the new Blackberry 10 virtual keyboard, I decided to spent a couple of hours on a proof of concept, this video being the result.

I had to hack the Presage engine a bit to provide word prediction in a similar fashion to what you see in the 2-3 seconds of the Blackberry video. Then I added some space between the rows of the keyboard, so that I could place additional word ribbons there. The word candidates appear next to their starting letters, though it’s only one candidate per letter. I need to find a better solution here, but then again Blackberry guys also haven’t solved it either ;–) Tapping on the word candidates inserts then into the text editor (no gestures, for now).

The code is very hackish, certainly nothing I would publish. I am going to put it onto a tablet so that I can show it around to you guys at the TiZen Developer Conference or the Ubuntu Developer Summit.

Jon is going to bring a camcorder on Sunday, so perhaps we can actually record a real, youtube-worthy video then.

The challenges of mobile text input

I am going to speak about mobile text input at the TiZen Developer Conference (7-9 May, San Francisco, CA). My session is scheduled for Tuesday afternoon (PDF).

I had planned to only focus on the technical aspects, but I am also going to talk about other aspects, such as the media coverage a cleverly designed text input method can achieve, and which features are apparently important to consumers.

The most important feature is, unsurprisingly perhaps, the overall performance: How fast can consumers insert or manipulate text on their mobile devices — while still being accurate? I will explain techniques that can help to improve responsiveness, accuracy and speed of a virtual keyboard.

I’ll be in the bay area until end of next week and I am generally interested to discuss the finer details of (text) input methods, accessibility or display servers and how one could improve the situation for accessibility on Linux. Contact me on Twitter in case you want to meet up and go somewhere for dinner (or drinks) in the beautiful city of San Francisco!

Maliit Keyboard now with word prediction and error correction (video)

This is a feature that I personally wanted for a long time. Interrupted by FOSDEM, some Wayland research and many other things, I finally managed to get word prediction and error correction beyond prototype quality. The video shows just how amazingly good the Presage word prediction can be, even without extensive training (in fact, for the video we used the minimal language model training that comes with a regular Presage installation). The second part of the video shows how combining Presage with a spellchecker such as Hunspell further improves the provided word candidates.

Presage uses a very scalable approach called text n-grams. There is a lot of research in that area, but language models of contemporary language usage are either well guarded or cannot be freely distributed. Luckily, Presage comes with training tools such as text2ngram. Users can feed arbitrary language corpora to it, though one should be careful to perhaps not mix different languages too much.

Matteo Vescovi, the author of Presage (formerly known as Soothsayer), started the work as part of his master thesis a couple of years ago. The heart of Presage are the different predictors. They can be queried in parallel and the result lists are merged, using probability analysis.

Presage certainly has a lot of potential. It comes with an easy to use C++ API but also provides bindings for C and Python. In fact, it even provides a D-Bus API, which would make it possible to run it as a system service. The user could then benefit from (and train!) the same language models from different applications.

Hunspell probably doesn’t need much introduction. It is used in many Linux desktops. It’s a fine library and comes with many dictionaries. However, one should be aware that Hunspell itself cannot provide word prediction, which is why it wasn’t enough for mobile text input. As a fallback for Presage, it works very well though.

Of course there are a couple of things we could do from here. As Jon mentioned in the video, the virtual keyboard’s word ribbon UI could host word suggestions from other applications, such as the Google search in the browser. For Unity’s dasher input or Gnome Shell’s search, the application names could be shown instead. Or we could hook it up to Bash completion.

PS: Anyone up to package Presage for Fremantle or Nemo? It could be pretty interesting to see the next release of Maliit running on the N900.