ariya.io

Not Everything is an Agent

Mon, 31 Mar 2025 22:47:17 -0800

“Agent” is likely going to be the word that will cause existential dread to true LLM enthusiasts.

Everyone’s got a different idea of what it means. In our modern age of innovation theater, lots of organizations gleefully slap the “agentic” label on anything that vaguely resembles a regular program (and pocket tons of money). Even a simple HTTP call to an LLM-as-a-Service can be called an agent, if you try desperately hard enough.

The internet, as always, is flooded with “groundbreaking” tutorials on building these so-called agents. Often authored by the latest hypefluencers, they typically involve a few lines (probably generated by whatever coding assistant is currently trending on Hacker News) that compose LangChain and an Ollama instance, often being presented as the pinnacle of AI autonomy. Because why bother with actual innovation when you can just repeat the quasi-boilerplate code ad nauseam?

That’s why I liked it a lot when the Anthropic article, Building effective agents, came out, as it dares to suggest that simply bolting on retrieval or memory to an LLM does not, in fact, make an agent. And chaining or routing? That’s just glorified control flow, folks. Only when an LLM is tasked with truly complex, real-world tasks such as coding or using a computer, does it begin to resemble the autonomous agent we’ve been promised

So how do you identify a real agent? Don’t be fooled by the grand pronouncements of those rearranging deck chairs on the Titanic. Ask for the receipts of successful evaluations! Anecdotal evidence of a few successful LLM calls isn’t that useful. Remember, in the world of LLMs, as in life, the loudest claims are often the emptiest!

Afterburner and Power Limit

Fri, 28 Feb 2025 21:37:19 -0800

Ever witnessed a fighter jet spewing hot flames as it kicks into afterburner? In that moment, efficiency is deliberately sacrificed for maximum acceleration.

In the midst of combat, efficiency means nothing when your life is on the line. The jet engine must keep roaring, before the pilot gets taken down by the enemy (and potentially meets their maker).

A GPU faces a similar fate. When pushed to consume hundreds of watts to churn out LLM tokens at the user’s breakneck speed, there’s no choice but to run as fast as possible, even if sweat is pouring and muscle fatigue reaches its peak.

Fortunately, nvidia-smi, with its pl (power limit) option, can be used to set an upper limit on power consumption, so the GPU doesn’t go completely overboard. Those last few dozen watts often don’t make a significant difference in performance, but they definitely contribute to heat generation, which needs to be monitored.

From the graph (measured with llama-bench, for the Mistral-7B-Instruct model, Q4), it’s evident that pushing the power further doesn’t lead to increased LLM speed. 250, 300, or even 350 watts, it’s more or less the same. Meanwhile, dropping to 200 watts does slightly decrease the speed, but it’s very worthwhile considering the power consumption is reduced by a third.

Saving energy is always a wise choice!

Privacy-Preserving Personal Search Appliance

Thu, 30 Jan 2025 20:27:09 -0800

This is powered by SearXNG, an excellent open-source meta search engine.

Unlike traditional search engines like Google or Bing, SearXNG doesn’t crawl the web and index content. Instead, it leverages other search engines like DuckDuckGo, Qwant, and Mojeek to fetch results while protecting your privacy. This means your personal information isn’t tracked by those upstream services.

With the rise of LLMs and RAG, SearXNG has gained even more popularity. But I’ll dive into that in a future post.

Setting up SearXNG is a breeze. You can use Docker or Podman (my favorite Docker replacement, everyone should use it!) to get it running quickly. In fact, I encourage you to try it on your main machine. You’ll be surprised how easy it is!

A little fun fact about the name. SearXNG is an active fork of SearX. As typically the case, NG usually stands for “next generation”. However, the X in SearX is actually the Greek letter chi, which is often transliterated as “ch”. So, you could think of SearXNG as “searching”.

While you can run SearXNG on your main machine, a dedicated device offers several advantages. You can share it with your family or colleagues and add extra security layers like Tailscale, Wireguard, or the good old OpenVPN.

For my little box, I chose a used Shuttle DH170 with an Intel Core i3 6100 (2 cores, 4 threads) and 16GB of RAM. This might seem like overkill, but it’s more than enough for SearXNG. The 200GB SSD is also plenty of storage. The total cost of the hardware was just $70. I could have saved more by using less RAM and storage, but I had these components on hand.

In terms of power consumption, the appliance idles at around 10W. I haven’t optimized it yet using tools like PowerTOP, but even so, it’s quite efficient. The off-the-shelf x86 architecture offers excellent upgrade potential. I could easily swap in a more powerful CPU like an Intel Core i7 6700 (4 cores, 8 threads) or add more RAM and storage if needed.

Initially, I considered using Proxmox to manage the system and run SearXNG in a container. However, I found this to be too complex. Instead, I opted for a simpler approach using vanilla Debian and Podman.Then, I remembered that there is project, CasaOS, a user-friendly home server OS. It offers a web-based interface for remote management and can easily run SearXNG. If you’re new to home servers, CasaOS is a great way to get started.

For those who prefer a more resource-constrained solution, you could use a Raspberry Pi or similar device. CasaOS also works well on ARM-based systems.

In today’s digital age, privacy is a fundamental right. Unfortunately, our digital footprints are being exploited, and our personal data is being harvested. Rampant privacy violations are becoming the norm. Let’s take proactive steps to protect ourselves and our loved ones!

LLM Inference Machine for $300

Fri, 27 Dec 2024 20:17:14 -0800

You can absolutely run Qwen-2.5 32B. And of course, Llama-3.1 8B and Llama-3.2 Vision 11B are no problem at all.

Now, before you get too excited, there’s a catch: this rig won’t break any speed records (more on that later). But if you’re after a budget-friendly way to do LLM research, this build might be just what you need.

Here’s a breakdown of the parts and the amazing prices I got them for:

AMD Ryzen 5 3400G: $50
Gigabyte X570 motherboard: $30
16 GB DDR4-3200 RAM: $30
512 GB SSD: $20
NVIDIA Tesla M40: $100
Cooler for M40: $30
EVGA 750W PSU: $20
Silverstone HTPC case: $20

The motherboard was a crazy good find, a broken PCIe latch got me a killer deal. The Ryzen 3400G is outdated by today’s standards, with only 4 cores and 8 threads, but for a GPU-focused inference rig, it’s more than enough. Bonus: its Vega iGPU frees up the PCIe slot for the real star of the show, the M40 GPU.

Speaking of the GPU, it’s a Maxwell-era data center card with a massive 24GB of VRAM. That much memory is essential for running hefty 32B models (quantized, of course).

While you can find a used M40 on eBay for around $90 these days, I had to buy an additional cooling solution (two small fans in a 3D-printed shroud), since data center GPUs usually don’t come with coolers or blowers like their consumer counterparts.

Here are the token generation speeds for several instruction-tuned models, quantized to 4-bit (Q4_K_M), measured with llama-bench:

Phi-3.5 Mini: 47 tok/s
Mistral 7B: 30 tok/s
Llama-3.1 8B: 28 tok/s
Mistral Nemo 12B: 19 tok/s
Qwen-2.5 Coder 32B: 7 tok/s

Performance is all relative. Compared to the latest RTX 3000 series, the M40 is definitely the slower sibling: about 5x slower, to be exact. But then again, an RTX 3090 is roughly 10x more expensive. Meanwhile, a more affordable RTX 3080 might limit your options with its 10GB (or 12GB for the enthusiast version) of VRAM.

An RTX 2080 Ti with 11GB VRAM could be a nice upgrade. Prices in the used market are dropping ($250 or less at the time of writing), and it delivers a solid 3x speed boost compared to the M40. Double the cost for triple the speed? That’s a pretty sweet deal!

How about Apple Silicon? The M2 Pro with its Metal GPU is roughly 25% faster than the M40. It wins easily in areas like portability, efficiency, and noise levels, but it comes with a significantly higher cost.

Coding assistance is a proven home-run use case for powerful LLMs. This is where the M40’s massive 24GB VRAM shines, enabling you to run the fantastic Qwen-2.5 Coder 32B model. Pair it with Continue.dev as your coding assistant, and you’ve got a powerful combo that could replace tools like GitHub Copilot or Codeium, particularly for medium-complexity projects.

The best part? Privacy and data security. With local LLM inference, your precious source code stays on your machine.

Now, should I go all in? Is it time to add a second M40?

Deploying an Uberjar to Dokku

Thu, 23 Feb 2023 11:37:36 -0800

Dokku is a self-hosted Platform-as-a-Service (PaaS) that offers a compelling alternative to popular PaaS solutions like Heroku. With built-in support for Linux containers, deploying an application on Dokku is straightforward. However, there is a lesser-known deployment method that involves sending a build artifact, such as a JAR package for Java apps, directly to Dokku.

This deployment method is useful when there is a need to quickly and frequently deploy the latest version of a custom application. By skipping the process of creating a container image, developers can focus on building the artifact for local development. This approach can be applied to packaged applications built with various programming languages, including Python, Java, JavaScript, PHP, etc.

To follow this process, it is necessary to have the packaged Java application in the form of an Uberjar, i.e. a JAR archive that contains all dependencies and can be executed by the JVM without requiring additional packages at runtime. The process assumes that Dokku has been installed on a machine named dokku.homelab.lan, and the dokku command is working properly:

$ dokku version
dokku version 0.29.4

Also, due to some heavy building that is going to happen on that Dokku machine, make sure there is an ample free capacity on the disk, ideally 8 GB or more (depending on the application).

If we are deploying an Uberjar, obviously that Uberjar needs to exist first. For this example, I am using an Uberjar from the open-source edition of Metabase (adjust things to suit your need). Note that Metabase is written in Clojure, not Java, though it runs on JVM. In theory, any other JVM languages (e.g. Kotlin, Scala, etc) can work as well.

$ curl -OL https://downloads.metabase.com/v0.45.2/metabase.jar
$ file ./metabase.jar 
./metabase.jar: Zip archive data, at least v1.0 to extract, compression method=store

Two auxiliary files, a Procfile and a Dockerfile, are required. The Procfile contains a single line of code that specifies how the application is executed, while the Dockerfile details the construction of the container.

The first file, Procfile, is this one-liner:

web: java -jar metabase.jar

The second file, Dockerfile, is not too strange to those who are familiar with Docker:

FROM eclipse-temurin:17
WORKDIR /app
COPY . ./ 
RUN java -version
RUN ls -l /app 
EXPOSE 3000

The base image is set to the latest Long-Term Support (LTS) version of OpenJDK v17 using the Eclipse Temurin distribution from Adoptium. The EXPOSE line indicates the port Metabase uses, which is port 3000. The two optional RUN lines are useful for debugging or resolving any issues that may arise.

Next, we package the necessary files into a tarball by executing the following commands:

$ tar cvf package.tar Procfile Dockerfile metabase.jar
$ file ./package.tar 
./package.tar: POSIX tar archive (GNU)

Before sending the tarball to Dokku, we must create an application. This is achieved by executing the following commands:

$ dokku apps:create metabase
-----> Creating metabase...
-----> Creating new app virtual host file...
$ dokku proxy:ports-set metabase http:80:3000
dokku proxy:ports-set metabase http:80:3000
-----> Setting config vars
       DOKKU_PROXY_PORT_MAP:  http:80:3000

The last command maps the host’s port 80 to the container’s exposed port 3000. And now, the fun starts!

$ cat package.tar | dokku git:from-archive metabase --
-----> Fetching tar file from stdin
-----> Generating build context
       Striping 0 worth of directories from tarball
       Moving unarchived files and folders into place
-----> Updating git repository with specified build context
-----> Cleaning up...
-----> Building metabase from Dockerfile
-----> Setting config vars
       DOKKU_DOCKERFILE_PORTS:  3000
Sending build context to Docker daemon  271.7MB
Step 1/12 : FROM eclipse-temurin:17
Digest: sha256:f6562feb32844d0059616d6e54c6cc3127ccf77fb594ccb98cc4279ca15887ed
Status: Downloaded newer image for eclipse-temurin:17
 ---> 1e117025f42d
Step 2/12 : WORKDIR /app
 ---> Running in 89d26eed69f3
 ---> db157924a857
Step 3/12 : COPY . ./
 ---> 59e836261c66
Step 4/12 : RUN java -version
 ---> Running in 21df4266e534
openjdk version "17.0.6" 2023-01-17
OpenJDK Runtime Environment Temurin-17.0.6+10 (build 17.0.6+10)
OpenJDK 64-Bit Server VM Temurin-17.0.6+10 (build 17.0.6+10, mixed mode, sharing)
 ---> 16d451db8f1a
Step 5/12 : RUN ls -l /app
 ---> Running in 829a2df4f10e
total 265328
-rw-r--r-- 1 root root        92 Jan 31 02:57 Dockerfile
-rw-r--r-- 1 root root 271686194 Jan 31 02:57 metabase.jar
-rw-r--r-- 1 root root        28 Jan 31 02:57 Procfile
 ---> 67b7b1179da7
Step 6/12 : EXPOSE 3000
Step 7/12 : LABEL com.dokku.app-name=metabase
Step 8/12 : LABEL com.dokku.builder-type=dockerfile
Step 9/12 : LABEL com.dokku.image-stage=build
Step 10/12 : LABEL dokku=
Step 11/12 : LABEL org.label-schema.schema-version=1.0
Step 12/12 : LABEL org.label-schema.vendor=dokku
Successfully built a246db231b6f
Successfully tagged dokku/metabase:latest
-----> Releasing metabase...
-----> Checking for predeploy task
       No predeploy task found, skipping
-----> Checking for release task
       No release task found, skipping
-----> Checking for first deploy postdeploy task
       No first deploy postdeploy task found, skipping
-----> Deploying metabase via the docker-local scheduler...
-----> Configuring metabase.dokku.homelab.lan...(using built-in template)
-----> Creating http nginx.conf
       Reloading nginx
=====> Application deployed:
       http://metabase.dokku.homelab.lan

The log may appear lengthy, but the steps it displays should be straightforward. On the Dokku target machine, a container image is constructed using the information in the tarball. From that image, a container is created and deployed using the standard Dokku machinery. If all goes well, the application (Metabase in this instance) will be up and running at the specified hostname.

As you become more familiar with this method, sending build artifacts to Dokku after each change will become a natural part of your workflow!

Continuous Integration for React Native Apps with GitHub Actions

Tue, 29 Dec 2020 18:03:18 -0800

For React Native mobile apps targeting Android and iOS, an easy way to setup its continuous integration is to take advantage of Actions, an automation workflow service provided by GitHub. Even better, for open-source projects, GitHub Action offers unlimited free running minutes (at the time of this writing).

The advantage of React Native is a single code-base targeting two major mobile platforms, iOS and Android. However, care must be taken so that when one developer focuses on implementing features on fixing defects on Android, whatever they check in into the code will not break iOS and vice versa. Ideally, that developer should always check and verify for both platforms. But mistakes happen and the best way to catch them is to ensure that the corresponding continuous integration (CI) is running smoothly to catch those potential problems early on.

Thanks to GitHub Actions supporting running the workflow on macOS and Linux (also actually Windows, but that is not too relevant for this purpose), creating a CI for React Native is easy enough. To follow along, check the sample project (in the style of Hello world) that I have created at github.com/ariya/hello-react-native.

Let us start with the Android build since it is the easiest. Create a file with the name android.yml under the directory .github/workflows. The content should be like this:

name: Android

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-20.04
    steps:
    - uses: actions/checkout@v2
    - name: Use Node.js v12
      uses: actions/setup-node@v1
      with:
        node-version: 12.x

    - run: npm ci

    - run: ./gradlew assembleDebug -Dorg.gradle.logging.level=info
      working-directory: android
      name: Build Android apk (debug)

The above YAML declares that this workflow must be executed for every pull request and also once it is merged, as well as when a commit is pushed into the source repo. The workflow runs on a Ubuntu 20.04 machine which is, thanks to GitHub, already equipped with some development packages, including Java, Android SDK, and many other bits and pieces necessary for Android development. The first step is to check out the code (obvious) followed by another step to pick the Node.js version (12 in this case, feel free to adjust it to your project). The step npm ci will install all the dependencies. The next step after that is invoking Gradle to build the app, just as it is being done for a local development machine.

Once this file is ready, commit it to the repo, push the branch, and voila! GitHub will start to execute that build process for any future branch push and also for all pull requests (for this simple demo project, the build process takes about 3 minutes or less, not bad at all!). If the pull request does not break the Android build, we will see the usual green checkmark, as illustrated below. Of course, if the build breaks, the failure will be displayed and we can track the build log to find out what has gone wrong (this helps to accelerate the troubleshooting).

For completeness, we can also have the build artifact, the APK file generated by Gradle, archived for every workflow run. To do that, add the following lines:

    - uses: actions/upload-artifact@v2
      with:
        name: android-apk
        path: '**/*.apk'

Clicking on the green mark icon on the commit view will lead to the detailed result of Action workflows for that particular commit. We can also find the link to the archived artifact, in this case the APK files. Since the artifacts are retained for some time (based on the project settings, default to 30 days), this can be very handy when we want to troubleshoot a problem. Let us say a certain feature does not work anymore with today’s build but we are confident that the same feature still worked with the build from last week. Rather than checkout out different revisions and rebuild the app, we can just grab the APK files. Since these are built in debug mode, we can comfortably launch it in an emulator and debug it just like an APK that we build on a local development environment.

How about building for iOS? It is not exactly the same but it follow the same principles. Here is a minimalistic workflow file, ios.yml, as a starting point:

name: iOS
on: [push, pull_request]
jobs:
  build:
    runs-on: macos-latest
    steps:
    - uses: actions/checkout@v2
    - name: Use Node.js
      uses: actions/setup-node@v1
      with:
        node-version: 14.x
    - run: npm ci
    - run: xcode-select -p
    - run: pod install
      working-directory: ios
      name: Install pod dependencies
    - name: Build iOS (debug)
      run: "xcodebuild \
        -workspace ios/HelloReactNative.xcworkspace \
        -scheme HelloReactNative \
        clean archive \
        -sdk iphoneos \
        -configuration Debug \
        -UseModernBuildSystem=NO \
        -archivePath $PWD/HelloReactNative \
        CODE_SIGNING_ALLOWED=NO"

The first few lines are just like the Android workflow. The major difference here is that the workflow needs to run on a macOS machine, as iOS SDK isn’t available on neither Linux nor Windows. The build steps follow a similar pattern: check out the code, setup Node.js, and install dependencies. There are two extra steps. The first one is to run xcode-select -p to ensure the readiness of the correct Xcode and its related tool. The second one, pod install, is used to install any dependencies for CocoaPods, assuming that the project is using CocoaPods to manage iOS-specific dependencies (usually it is). After that, we invoke the command-line debug build with Xcode, just like what we would do on a local machine. Since building for iOS is a bit more complicated, for this simple demo project, it will run for around 10 minutes, give or take.

Note that the above YAML files cover the build for iOS and Android. Please do not forget to create another workflow file that runs the tests, typically with Jest, to catch potential regression on the unit tests and/or integration tests. Often times, this workflow is also the best place to run various static and dynamic code analyzers (linters, code formatter, security scanners, etc).

Armed with three workflow YAML files, we fully established a simple and yet powerful continuous integration for React Native apps. Happy developing!

On GitHub Actions with MSYS2

Fri, 31 Jul 2020 20:33:31 -0700

Thanks to the complete GitHub Actions for MSYS2, it is easier than ever to construct a continuous integration setup for building with compilers and toolchains which can run on MSYS2.

The details are available on the official page, github.com/marketplace/actions/setup-msys2. However, perhaps it is best illustrated with a simple but concrete example. As usual, for this illustration, you will see the use of this simplistic Hello, world program in ANSI C. To follow along, check out its repository at github.com/ariya/hello-c90.

Let us take a look at this workflow setup to build this C program with GCC on MSYS2 (on Windows, obviously):

name: amd64_windows_gcc
jobs:
  amd64_windows_gcc:
    runs-on: windows-2019
    defaults:
      run:
        shell: msys2 {0}
    steps:
    - uses: actions/checkout@v2
    - uses: msys2/setup-msys2@v2
      with:
        install: gcc make
    - run: gcc -v
    - run: make CC=gcc
    - run: file ./hello.exe
    - run: ./hello

The important lines are for the setup-msys2 section. The install value allows an easy selection of various packages which shall be installed before proceeding to the next step. For this purpose, it is sufficient to install gcc and make, but YMMV.

The rest is self-explanatory. Please note also the defaults section earlier, this is convenient to set the default shell so that we do not need to explicitly call it for every single run thereafter.

Now let us come up with another variant, this time for Clang instead of GCC (read also my previous post: Clang for Windows)

name: amd64_windows_clang
jobs:
  amd64_windows_clang:
    runs-on: windows-2019
    defaults:
      run:
        shell: msys2 {0}
    steps:
    - uses: actions/checkout@v2
    - uses: msys2/setup-msys2@v2
      with:
        install: make mingw-w64-x86_64-clang
    - run: clang --version
    - run: make CC=clang
    - run: file ./hello.exe
    - run: ./hello

Pretty straightforward, isn’t it? We just change the package to be installed and the compiler to be used. Since the two YAML files are very similar, to avoid a lot of repeated steps, we can parametrize it as follows. This is basically taking advantage of the matrix strategy feature of GitHub Actions.

  amd64_windows:
    runs-on: windows-2019
    strategy:
      matrix:
        compiler: [gcc, clang]
    defaults:
      run:
        shell: msys2 {0}
    steps:
    - uses: actions/checkout@v2
    - uses: msys2/setup-msys2@v2
    - run: pacman --noconfirm -S make gcc
      if: ${{ matrix.compiler == 'gcc' }}
    - run: pacman --noconfirm -S make mingw-w64-x86_64-clang
      if: ${{ matrix.compiler == 'clang' }}
    - run: ${{ matrix.compiler }} --version
    - run: make CC=${{ matrix.compiler }}
    - run: file ./hello.exe
    - run: ./hello

To take it one step further, we can also support both i686 and AMD64 platforms in the same YML, again by parametrizing the architecture. Here is how it looks:

name: windows
jobs:
  windows:
    runs-on: windows-2019
    strategy:
      matrix:
        compiler: [gcc, clang]
        msystem: [MINGW32, MINGW64]
    defaults:
      run:
        shell: msys2 {0}
    steps:
    - uses: actions/checkout@v2
    - uses: msys2/setup-msys2@v2
      with:
        msystem: ${{ matrix.msystem }}
        install: make
    - run: pacman --noconfirm -S gcc
      if: ${{ matrix.compiler == 'gcc' }}
    - run: pacman --noconfirm -S mingw-w64-x86_64-clang
      if: ${{ (matrix.msystem == 'MINGW64') && (matrix.compiler == 'clang') }}
    - run: pacman --noconfirm -S mingw-w64-i686-clang
      if: ${{ (matrix.msystem == 'MINGW32') && (matrix.compiler == 'clang') }}
    - run: ${{ matrix.compiler }} --version
    - run: make CC=${{ matrix.compiler }}
    - run: file ./hello.exe
    - run: ./hello

That’s all 4 combinations, 32-bit and 64-bit, for each GCC and Clang, in a simple configuration!

Cross-compiling with musl Toolchains

Mon, 22 Jun 2020 05:37:59 -0700

When working on command-line utilities which can be useful for various platforms, from Windows on x86 to Linux on MIPS, the existence of a cross-compilation is highly attractive. A number of different binaries can be constructed conveniently from a single, typically powerful host system.

Alpine Linux popularizes the use of musl a no-frills C standard library for Linux. According to its website:

musl is lightweight, fast, simple, free, and strives to be correct in the sense of standards-conformance and safety.

In addition, thanks to Zach van Rijn, we have a collection of static toolchains based on musl at musl.cc at our disposal. The number of supported systems is rather mind blowing, you got everything from the usual i686 to MIPS to Microblaze and many others. https://github.com/ariya/fastlz/actions

As I search for a viable alternative to the cross-compilation method based on Dockcross (see my previous blog post: Cross Compiling with Docker on WSL 2), musl.cc fits the requirements nicely. I am in the process of migrating the continuous integration of FastLZ, my implementation of byte-aligned LZ77 compression algorithm, to be completely based on musl.cc.

Here is a quick walkthrough. As long as you are on Linux x86-64, you can follow along easily (and yes, this also works great on WSL, Windows Subsystems for Linux). As a reference, we will use the simplest ANSI C/C90 program available at github.com/ariya/hello-c90.

First and foremost, we need QEMU so we can test our binaries not native to x86-64. For convenience, GNU Make is also necessary.

$ sudo apt install -y qemu-users make

After that, let us grab the Hello C90 program:

$ git clone https://github.com/ariya/hello-c90.git
$ cd hello-c90

For a start, let us try to produce MIPS64 binary of our little Hello C90 program. Thus, we ought to grab the toolchains first, weighing at about 90 MB.

$ curl -O https://musl.cc/mips64-linux-musl-cross.tgz
$ tar xzf mips64-linux-musl-cross.tgz

To ensure that this fresh cross-compiler works, do a quick sanity check:

$ ./mips64-linux-musl-cross/bin/mips64-linux-musl-gcc --version
mips64-linux-musl-gcc (GCC) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

This looks good! Now we can compile our Hello C90 program statically:

$ make CC=./mips64-linux-musl-cross/bin/mips64-linux-musl-gcc LDFLAGS=-static
./mips64-linux-musl-cross/bin/mips64-linux-musl-gcc -O -Wall -std=c90 -c hello.c
./mips64-linux-musl-cross/bin/mips64-linux-musl-gcc -static -o hello hello.o

Checking the resulting binary should give the following:

$ file ./hello
./hello: ELF 64-bit MSB executable, MIPS, MIPS-III version 1 (SYSV), statically linked, not stripped

It is exactly what we want! To run the executable:

$ qemu-mips64 ./hello
Hello, world! From C90 with love...

Now, if you are doing this on WSL, or generally have a Windows machine available elsewhere, there is this fun activity of cross-compiling the above app for Windows, without the need for any Windows compiler and SDK. Same steps as before:

$ curl -O https://musl.cc/x86_64-w64-mingw32-cross.tgz
$ tar xzf x86_64-w64-mingw32-cross.tgz
$ make CC=./x86_64-w64-mingw32-cross/bin/x86_64-w64-mingw32-gcc LDFLAGS=-static
$ file ./hello.exe
./hello.exe: PE32+ executable (console) x86-64, for MS Windows

To really test it, just bring hello.exe to Windows and it is going to run as expected.

For more details and elaborated examples, check the collection of workflows YAML files of this Hello C program.

Combined with the continuous integration system of your choice, whether it is DIY via Jenkins or using one of the many services out there (GitHub Actions, Azure Pipelines, Travis CI), creating binaries for various operating systems becomes easier than ever!

Nix Package Manager on Ubuntu or Debian

Sat, 30 May 2020 20:11:45 -0700

Even though Ubuntu/Debian is equipped with its legendary powerful package manager, dpkg, in some cases, it is still beneficial to take advantage of Nix, a purely functional package manager.

The complete manual of Nix does a fantastic job on explaining how to install and use it. But for the impatients among you, here is a quick overview. Note that this also works well when using Ubuntu/Debian under WSL (Windows Subsystem for Linux, both the original and the newest WSL 2.

First, create the /nix directory owned by you (this is the common single-user installation):

$ sudo mkdir /nix
$ sudo chown ariya /nix

And then, run the installation script:

$ sh <(curl -L https://nixos.org/nix/install) --no-daemon

Note that if you use WSL 1, likely you will encounter some error such as:

SQLite database '/nix/var/nix/db/db.sqlite' is busy

This a known issue, the workaround is to create a new file ~/.config/nix/nix.conf with the following content

sandbox = false
use-sqlite-wal = false

and repeat the previous step.

If nothing goes wrong, the script will perform the installation. Grab a cup of tea while waiting for it!

downloading Nix 2.3.4 binary tarball for x86_64-linux
performing a single-user installation of Nix...
copying Nix to /nix/store......................................
replacing old 'nix-2.3.4'
installing 'nix-2.3.4'
unpacking channels...

Note that the last step (unpacking channels) can run for a very long time (no idea why, hope it will be fixed at some point). Just be patient.

To check whether Nix is successfully installed, we use the Hello, world tradition:

$ nix-env -i hello
installing 'hello-2.10'
these paths will be fetched (6.62 MiB download, 31.61 MiB unpacked):
/nix/store/9l6d9k9f0i9pnkfjkvsm7xicpzn4cv2c-libidn2-2.3.0
/nix/store/df15mgn0zsm6za1bkrbjd7ax1f75ycgf-hello-2.10
/nix/store/nwsn18fysga1n5s0bj4jp4wfwvlbx8b1-glibc-2.30
/nix/store/pgj5vsdly7n4rc8jax3x3sill06l44qp-libunistring-0.9.10
$ which hello
/home/ariya/.nix-profile/bin/hello
$ hello
Hello, world!

In the above illustration, hello is a test package that does nothing but to display the famous message. It looks simple, and yet it is very useful!

To get the feeling of packages available at your disposal (almost 29 thousands of them):

$ nix-env -qa  > nix-packages.list
$ wc -l nix-packages.list
28974 nix-packages.list
$ less nix-package.list

While it is not a substitute for a large collection of existing Debian/Ubuntu packages, very often what you get from Nix is more up-to-date. For instance, if you are stuck with a typical Ubuntu 18.04 LTS, it offers git 2.17.1, tmux 2.6.3, jq 1.5, curl 7.58, and Neovim 0.2.2. But, with Nix on that same Ubuntu system, at the time of this writing, you can enjoy git 2.26.2, tmux 3.1b, jq 1.6, curl 7.69, and and Neovim 0.4.3.

The way I use Nix however is not merely as a mechanism to get fresher software and other utilities. Rather, the functional nature of Nix leads to the possibility of multiple working environment, with each distinctive set of applications and tools, and the ability of switching cleanly between them. Those who use nvm (for Node.js) or virtualenv (for Python) probably can appreciate this. Now, imagine nvm/virtualenv but not only for Node.js/Python, and rather applied to an arbitrary set of packages. I have covered this in details before in my previous blog post, Isolated Development Environment using Nix. That blog post talked about Nix on macOS but obviously the experience is very suited for Nix on Debian, Ubuntu, or any other Linux distributions for that matter.

I hope this will inspire you to explore Nix in depth!

Practical Testing of Firebase Projects

Wed, 29 Apr 2020 12:10:04 -0700

Your little Firebase project is getting bigger every day? Never underestimate the need to establish a solid and firm integration tests from the get go.

Once you start to utilize various features of Firebase, from Hosting, Functions, and Firestore, it is imperative to incorporate practical local testing as soon as possible. Not only it will save you from some potential nightmares down the road, it can also facilitate faster iterations and quick(er) turn-around time during refactoring and feature implementation. Here is a few random suggestions to get you started. To follow along, you can also check the git repository containing the sample code at github.com/ariya/hello-firebase-experiment.

First thing that you always need to do is to implement a health check functionality. The name could be as simple as ping. Hence, inside your main Firebase Functions, there should be a block of code that looks like:

exports.ping = functions.https.onRequest((request, response) => {
    response.send('OK');
});

Now if you want to be fancy, it does not hurt to show the timestamp (Unix epoch) which can be valuable to know that this is not a cached or outdated HTTP response. If you wish, feel free to extend it with useful tidbits (but be careful not to reveal sensitive information).

exports.ping = functions.https.onRequest((request, response) => {
    response.send(`OK ${Date.now()}`);
});

In your test code (shown here with Axios to perform an HTTP request, but the concept applies to any library), do a quick sanity check that this /ping is working. This is an important step towards a reliable local testing.

it('should have a working ping function', async function () {
    const res = await axios.get('http://localhost:5000/ping');
    const status = res.data.substr(0, 2);
    const timestamp = res.data.substr(3);
    expect(status).toEqual('OK');
    expect(timestamp).toMatch(/[0-9]+/);
});

Now, the test might fail miserably. If that is the case, you do not have the proper setup yet to use and run Firebase emulators. Using npm, make sure to install all the following packages:

firebase-tools
firebase-functions
firebase-functions-test
firebase-admin
@google-cloud/firestore

And check that your firebase.json looks like the following:

{
  "hosting": {
    "public": "./public"
    "rewrites": [
      {
        "source": "/ping",
        "function": "ping"
      }
    ]
  },
  "emulators": {
    "functions": {
      "port": 5001
    },
    "firestore": {
      "port": 8080
    },
    "hosting": {
      "port": 5000
    }
}

Note the rewrites section. This makes /ping handily available from the main Firebase Hosting domain, instead of the long and cryptic one such as us-central1-YOURFIREBASEPROJECT.cloudfunctions.net/ping.

Before running tests, make sure to launch the emulators for Functions, Firestore, and Hosting:

npm run firebase -- emulators:start --project MYPROJECT

In the above command, npm run firebase works because of the run script definition. Also, substitute the name of your Firebase project accordingly. If the setup is correct, your terminal should show something like:

emulators: Starting emulators: functions, hosting
hub: emulator hub started at http://localhost:4400
functions: functions emulator started at http://localhost:5001
hosting: Serving hosting files from: ./
hosting: Local server: http://localhost:5000
hosting: hosting emulator started at http://localhost:5000
functions[ping]: http function initialized
emulators: All emulators started, it is now safe to connect.

At this point, if you point your browser to localhost:5000/ping, you should get the OK message (followed by the number representing the timestamp as Unix epoch). Of course, running the full tests (npm test) should also yield in a successful run.

When setting up the tests for CI (continuous integration), it might be easier to let the emulators run the test automatically. Here is how it is done:

npm run firebase -- emulators:exec "npm test" --project MYPROJECT

The exec option run the subsequent command, in this case the usual npm test, after starting the emulators. Once the command is completed (whether successfully or not), the emulators are automatically terminated. This is perfect for the CI run!

Next trick on our sleeve: fixtures for Firestore. Let us assume that your application uses this NoSQL datastore via this simple function for illustration (and do not forget to add a new URL rewrite for /answer/):

admin.initializeApp(functions.config().firebase);
const db = admin.firestore();
exports.answer = functions.https.onRequest(async (request, response) => {
    try {
        const doc = await db.collection('universe').doc('answer').get();
        const value = doc.data().value;
        console.log(`Answer is ${value}`);
        response.send(`Answer is ${value}`);
    } catch (err) {
        console.error(`Failed to obtain the answer: ${err.toString()}`);
        response.send(`EXCEPTION: ${err.toString()}`);
    }
});

And the corresponding test:

it('should give a proper answer', async function () {
    const res = await axios.get('http://localhost:5000/answer');
    const answer = res.data;
    expect(answer).toEqual('Answer is 42');
});

Launching the emulators (using the previous instructions) and running the tests however will result in a failure. And if you go to localhost:5000/answer, you fill discover an expected response:

EXCEPTION: TypeError: Cannot read property 'value' of undefined

This should not come as a surprise. When Firebase Emulators launched, its database (for Firestore) is empty. Hence, there is still no proper document, let alone a collection. It will be unnecessarily tedious to populate the database (it works for this simple example but a real-world app might have tons of collections and documents). How do we prepare a fixture for this?

Well, again the Firestore emulators to the rescue! While it is running, and you can perform another steps to populate the database (outside the scope of this blog post, perhaps we will discuss in some other time), you can snapshot the database and save it as the test fixture:

npm run firebase -- emulator:export spec/fixture --project MYPROJECT

Once the fixture is available, rerun the emulator (either as start or through exec) with the import option and the Firestore database will not be empty anymore, as it is populated with the previous snapshot.

npm run firebase -- emulators:start --import spec/fixture --project MYPROJECT

Last but not least, let us run this test as an automation workflow using GitHub Actions. All you need is a file named .github/workflow/test.yml with the following content:

name: Tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Use Node.js
      uses: actions/setup-node@v1
      with:
        node-version: 10.x
    - run: npm ci
    - run: npm run firebase -- emulators:exec "npm test" --import spec/fixture
      env:
        CI: true

As it turns out, it is not too difficult to set up some practical tests of a Firebase project!

Search Box and Cloud Function

Tue, 31 Mar 2020 23:45:57 -0700

For a blog hosted with Firebase Hosting, it turns out that a little search box is fairly easy to implement by using Cloud Functions for Firebase.

As with the current trend nowadays, this blog is a static site prepared with Hugo and deployed to Firebase (see my previous blog: Static Site with Hugo and Firebase). Some time ago, I realized that since I am using Firebase anyway, might as well take advantage of its Cloud Functions to add a little search functionality to the blog, particularly for its 404 page.

Of course, I am cheating a little bit. Using the above search box actually just redirects the search to my favorite search engine, DuckDuckGo, resulting in the following:

Implementing it is almost trivial. First, we need index.js inside the functions subdirectory with the content as short as this (obviously, for your blog, replace site accordingly):

const functions = require('firebase-functions');
exports.search = functions.https.onRequest((request, response) => {
  const q = request.query.q || '';
  response.redirect(`https://duckduckgo.com/?q=site:ariya.io+${q}`);
});

Once it is properly deployed, the trigger URL will be in the form of us-central1-YOURFIREBASEPROJECT.cloudfunctions.net/search. This is rather ugly. To overcome that, set up a rewrite inside firebase.json so that it looks something like:

{
    "hosting": {
      "rewrites": [{
          "source" : "/search",
          "function": "search"
        }
      ]
    }
  }

and thus, the function is available as the top-level /search of your Firebase Hosting URL, including if it is a custom domain.

After this, inserting the search box is also equally fun:

<form action="/search">
<p><input type="text" name="q" required> <button type="submit">Search</button></p>
</form>

When a visitor uses the search, they will get redirected to DuckDuckGo and be presented with the search result. Fast and easy!

Automatic Merge of Pull Requests

Sat, 29 Feb 2020 23:01:57 -0800

After using Azure DevOps for a while, I am totally sold on its Auto Complete feature for pull requests. While it does not apply universally, I do believe that any development process should be at the level where merging pull requests, or generalizing it, integrating all forms of contribution, should be as automatic and as hassle-free as possible.

If you are not familiar yet with Azure DevOps, it is basically a pay-as-you-go service for code repositories, automatic build runs, task tracker, artifact management, etc. Azure DevOps is pretty much comparable to various other similar services, such as GitHub, GitLab, Bitbucket, and many others. Note that although it bears the name Azure, you do not need to use any other Azure services to be able to take advantage of Azure DevOps offering (similar to how you can use Google Maps but without the need to store your files at Google Drive or host your email with Gmail).

One feature that makes Azure DevOps (at the time of this writing) unique compared to others is its ability to mark a PR (pull request) as Auto Complete. To do this, go to the sidebar and choose Branches (under Repo menu group). Once the branch list is displayed, hover on e.g. master and pick its context menu (rightmost three-dot menu) and choose Branch policies. Pick some settings which suit your need. Make sure to customize the Build validation, this is done by adding a simple build policy.

Now, whenever you create a pull request, there is a noticeable blue button, Set Auto complete, on the pull request page. Basically what it does is the automatic merging of the pull requests of two conditions are fulfilled:

the pull request is approved (by one or more reviewers, per branch policy)
the build succeeds, i.e. as configured with its continuous integration

There are also a few tweaks possible. For instance, you have the option to squash the branch, rebase and fast-forward, etc. Even better, there is an option to automatically delete the branch once it is merged, which can really help to reduce clutter.

Removing the manual step of merging an approved pull request will eliminate one more thing than we, human being, need to be involved with. Who would not enjoy less amount of cognitive load? I hope other services such as GitHub, GitLab, Bitbucket, and many more will follow suit and implement the same feature!

Clang on Windows

Sun, 05 Jan 2020 14:46:09 -0800

Thanks to the MSYS2 project, now there is an easy way to utilize Clang to build C/C++ application on Windows. This works equally well for both 32-bit and 64-bit programs.

MSYS2 is a fantastic (and better) reimagination of Cygwin, it is like taking the best part of a typical modern Unix environment (a familiar shell, a general collection of utilities, a porting layer, a package manager, and so on) while still working on Windows. Bootstrapping into MSYS2 is easy, either install it directly (using the GUI installer) or use Chocolatey: choco install msys2. Once inside its shell, pacman is the go-to, ever-so-powerful package manager, with thousands of packages available at your disposal.

This of course, includes the toolchain. Not only the latest GCC is there, but we also have Clang! To illustrate the concept, let us go back to the simple ANSI C/C90 program covered in the previous blog post. Once we clone the repository, open MSYS2 32-bit shell and try the following:

pacman -S msys/make mingw32/mingw-w64-i686-clang

It is a simple step to install both Make and Clang. Wait a bit and after that, do the usual magic:

CC=clang make

A caveat here, Clang for Windows does not append the .exe suffix for the executable. Thus, a quick rename to the rescue:

ren hello hello.exe

And now you can run, inspect, analyze the executable as usual.

To incorporate it into the continuous integration using Azure Pipelines (again, see the previous blog post), we shall construct a new job. The basic step is as follows.

- job: 'i686_windows_clang'
  pool:
    vmImage: 'vs2017-win2016'
  variables:
    PACMAN_PACKAGES: C:\tools\msys64\var\cache\pacman\pkg
    CC: clang

First, programmatically install MSYS2:

  - script: choco install --no-progress msys2
    displayName: 'Install MSYS2'

After that, perform some pacman maintenances:

  - script: |
      pacman -Sy
      pacman --noconfirm -S pacman-mirrors
    workingDirectory:  C:\tools\msys64\usr\bin\
    displayName: 'Check pacman'

And then, we install the required packages. At the time of this writing, Clang version 9.0 (the latest) will be installed.

  - script:  pacman --noconfirm -S msys/make mingw64/mingw-w64-x86_64-clang
    workingDirectory: C:\tools\msys64\usr\bin\
    displayName: 'Install requirements'

For the x86 architecture (aka, 32-bit Intel/AMD), install a different package:

  - script:  pacman --noconfirm -S msys/make mingw32/mingw-w64-i686-clang
    workingDirectory: C:\tools\msys64\usr\bin\
    displayName: 'Install requirements'

And now, down to the actual build step:

  - script: |
      set PATH=C:\tools\msys64\usr\bin;C:\tools\msys64\mingw64\bin;%PATH%
      make
      ren hello hello.exe
    displayName: 'make'

As a minor tweak, we can also cache pacman downloaded packages. In the above example, it hardly matters since we only install Make and Clang. But if you have a larger application, e.g. requiring Python, Qt, and so on, it is wide to avoid the CI run redownloading the same packages again and again (saving bandwith, and also being nice to those mirrors). We can achieve this by using the Cache task from Azure Pipelines. Simply insert this after MSYS2 installation step.

  - task: Cache@2
    inputs:
      key: pacman
      restoreKeys: pacman
      path: $(PACMAN_PACKAGES)
    displayName: 'Cache pacman packages'

For the complete illustration of such a job, take a look at the actual azure-pipelines.yml for the hello-c90 project.

Clang everywhere, yay!

Continuous Integration of Vanilla C Programs for Intel, ARM, and MIPS Architecture

Mon, 22 Jul 2019 15:33:34 -0700

Developing cross-platform applications presents a major challenge:, how to ensure that every commit does not break some combinations of operating systems and CPU architectures. Fortunately, thanks an array of online services and open-source tools, this challenge becomes easier to tackle.

For this demo, I have the traditional Hello, world program written in ANSI C/C90 at this repository: github.com/ariya/hello-c90 (feel free to take a look). The objective is to verify its automatic build (for the purpose of continuous integration) for a number of different CPU architectures, operating systems, as well as the C/C++ compilers. Supported CPU architectures are (using Debian nomenclatures) are amd64, i386, i686, armhf, arm64, and mips. Among some C/C++ compilers to be tested are GCC, Clang, TCC, Visual C/C++ (as part of Visual Studio 2017 and also 2019), Pelles C, Digital Mars, as well as MinGW. Obviously, some combinations are not available. For instance, there is no such thing (at least, not yet) as Visual C/C++ for Linux or MinGW targeting macOS.

In this particular blog post, we will use Azure Pipelines, a hosted build system supporting all three major OS: Windows, macOS, and Linux. For the DIY among you, the same setup can be achieved by using something like Jenkins, GitLab CI, TeamCity, and many other alternatives, along with the some build agents for the corresponding OS you want to tackle.

The build itself is configured via the YAML file, azure-pipelines.yml. There is a job for each unique combination of (Architecture, Operating System, Compiler). For example, amd64_linux_gcc denotes the build job for binary for Linux on Intel/AMD 64-bit architecture, compiled using GCC. As for now, the total number of those jobs is 16.

The obvious build job is something like this. It is running natively on the hosted agent of Azure Pipelines. We just need to make sure that the right compiler (GCC in this case) is installed. For Linux and macOS, this can be via the package manager, apt and Homebrew, respectively.

- job: 'amd64_linux_gcc'
  pool:
    vmImage: 'ubuntu-16.04'
  steps:
  - script: sudo apt install -y make gcc
    displayName: 'Install requirements'
  - script: gcc --version
    displayName: 'Verify tools version'
  - script: CC=gcc make
    displayName: 'make'
  - script: file ./hello
    displayName: 'Verify executable'
  - script: ./hello
    displayName: 'Run'

On Windows however, there is no need to do that since the hosted Windows agent is already equipped with Visual Studio. However, because the build is carried out with a Makefile (more specifically, Makefile.win), we need GNU Make which is installed via Chocolatey. Note that a stage in the build job is verifying the executable (useful to know whether it is built correctly or not) using file (Linux and macOS) or dumpbin (Windows).

For two special Windows compilers, Digital Mars and Pelles C (the Windows flavor of a modified TCC), they need to be installed on the fly since they are not available on the Windows hosted agents. Digital Mars is installed with a little dance with curl and unzip. Meanwhile, Pelles C is readily available from Chocolatey.

To target non-Intel CPU architectures, we need to use some cross compilers. Since hosted Linux agent of Azure Pipelines supports Docker, the easiest way to achieve this is to use a Docker-based cross compilation using dockcross. This is explained in-depth in my previous blog post, Cross Compiling with Docker. One of such example is the following build job, for building for Linux running on ARM (32-bit). Note that since the resulting exectable is an ARM binary, we ought to use QEMU to run it.

- job: 'armhf_linux_gcc'
  pool:
    vmImage: 'ubuntu-16.04'
  steps:
  - script: sudo apt install -y qemu-user
    displayName: 'Install requirements'
  - script: |
      git clone --depth 1 https://github.com/dockcross/dockcross.git
     cd dockcross
      docker run --rm dockcross/linux-armv7 > ./dockcross-linux-armv7
      chmod +x ./dockcross-linux-armv7
    displayName: 'Prepare Dockcross'
  - script: ./dockcross/dockcross-linux-armv7 bash -c '$CC --version'
    displayName: 'Verify tools version'
  - script: ./dockcross/dockcross-linux-armv7 make LDFLAGS=-static
    displayName: 'make'
  - script: file ./hello
    displayName: 'Verify executable'
  - script: qemu-arm ./hello
    displayName: 'Run'

The same approach using Docker and QEMU works well for other CPU architectures such as MIPS, ARM 64-bit, and in fact Intel x86. The last one is quite necessary, since the hosted agent of Azure Pipelines is running in 64-bit mode. Thus, we use this virtualization layer (QEMU) to verify the correct execution of 32-bit binary.

As an illustration, two examples for MingW are illustrated. The first, MinGW is installed on the Windows agent. This is self explanatory.

- job: 'amd64_windows_mingw'
  pool:
    vmImage: 'vs2017-win2016'
  variables:
    CC: 'gcc'
  steps:
  - script: choco install mingw --version 8.1.0
    displayName: 'install MinGW-w64'
  - script: gcc --version
    displayName: 'Verify tools version'
  - script: make
    displayName: 'make'
  - script: file hello.exe
    displayName: 'Verify executable'
  - script: hello.exe
    displayName: 'Run'

For the second example, MinGW is used in a cross compilation fashion. Again, we use the Docker-based dockcross to achieve this. The compiler (GCC) runs inside the Docker container on the hosted Linux agent, however it produces a Windows executable. How do we run the resulting executable? QEMU is not suitable here (since we still need to install or run Windows, remember the host is Linux). But, we have WINE to the rescue!

- job: 'i386_windows_mingw_static'
  pool:
    vmImage: 'ubuntu-16.04'
  steps:
  - script: |
      git clone --depth 1 https://github.com/dockcross/dockcross.git
     cd dockcross
      docker run --rm dockcross/windows-static-x86 > ./dockcross-windows-static-x86
      chmod +x ./dockcross-windows-static-x86
    displayName: 'Prepare Dockcross'
  - script: ./dockcross/dockcross-windows-static-x86 bash -c '$CC --version'
    displayName: 'Verify tools version'
  - script: ./dockcross/dockcross-windows-static-x86 make
    displayName: 'make'
  - script: file ./hello
    displayName: 'Verify executable'
  - script: docker run -v $PWD:/app tianon/wine:32 bash -c "wine /app/hello"
    displayName: 'Run'

In fact, to avoid the hassle of on-the-fly installation/configuration of WINE, we just use the Dockerized WINE.

The whole ordeal of running 16 jobs will take anywhere from 5 minutes to 20 minutes. Obviously, if you are constrainted by the free tier of Azure Pipelines, you can purchase access to more hosted agents or attach your own build agents, which will definitely parallelize and speed things up.

I hope that the idea outlined in this post will inspire to continue to work on more cross-platform apps. Of course, it does not have to be an application written in ANSI C. The concept can be applied to D, Go, Rust, and many other modern compilers.

Cross Compiling with Docker on WSL 2

Sun, 30 Jun 2019 14:24:51 -0700

Now that WSL 2 packs a true Linux kernel and supports Linux containers (via Docker), it can be a perfect setup to perform application cross compilations.

While Docker for Windows will soon support WSL 2, it is just easier to use WSL 2 as is, install Docker, and use it. In case you are new to the wonderful world of WSL, check the documentation to have it installed. Note that for WSL 2, you need to be using Windows Insiders for now. Update: No need to use Windows Insiders anymore as WSL 2 is now included with Windows 10 June 2020 update (version 2004).

Once Ubuntu 18.04 is installed (the default for WSL), you can verify that it is indeed working:

$ uname -a
Linux XPS 4.19.43-microsoft-standard #1 SMP Mon May 20 19:35:22 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"

If uname -a returns something like this instead:

$ uname -a
Linux XPS 4.4.0-18362-Microsoft #1-Microsoft Mon Mar 18 12:02:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux

then you still got the original WSL, not WSL 2. Refer to the documentation again on how to enable WSL 2 instead.

Next step is to install Docker. There are tons of tutorial on this subject, see for instance this guide on Digital Ocean. Once it is properly installed, start the daemon by running:

$ sudo service docker start

At this point, it is likely wise to add yourself to the proper group, to avoid using sudo all the time:

$ sudo usermod -aG docker $USER

Before we do something crazy, let us ensure that Docker works:

$ docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.

Since the topic is cross-compilation, let us assume there is a rather simplistic Hello world in ANSI C/C90/C99:

#include <stdio.h>

int main(int argc, char** argv) {
  printf("Hello, world!\n");
  return 0;
}

with the following Makefile:

.POSIX:
.SUFFIXES:

CFLAGS = -O -Wall -std=c90

all: hello
hello: hello.o
	$(CC) $(LDFLAGS) -o hello hello.o

.SUFFIXES: .c .o
.c.o:
	$(CC) $(CFLAGS) -c $<

Run a quick test on your host system (assuming gcc and friends are already available):

$ gcc -o hello hello.c
$ ./hello
"Hello, world!"

What about compiling this marvelous C program for ARM? Another tool we would need is this excellent project, Dockcross, a Docker-based setup for painless cross-compilation.

$ git clone https://github.com/dockcross/dockcross.git && cd dockcross
$ docker run --rm dockcross/linux-armv7 > ./dockcross-linux-armv7
$ chmod +x ./dockcross-linux-armv7

Cross-compile the program for ARM v7 (this is for 32-bit ARM architecture):

$ ./dockcross-linux-armv7 bash -c "$CC -o hello hello.c -static"

Or better, use the Makefile, and hence the simplified command:

$ ./dockcross-linux-armv7 make LDFLAGS=-static
/usr/xcc/armv7-unknown-linux-gnueabi/bin/armv7-unknown-linux-gnueabi-gcc -O -Wall -std=c90 -c hello.c
/usr/xcc/armv7-unknown-linux-gnueabi/bin/armv7-unknown-linux-gnueabi-gcc -static -o hello hello.o

As the evidence that this is not a native host binary anymore, verify the freshly baked executable:

$ file ./hello
hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), statically linked, for GNU/Linux 4.10.8, with debug_info, not stripped

Whoa! That was indeed quite painless.

How do we execute this file to ensure that it is working as expected? QEMU, another great open-source project, to the rescue! First make sure that it is there (we need the qemu-user package).

$ sudo apt -y install qemu-user

And now the fun time: running the ARMv7 binary we built earlier:

$ qemu-arm ./hello
"Hello, world!"

Of course, this is just one target architecture. Amazingly, Dockcross supports a wide range of cross-compilation targets, including for MIPS and PowerPC architectures, or even building for Windows and WebAssembly.

Here is how to build a Windows executable in 3 easy steps:

$ docker run --rm dockcross/windows-static-x86 > ./dockcross-windows-static-x86
$ chmod +x ./dockcross-windows-static-x86
$ ./dockcross-windows-static-x86 make
/usr/src/mxe/usr/bin/i686-w64-mingw32.static-gcc -O -Wall -std=c90 -c hello.c
/usr/src/mxe/usr/bin/i686-w64-mingw32.static-gcc  -o hello hello.o
$ file ./hello
./hello: PE32 executable (console) Intel 80386, for MS Windows

And thanks to the excellent Windows-Linux interoperability of WSL, you can also run the executable directly.

$ ./hello
"Hello, world!"

In summary, even from a shiny Windows laptop, the combination of WSL 2, Docker, dockcross, and QEMU allows us to cross-compile apps for a number of processor architecture and operating system combinations.

Now, what kind of great apps do you plan to build today?