Mainline and APEX

Treble's insight was that the vendor image and the system image should be separable. Mainline's insight is that the system image should not be one thing either: half of it changes for reasons that have nothing to do with the other half.

TL;DR#

• Project Mainline (glossary definition) arrived with Android 10 (API 29·Android 10): individual platform components ship as modules that can be updated without an OS release.
• Two formats: an APK (glossary definition) for app-shaped modules, and an APEX (glossary definition) for components that are libraries, binaries or services.
• An APEX is a signed ZIP containing an ext4 image, mounted by apexd at boot through a loopback device and dm-verity.
• Updates are staged: the new version is recorded, activated on the next reboot, and can be rolled back.
• Delivery is Play System Update (glossary definition), which reuses the Play Store infrastructure and installs an atomic bundle of modules.

Mental model#

Think of a shipping port where the ship used to be one welded hull.

Treble cut the hull into a cargo section and an engine section that could be swapped separately. But the cargo section was still one welded block: repainting a container meant dry-docking the whole thing.

Mainline replaces the cargo section with containers on a deck. Each container has its own manifest, its own contents and its own lock. You can lift one out and put a newer one in without touching the others — and if the new one is wrong, you can put the old one back.

Where the analogy breaks#

A container is lifted off and replaced while the ship is docked. An APEX is replaced while the ship is sailing: the new version is installed to /data, then activated on the next reboot. The running system never sees the file change underneath it, which is what makes the update safe and also what makes it require a restart.

The big picture#

Deep dive#

What a module is#

A Mainline module (glossary definition) is a piece of the platform with its own version, its own owner and its own update cadence. The definition is organisational as much as technical: a module exists because someone can own its release independently.

The first cohort, at API 29, was chosen for a specific reason: they were all things that parse untrusted input or change frequently.

That table is a good summary of the selection criteria: security-critical parsers, fast-changing data, and user-facing policy. A module that changes rarely and is deeply entangled with the framework is a poor candidate, because the boundary costs more than it saves.

APEX: the container#

An APEX (glossary definition) is "Android Pony EXpress", and it is a valid APK in the structural sense: a signed ZIP with an AndroidManifest.xml. Inside are four things:

• apex_manifest.json — the package name and version.
• AndroidManifest.xml — a minimal manifest so APK tooling (adb, PackageManager, aapt) can handle it.
• apex_payload.img — an ext4 image containing the actual files, with a dm-verity hash tree and vbmeta metadata created with libavb.
• apex_pubkey — the public key used to verify the payload signature.

The payload is the point. An APEX is not a bag of files extracted into the system; it is a filesystem image mounted read-only. That is what lets a library live at a stable path with a versioned mount point.

The kernel requirements follow from that: the loopback driver and dm-verity must be enabled. AOSP documents CONFIG_BLK_DEV_LOOP=Y and CONFIG_DM_VERITY=Y as required, with kernel 4.4+ supported and 4.9+ required for devices launching with Android 10.

Why not just an APK?#

An APK is designed for app code: DEX, resources, a manifest, and a process model where the package is loaded by a runtime. An APEX exists for things that do not fit that model:

• A native library that other processes dlopen.
• A binary that init starts, such as adbd.
• A service that runs as its own process.
• ART itself, which is the runtime every app depends on.

You cannot put libcrypto.so in an APK and have system_server link against it, because the APK's files are not on a filesystem path the linker can see until the app is loaded. An APEX is mounted as a filesystem, so its contents are just files at paths.

Staged updates and rollback#

The update flow is deliberately conservative:

1. The APEX is downloaded through PackageManager, adb or another installer.
2. PackageManager recognises the APEX and hands it to apexd.
3. apexd verifies it and records it in its internal database for activation at the next boot.
4. The system reboots.
5. At boot, apexd verifies and mounts the new version.

The consequence is that an APEX update always costs a reboot, unlike an app update. The exception is supportsRebootlessUpdate, which allows an update without a reboot when the installed APEX is not in use.

Updates live in /data. A built-in APEX in /system/apex is shadowed by a newer one in /data, but the built-in copy remains as the fallback. That is the rollback story in its simplest form: the old version is still on the read-only partition.

The module set, by release#

The module list grew in waves. The table below is a cut of this site's module dataset, showing the first release and the format.

Delivery and atomicity#

Modules ship through Play System Update, which reuses Play Store infrastructure. The user sees a monthly update alongside app updates; the device installs a bundle of modules and activates them on reboot.

The atomicity is the important property: a bundle either applies entirely or not at all. That avoids the failure mode where three of five modules update and the combination is broken in a way none of the three teams tested.

What cannot be modularised#

Some things are still not modules, and the reasons are instructive:

• The kernel — that is GKI's job, and it is a different mechanism.
• system_server itself — too entangled with the boot sequence and too large to containerise.
• The HAL implementations — those belong to the vendor and move with the vendor image.
• Parts of the framework with deep native dependencies — Bionic, for example, cannot be swapped under a running system.

The boundary of Mainline is roughly "code that can be loaded or started after init and does not define the ABI everything else links against".

How it evolved#

Four releases tell the story: Android 9 laid the groundwork, Android 10 shipped the first cohort and the APEX format, Android 11 added Wi-Fi and adbd, and Android 12 added ART — the point at which the runtime itself became updatable. Everything after that is expansion.

Under the hood#

• ‹›↗apexd and the APEX tooling
• ‹›↗the source tree for Mainline modules; each subdirectory is one module
• ‹›↗the ART module packaging
• ‹›apexd.rc↗when apexd starts and what it depends on

Curiosity corner#

• An APEX is a valid APK. It has a manifest and is signed as one, so adb install can hand it to PackageManager, which then hands it to apexd.
• /apex contains versioned mount points. Looking at it on a device shows the versioned directory and the bind mount, which is the clearest way to see how shadowing works.
• ART as a module means the runtime can change without a platform release. That is the single biggest performance-relevant update channel Android has.

Try it#

$ 
adb shell ls /apex
adb shell dumpsys package com.android.art | head -20
adb shell cmd apex list 2>/dev/null | head -20

/apex lists the mounted modules by name. The versioned directories are the actual mount points; the unversioned ones are bind mounts to the active version.

Check yourself#

Sources#