The media framework

"Play a video" sounds like one operation. In Android it is at least four: parse the container, decode the compressed stream, render decoded frames to a surface, and keep all of it synchronised to an audio clock. The framework's history is the story of pulling those apart — and of one security disaster that accelerated the process.

TL;DR#

The chain is container → extractor → decoder → renderer, with an audio clock as the master.
MediaPlayer wraps the chain; MediaCodec exposes the decoder; MediaExtractor exposes the parser. Most real apps now use Media3/ExoPlayer rather than MediaPlayer.
The 2015 Stagefright vulnerabilities were reachable by a media file arriving before the user opened it, which made the parser itself an attack surface.
The structural response was to split the media stack into sandboxed processes in Android 7.0 and to harden the codecs — not merely to patch the bugs.
The media framework now ships largely as Mainline modules, so codec and extractor fixes can arrive without an OS update.

Mental model#

Think of a cinema.

The projectionist (the extractor) opens the film can and finds the frames and the soundtrack. The translator (the decoder) turns the compressed frames into pictures. The screen (the surface) shows them. The conductor (the audio clock) keeps the soundtrack and the pictures in step.

The Stagefright era was a cinema where the projectionist, the translator, the screen and the conductor were all the same person, working in the same room, with the keys to the building.

Where the analogy breaks#

A cinema has one film at a time. A phone decodes several streams concurrently — a video call, a background music track, a notification sound — and they compete for the same hardware codecs. That contention is a large part of why media programming on Android is harder than it looks.

The big picture#

Stage	Interface	What it answers
Container parsing	`MediaExtractor`	where are the tracks, what codec, what timestamps
Decoding	`MediaCodec`, Codec2	turn compressed samples into raw frames
Rendering	`Surface`, `BufferQueue`	where the frames go
Synchronisation	audio clock, `MediaSync`	when each frame is shown

Deep dive#

MediaPlayer and its limits#

MediaPlayer is the high-level convenience API. It takes a URI or a file descriptor, prepares, starts, and reports state changes. It is fine for "play this MP3".

It is a poor fit for anything more complex, because it owns too much: you cannot easily control buffering policy, you cannot attach a custom renderer, and its error reporting is coarse. That is why the ecosystem moved to ExoPlayer, and then to Media3 as the Jetpack packaging of it.¹

MediaCodec, and why it is asynchronous#

MediaCodec is the direct decoder interface. You feed it compressed input buffers and drain decoded output buffers; the buffers can be backed by a Surface, in which case the decoder writes straight into a graphic buffer that the compositor can display.

The interface is asynchronous because hardware codecs are asynchronous. A software decoder could block until it had a frame; a hardware decoder has a pipeline of buffers and a queue. Modelling it as a queue from the start avoided a rewrite later.²

Codec2 and the move away from OMX#

The original codec interface was OpenMAX AL (OMX), a standard with a vendor-friendly but awkward component model. Codec2 replaced it with a model built around components, parameters and a buffer queue, and — importantly — it was designed to be a HAL, so codecs could live in a separate process and be updated independently.²

That separation is the security story again: a codec that runs in its own process with a narrow interface cannot take over the media server when it crashes or is exploited.

Stagefright: what actually happened#

In 2015 a set of vulnerabilities was disclosed in the media framework, several of them reachable by simply receiving an MMS. The user did not have to open the file, because the system parsed it to generate a preview. The parsing happened in mediaserver, a process with broad permissions.

The response had three parts:

Patch the specific vulnerabilities.
Change the process architecture so that parsing and decoding happen in sandboxed processes with the minimum permissions needed (Android 7.0).
Change the update process — the monthly security bulletin, and later Mainline modules, so fixes could ship faster than a full OS upgrade.

The architectural lesson is the durable one: a parser is an attack surface, and the correct place for an attack surface is behind a boundary.

Stagefright in the process table#

Loading diagram…

Text description of this diagram

A comparison of the process architecture before and after Android 7.0. Before, one mediaserver process held the parsers, the codecs, the camera path and the audio path with broad permissions, so a malformed file parsed by vulnerable code was a compromise of a privileged process. After the split, the media extractor, the codec process and the audio and camera paths run in separate processes with their own SELinux domains and least-privilege rules, so a decoder bug is contained.

It is worth being precise about what the split does not do: it does not make the codec correct. It makes a codec bug survivable.

The extractor problem#

A container format is a set of rules for interleaving streams, and parsers are written by hand against specifications that are often ambiguous. Every media file is therefore untrusted input in a format with a large attack surface.

The modern approach is a combination of:

A hardened extractor process.
Codec processes with sandboxing.
Fuzzing as a continuous process rather than a one-off audit.
The Mainline module model, so a fix reaches devices without waiting for an OEM release.³

Where Media3 fits#

Media3 is the Jetpack library that packages ExoPlayer, the media session APIs and the UI components. It matters architecturally because it moves the player out of the platform and into the app's dependency graph. An app can update its player without an OS update, and can rely on behaviour that is identical across device versions.

That is the same trade as AndroidX generally: the platform provides the primitives (codecs, surfaces, audio), and the library provides the policy (buffering, retries, DRM handling, track selection).

Permissions are small; the interesting surface is elsewhere#

Media playback needs almost no permissions. Reading media from shared storage is the exception: READ_MEDIA_* permissions (API 33) replaced broad storage reads, and the photo picker is the recommended way to get a user-selected item without any permission at all.⁴

The reason is that the risk in media is not "can an app play a file" — it is "can an app read your library" and "can an app exploit a decoder". The permission model addresses the first; the process architecture addresses the second.

How it evolved#

Security and privacy changes7 releases

API	Version	What changed
18	4.3 Jelly Bean	App Ops (hidden) introduced for permission control
23	6.0 Marshmallow	requestPermissions() and the runtime permission flow
28	9 Pie	Foreground service permission
30	11	One-time permissions
31	12	Bluetooth scan/connect permissions
33	13 Tiramisu	POST_NOTIFICATIONS runtime permission
37	17 Cinnamon Bun	ACCESS_LOCAL_NETWORK runtime permission for local network access

2008–2015One processParsing, decoding and playback in mediaserver.API 1

2015 →StagefrightA parser bug reachable by MMS changes the security posture.

2016 →SplitSandboxed media processes with least-privilege SELinux domains.API 24

2019 →Codec2Codecs become a HAL with a process boundary.API 29

2022 →Media3The player moves into the app's dependency graph.API 33

Under the hood#

the media framework: extractors, codecs, players
the broker for codec instances
the Codec2 implementation
MediaCodec.javathe app-facing decoder API

Curiosity corner#

MediaPlayer has a prepareAsync for a reason. Preparing involves opening a file, parsing a container and configuring a codec; doing that on the main thread is a guaranteed ANR.
The audio clock is the master. Video frames are dropped or repeated to follow audio, not the other way round, because a gap in audio is far more noticeable than a repeated frame.
"Supported codec" is a device claim, not a platform guarantee. The platform defines the API; the SoC vendor decides which codecs exist, and CDD sets a floor rather than a ceiling.

Why is audio latency higher on some Android devices than others?Because the end-to-end path — app, AudioFlinger, the audio HAL, the DAC — carries buffers at each stage, and their sizes are a dev…Open card →Why is Camera2 so much harder than Camera1?Because it exposes the real pipeline: you submit capture requests, and results arrive asynchronously with metadata and images. Tha…Open card →Are all codecs on a phone hardware?No. Hardware codecs handle common formats efficiently, while software codecs cover everything else at higher CPU cost. MediaCodec …Open card →

Try it#

Ask the device what it can decode

$ adb shell dumpsys media.player | head -40
adb shell dumpsys media.codec | head -40
adb shell cmd media.codec list

Example output

MediaCodecList:
name: c2.android.avc.decoder  type: video/avc  software: true
name: c2.qti.hevc.decoder    type: video/hevc  software: false
name: c2.android.mp3.decoder type: audio/mpeg  software: true

The software: true flag matters: a software decoder works everywhere but costs CPU and power, while a hardware decoder is fast but may have device-specific quirks.