I needed to build /e/OS to work on the prototype of a new feature. I had the idea of restarting an old machine, and I honestly wondered whether it would still cope with a modern AOSP tree.
It did, after two repairs: I added 32 GB of RAM (48 GB in total) and replaced a failing disk in the RAID array. Here is the full /e/OS build for the Fairphone 6 (AOSP 16, LineageOS 23.0 base) on a rack server whose CPUs predate AVX: the time, memory, disk, bandwidth and power it takes, and what breaks along the way.
Result: 8 h 54 min of build time, 124 GB of build output, about 344 GB of disk in total, and roughly €0.55–0.70 of electricity.
The machine
The x3650 M3 (machine type 7945) shipped in 2010, one generation after the 2009 M2. It is a 2U, dual-socket Westmere-EP box with triple-channel DDR3 per socket.
| CPU | 2 × Xeon E5620, 32 nm Westmere-EP, 4 cores / 8 threads each, 2.40 GHz (2.66 turbo), 12 MB L3 each, 80 W TDP each |
| Threads | 16 |
| ISA | SSE4.2, AES-NI. No AVX (that arrived with Sandy Bridge in 2011) |
| Memory | 48 GB DDR3 ECC RDIMM over six channels; DDR3-1066 is this CPU’s ceiling |
| Storage | ServeRAID M5014 (LSI SAS2108). Sources and output on a 5.9 TB RAID 5 volume, OS on a separate volume |
| Host OS | Ubuntu 20.04, kernel 5.4, glibc 2.31, Python 3.8 |
AOSP’s host toolchain still runs on a pre-AVX x86-64 CPU: the prebuilt Clang, the JDK 21 used by the build, and Rust 1.83 all ran without complaint. The old userland is the real constraint (see below).
The bill
| Source sync | 34 min for about 970 repositories (repo sync -c -j8 --no-tags) |
| Git objects | 81 GB |
| Checked-out tree | 93 GB |
| Device images | 3.75 GB vendor factory image + 2.4 GB official /e/OS OTA, used to extract proprietary blobs and firmware |
| Build graph | 174,567 actions |
| Soong / Kati analysis | about 10 min before the first compile step, with about 26 GB of RAM in use at the peak |
| During compile | 16 threads at 92–100%, load average around 18, 8–10 GB of RAM in use (Java-heavy phases) |
| Main compile | 7 h 52 min to reach 96% (168,409 actions), when a packaging step failed |
| Finishing runs | 10 min + 19 min + 33 min (three incremental restarts after fixes) |
| Total build time | 8 h 54 min |
Build output (out/) | 124 GB |
| ccache | 20 GB |
| Disk, everything | about 344 GB: git objects, tree, out/, ccache and the two device images |
| Deliverables | 2.44 GB OTA package, 2.71 GB fastboot image package |
| Previous run, AOSP 15 | Full build of the same device: 10 h 23 min, with only 20 GB of RAM. Incremental rebuilds: 27 min to 1 h 22 min |
| Power | about 250–300 W at the wall under full load, so 2.2–2.7 kWh for the 8 h 54 min |
| Electricity | roughly €0.55–0.70 per full build at €0.25/kWh |
| Uninterrupted build | about 8 h 30 min, if none of the failures below had happened |
The electricity is cheap. The real cost is time: a current 32-thread desktop does the same build in a fraction of it, and every failed run on this box costs minutes to hours before the error shows up.
1 [||||||||||| 94.1%] 2 [||||||||||| 93.5%] 3 [||||||||||| 94.1%] 4 [||||||||||| 94.7%] 5 [||||||||||| 94.8%] 6 [||||||||||| 92.9%] 7 [|||||||||||| 96.8%] 8 [||||||||||| 94.8%] 9 [||||||||||| 93.5%] 10 [||||||||||| 95.4%] 11 [||||||||||| 93.5%] 12 [||||||||||| 92.1%] 13 [|||||||||||| 96.1%] 14 [||||||||||| 94.0%] 15 [||||||||||||100.0%] 16 [||||||||||| 92.7%] Mem 8.56G/47.1G Tasks 95, 471 thr; 16 running Load 18.29 15.44 10.99 CPU% COMMAND 161. prebuilts/jdk/jdk21/linux-x86/bin/java -XX:OnError="cat hs_err_pid%p.log" … 125. java -XX:CICompilerCount=6 -XX:+UseDynamicNumberOfGCThreads … 99.9 prebuilts/rust/linux-x86/1.83.0/bin/rustc -C linker=out/host/linux-x86/bin/… 37.9 prebuilts/jdk/jdk21/linux-x86/bin/javac -J-Xmx4096M … … about 40 more JVMs
htop about 25 minutes into the compile phase. Dozens of JVMs (javac, plus the Java tools that generate API stubs and docs) and rustc keep all 16 threads busy.
What broke, in order
- Rate limiting during sync. One AOSP repository (
external/crosvm) failed with HTTP 429 from googlesource at-j8. Re-syncing that single project with-j1fixed it. - A private dependency. The device tree inherits a launcher package whose repository is no longer public. An empty
config.mkin its place removes it from the product cleanly. - glibc 2.34 required by a prebuilt.
prebuilts/extract-tools/.../stripzipis linked against glibc 2.34; Ubuntu 20.04 has 2.31. It only normalises zip timestamps during blob extraction, so a no-op stub keeps the files byte-identical to the vendor’s. The other extract tools and the compiler toolchain still run on 2.31. - EROFS partitions. The official /e/OS OTA ships EROFS images, which Ubuntu 20.04 cannot read. Building
fsck.erofsfrom the tree’s ownexternal/erofs-utils, statically linked against the in-treeexternal/lz4, took a minute and allowedfsck.erofs --extract. - Pinned firmware from an older release. Sixteen Wi-Fi firmware files (
vendor/firmware/qca6750/*) are pinned by SHA-1 to an older vendor release (FP6.QREL.15.176.0) that is no longer downloadable. The official /e/OS image contains exactly those hashes, so they were taken from there and verified one by one. - Bootloader firmware. 20 of the 24 radio images (xbl, abl, tz, hyp, …) in the vendor factory image differed from the ones the official /e/OS release ships. To avoid flashing anything older than what the phone already runs (anti-rollback), all 24 were taken from the official release. The generated
Android.mkchecks each radio file’s SHA-1, so those checksums had to be regenerated to match. - Missing HAL repository. The product requires
android.hardware.nfc-service.sec, provided byhardware/samsung_slsi/nfc, which is not in the manifest. LineageOS lists it in the device’s dependencies; adding it to a local manifest (branchlineage-23.0) let Kati finish. repoinside the build sandbox, at 96%. The rule that writesproduct/etc/build-manifest.xmlrunsrepo manifest, which keeps a JSON cache of the user’s git config in~/.repo_.gitconfig.json. When~/.gitconfigdoes not exist, repo considers the cache stale on every call and rewrites it, and the build sandbox mounts$HOMEread-only. Creating an empty~/.gitconfigand runningrepo manifestonce outside the build fixed it.- A presigned app in the image. Prebuilt APKs that keep their own signature and target API 30 or higher must be declared
preprocessed: true, so the build checks them (zip-aligned, uncompressed dex for a privileged app) instead of rewriting them and breaking the v2 signature.
Most failures surfaced during Kati, 2 to 7 minutes into a run. The sandbox one appeared after 7 h 52 min. Thanks to the incremental build, fixing it cost 10 minutes rather than a second night.
Takeaways
- A 16-thread Westmere box with 48 GB of RAM can still build AOSP 16. CPU age is not the blocker; memory and the host’s userland are.
- Give Soong’s analysis phase about 30 GB of headroom, or enable swap before the first run.
- Ubuntu 20.04 works as a host if you are ready to work around a few tools that expect glibc 2.34.
- Take radio firmware from the release the phone is running, not from the newest vendor image.
- Make sure
~/.gitconfigexists before a long run, even empty. - Use ccache from the start. The first build pays for every later one.
Build target: lineage_FP6-userdebug, /e/OS 4.3.1-a16, BUILD_ID BP2A.250805.005, AOSP 16. Built with breakfast FP6 userdebug && m bacon -j12, ccache enabled.
And you?
What does it cost you to build an AOSP-based OS: which machine, how many hours, how much disk? Which tricks and practices do you recommend to make it faster or less painful? Share your numbers and setups in the comments.
Gaël Duval
CEO at Murena, President at e Foundation
Written with the help of an AI assistant. The machine, the build and the ideas are mine.
