Skip to content
Picasso Soft
All posts

Shipping a 2.6 GB LLM inside a SQL client — no key, no network, no cloud

Sean Park · Picasso Soft8 min read

My SQL client DBeast has had bring-your-own-key AI for a while — OpenAI, Anthropic, Gemini, or your own Ollama box. But the users I most want to serve are the ones who can't use any of that: DBAs in environments where schema names can't leave the building, let alone query results. For them I built a third tier: an LLM that runs entirely on the phone. Airplane mode is a supported configuration.

This is what that actually took — model licensing, a 2.6 GB download you can't get wrong, native-engine lifecycle, and why the iOS build ended up with a completely different (and better) engine.

Picking a model you're legally allowed to download in-app

The blocker for on-device LLMs in consumer apps was never inference — it's distribution. Gemma 3-era models sat behind a gated Hugging Face license: users had to log in and accept terms, which kills any "tap Download" UX.

Gemma 4 changed that. The E2B instruction-tuned build is Apache-2.0 and ungated — an app may fetch it directly, no token, no login. That single licensing change is what makes this feature shippable at all. The artifact is a LiteRT-LM bundle: 2,588,147,712 bytes, and you pin its SHA-256 and never think about it again.

Runtime: com.google.ai.edge.litertlm:litertlm-android:0.13.1. Two build-system bruises worth writing down: it lives on Google Maven, not Maven Central (the Central path 404s and you'll question your sanity), and 0.13.1 ships Kotlin 2.3 metadata, so a Kotlin 2.1 compiler needs -Xskip-metadata-version-check to link against it.

A 2.6 GB download is a feature, not a fetch

At this size, every download edge case will happen to someone:

  • Resume, always. HTTP Range + a .part file. A dropped connection at 96% of 2.6 GB must not mean starting over — and it must not look like "never started": the UI says "interrupted at N% — tap to resume."
  • Verify, then promote. SHA-256 the completed file against the pinned hash, and assert the byte length, then atomically rename into place. The install check and the promote check must agree by construction, or a wrong size constant makes the app download 2.6 GB, delete it, and download it again on every launch. Forever.
  • The HTTP 416 brick. A completed-but-unverified file plus a naive Range request yields 416 Range Not Satisfiable in an infinite loop. On 416: stop downloading and verify what's on disk.
  • Delete must fight the pipeline. A user reclaiming space races the non-cancellable verify-and-promote tail. A @Volatile deleted flag checked through the promote path keeps a deleted model deleted.

The eligibility gate matters too: the model wants ~1.7 GB of RAM at runtime, so the download button only appears on devices with ≥6 GB RAM and 2 GB free storage headroom. Offering a feature the phone can't run is worse than not having it.

The engine is a resource, not a library call

LiteRT-LM's API is pleasant — Engine(EngineConfig(modelPath, backend = CPU)), createConversation(), a Flow<Message> — but the lifecycle has teeth:

  • initialize() takes ~10 seconds. Off the main thread, obviously, and you keep the engine alive for the process lifetime — cold call ~35 s, warm call ~13 s for NL→SQL on my Galaxy Tab. (Not GPT-fast. Fully offline. Users who need this tier accept the trade instantly.)
  • Delete must release the engine, not just the file. Freeing 2.6 GB of disk while a native engine holds ~1.7 GB of RAM — and keeps answering from the unlinked file handle — is a bug you only find by watching meminfo.
  • Key the engine cache by path + mtime. Delete, re-download, and a path-keyed cache happily serves the stale engine from the old bytes.
  • A failed initialize() still allocated native memory: close the half-built engine in the failure path or leak per retry.

APK math: the native .so files add ~11 MB per ABI (our AAB went 20 → 41 MB), but Play's per-ABI splits mean users download one. The model itself never touches the store — it's in-app, resumable, and deletable.

Then iOS made it weird

LiteRT-LM is an Android AAR. No iOS artifact. I was budgeting weeks for a llama.cpp integration when the obvious thing turned out to be true: Apple already ships the model. The Foundation Models framework (iOS 26) exposes the OS-resident ~3B model:

SystemLanguageModel.default.availability
LanguageModelSession(instructions: system).respond(to: prompt)

No download, no RAM gate, no lifecycle management — the OS owns all of it. The iOS implementation is a fraction of the Android code and strictly nicer for users: "on-device AI" is just on. Both platforms expose it as the same provider id, so a backed-up connection profile restores onto either OS with AI settings intact.

One real constraint: the Apple session gives you a 4096-token context. A wide schema plus a chatty prompt overflows it, so the on-device tier gets a trimmed schema digest that the cloud tiers don't need.

The asymmetry is worth naming: on Android you ship the model (2.6 GB, licensing, integrity, RAM); on iOS the OS ships the model (zero bytes, but you inherit its context window and its availability matrix). Neither is better engineering — but they're different jobs, and only one of them is mostly about downloads.

The rules I'd reuse

  1. License-check the distribution, not just the weights. Apache-2.0 + ungated is the whole ballgame for in-app download.
  2. Treat a multi-GB asset as a state machine (absent → partial → verifying → ready), and make every state's criteria agree with every other's.
  3. Native engines: lifecycle first, inference second. RAM release, cache keys, and failure-path cleanup are where the bugs live.
  4. On iOS, check what the OS already gives you before porting an inference stack.

Try it in DBeast (Android; iOS on the App Store) — Settings → AI Assistant → On-device. Download once, then turn on airplane mode and ask your database a question.