How I built a fully offline AI assistant on Android with Gemma 4 E2B

This is a submission for the Gemma 4 Challenge: Build with Gemma 4.

What I built

PocketClaw is an Android assistant that runs entirely on your phone. You can chat with it, talk to it (press and hold the mic, it transcribes live), show it photos, hand it a PDF and ask questions about it, or tell it to turn on the flashlight, set an alarm, open the dialer, send an SMS, drop something on the calendar, search the web, or fire a notification. All of that runs on a 1.5 GB model that lives on the device.

Once the model is downloaded the first time, you can switch on airplane mode. Nothing breaks.

I built it solo, 17 days, for this challenge.

Repo: github.com/ManojRakshu/pocketclaw (MIT)

Why on-device and why E2B

I've been building agents on cloud LLMs for about a year and a half. Claude mostly, GPT-4 for a few things. Every agent I've put in front of users has had the same set of problems sitting behind it. Latency adds up when you're chaining calls. The cost per call gets real once you have real traffic. And the whole thing stops the second the network goes down.

Phones are interesting because they fix all three of those at once. Model lives on the device, so there's no per-call cost. No network in the loop, so latency is just silicon. And the network can disappear without anything breaking.

The thing that constrains you is RAM. A mid-range Android phone gives an app something like 1.5 to 2 GB of usable working memory before the OS starts pushing back. That's enough to rule out most of the Gemma 4 family:

• E2B at about 1.5 GB INT4. Fits. Has vision built in. This is what I shipped.
• E4B at about 2.5 GB. Tight on high-end phones, OOMs on lower-end ones.
• 26B MoE. Workstation.
• 31B dense. Server.

I built and tested on a OnePlus Nord CE 4 (Snapdragon 7s Gen 3, Adreno GPU, 8 GB RAM, Android 14). First-token latency is around 1 to 3 seconds for chat, around 5 for vision. Slower than cloud. But there's no network in the way.

The 1.5 GB number is worth being precise about. Full precision E2B (fp32) is roughly 20 GB. fp16 is 10. INT8 is 5. INT4 with Google's litert-lm packaging gets you down to 1.5 GB. Same precision class as what Google ships in Pixel's Gemini Nano. For phones, INT4 is the only answer that makes sense.

How it's put together

Three layers.

The model. I'm using flutter_gemma, which wraps Google's MediaPipe LLM API and LiteRT-LM on Android. I picked it because among the Flutter plugins I evaluated, it had the most mature support for Gemma 4's vision input. There's a cost to this. The MediaPipe stack adds about 80 MB of native libraries to the APK. Lighter, text-focused alternatives (llama.cpp wrappers, etc.) can ship at 30-60 MB. My release APK is 152 MB, down from 185 after I trimmed image-generation and WebGPU runtimes via Gradle (the plugin bundles them, we never use them):

// android/app/build.gradle.kts
packaging {
    jniLibs {
        excludes.addAll(listOf(
            // We never generate images, only consume them.
            "**/libimagegenerator_gpu.so",
            "**/libmediapipe_tasks_vision_image_generator_jni.so",
            // WebGPU is for browsers, useless on Android.
            "**/libLiteRtWebGpuAccelerator.so",
            "**/libLiteRtTopKWebGpuSampler.so"
        ))
    }
}

If you don't need vision you can probably get below 80 MB by switching to a text-focused stack. I needed vision so I'm at 152.

RAG on-device. A second model handles embeddings. Gecko 110M, around 110 MB on disk. I went with Gecko over EmbeddingGemma 300M because Gecko is roughly 3x smaller and the retrieval quality on PDFs of a hundred pages or less was comparable. Could be different at larger corpora. The pipeline is Syncfusion for PDF extraction, my own chunker (paragraph split, merge tinies, sentence-aware sub-split for anything still over the threshold), Gecko for embedding, sqlite-vec with HNSW for the vector store. All on device.

Device actions. Gemma's job here is intent classification. The user types or says "set an alarm for 7:30 AM". Gemma emits a structured JSON object that identifies the tool and parameters. Dart parses it. A native Kotlin MethodChannel (pocketclaw/device) fires the right Android intent. Eight categories work this way. Flashlight via CameraManager.setTorchMode. Alarms via AlarmClock.ACTION_SET_ALARM. Dialer via ACTION_DIAL. SMS via ACTION_SENDTO. Calendar via ACTION_INSERT. Location settings panel. Web search by handing the query to the default browser. Local notifications via NotificationManager. Nothing in the loop touches the network. The LLM doesn't even know the network exists.

Things that broke

RAG dies on generic queries

Vanilla RAG works fine for specific questions. Someone uploads a PDF about PocketClaw, asks "who built PocketClaw", retrieval picks up a chunk that contains my name, Gemma summarises, done.

It falls over on the queries people actually type. I caught this Friday afternoon. I'd shipped what I thought was a working build. I uploaded llmaiedge.pdf (a PDF about edge LLMs I had lying around), typed "summarise the document", hit send. Claw answered with "Summarize the document." That's it. I tried twice more with different phrasing. Same answer. Eventually I typed "summarise llmaiedge.pdf" and got a real response. The filename was doing the work, not my retrieval.

The problem is that "summarise this doc" has no semantic overlap with the actual document text. The doc doesn't contain the words "summarise" or "this doc." Cosine similarity returns nothing useful, the prompt goes to Gemma with no real context, and Gemma fills in with whatever its training data feels like saying about generic documents.

The fix runs two heuristics:

final isGenericIntent = hits.length <= 1 && (
    lower.contains('summari') ||
    lower.contains('tldr') ||
    lower.contains('explain') ||
    lower.contains('describe') ||
    lower.contains('the document') ||
    lower.contains('the pdf')
    // ... a few more
);

if (isGenericIntent) {
    // Fall back to searching with each indexed doc's filename as the query.
    hits = await RagService.instance.getDocStarts(
        conversationId: _conversation.id,
    );
}

getDocStarts runs searchSimilar once per indexed doc, using the filename itself as the query. Filenames are rare distinctive tokens. Every chunk in the vector store has the filename in metadata. So this pulls back real chunks regardless of how the user phrased their question. Two lines of conditional logic, and the difference between a broken demo and a working one.

Small models drop facts buried mid-prompt

I wanted Claw to remember the user's name. Onboarding asks for it, prefs stores it, the system prompt includes it. User asks "what's my name?" Claw says "I do not know your name."

The first version of my system prompt looked like this:

You are Claw. You run locally and offline. You are talking to Manoj Shetty.
Match your answer length to the question. Prefer plain answers over preambles.
Never restate the question. If unsure, say so briefly.

The third sentence has the name. Gemma 4 E2B dropped it completely. I burned about an hour staring at this before I figured out what was happening. My theory is that "never restate the question" was acting as a dominant instruction that generalized to "don't reference user context at all." Small models do that. Cloud LLMs don't.

The fix was to move the fact to the top of the prompt:

final namePart = (userName != null && userName.trim().isNotEmpty)
    ? "The name of the user is ${userName.trim()}.\n\n"
    : '';
final systemPreamble = '${namePart}You are Claw, ...';

Same information. First line of the prompt, on its own, in a flat declarative sentence. Worked first try.

Lesson I've now learned twice. With small models, what you want the model to know goes at the front, in simple sentences, without competing instructions next to it. Cloud LLMs respect the whole prompt. 2B models don't.

Audio in flutter_gemma is framed for Gemma 3n, not Gemma 4

I wanted to skip the speech-to-text plugin entirely. Just feed audio bytes directly to Gemma 4 E2B's audio modality. The model card says E2B supports audio. Cleaner architecture, one fewer dependency.

I went to dig into flutter_gemma v0.15.1 source. Eight different files in the plugin frame audio as a Gemma 3n E4B feature, including this in the interface:

/// [supportAudio] — whether the model supports audio (Gemma 3n E4B only).
bool supportAudio = false,

The example app makes the real constraint clearer: .litertlm files that bundle TF_LITE_AUDIO_ENCODER work with supportAudio: true, .task files don't. So the actual limit isn't the Gemma version — it's whether your specific model file contains the audio encoder. I haven't found a Gemma 4 E2B .litertlm build that bundles the audio encoder, and the plugin's docs treat audio as 3n-territory.

So PocketClaw uses Android system STT via the speech_to_text package, which is a wrapper around RecognizerIntent. Side benefit: I get live transcription as the user speaks, which is genuinely a better UX than the "hold, release, wait three seconds, see both the transcription and the response" pattern you'd get from on-model audio. When a Gemma 4 E2B .litertlm build with audio encoder ships, the voice path collapses into a single multimodal call. Not blocking on v1.

Long chats and memory

Gemma 4 has a 128K context window. That's plenty in theory. In practice, every token costs latency and RAM, so I'd rather not feed the whole history every turn.

PocketClaw keeps the most recent 24 messages in full text. Anything older runs through a compaction pass:

1. Extract facts the user has stated explicitly ("I am X", "My name is Y", "Remember Z").
2. Capture unresolved goals (keywords like "fix", "todo", "issue").
3. Compile the whole thing into a single lightweight summary paragraph that gets prepended to the prompt as memory.

The more aggressive part: when an image message slides past the 24-message boundary, the raw image bytes get deleted. What stays is the assistant's prior textual description of that image:

String _imageMemoryFromAssistant({
  required String? imageName,
  required String assistantText,
}) {
  final label = imageName ?? 'uploaded image';
  return 'Assistant previously described $label as: ${_shorten(assistantText, 1000)}';
}

Claw still "remembers" what it saw, but without the bytes weighing on the prompt. This matters more than it sounds like it should. A 1 MB photo as base64 is around 30K tokens. The textual description of the same image is around 100. So this is roughly a 300x compression of image memory, with surprisingly little loss for the kinds of follow-up questions users actually ask.

Stack

Flutter 3.41 / Dart 3.11. Chat model is Gemma 4 E2B INT4 at 1.5 GB, loaded through flutter_gemma on top of MediaPipe LLM and LiteRT-LM. Embedder is Gecko 110M (110 MB) for RAG. Vector store is sqlite-vec with HNSW, on device. Speech-to-text is the system STT via speech_to_text. Eight device actions go through a Kotlin MethodChannel. Hive for state (separate boxes for conversations, documents, prefs). syncfusion_flutter_pdf for PDF extraction. The theme is custom neobrutalistic dark with hard borders, monospace type, and cyan/purple/mint accents. I wanted it to feel like a piece of hardware, not a Material 3 chat app.

What didn't make v1

A floating overlay bubble (Android 13+ accessibility-style overlay) was in the original design. I cut it because the overlay permission UX has a long tail I didn't want to ship rough.

Direct Gemma 4 audio input. Waiting on a Gemma 4 E2B .litertlm build with the audio encoder. v2 when that's available.

Per-message RAG toggle. Right now if you have a document attached, every message in that conversation retrieves against it. Sometimes users want to ask an unrelated follow-up without doc context. v2.

Chunk citations in responses. Claw answers from retrieved chunks but doesn't surface which chunk. The retrieval data is sitting there. It's a UI add. v2.

What I learned

Most "run an LLM on your phone" tutorials I've read this year stop at model loading. flutter_gemma handles loading in about 10 lines. That part is easy.

The work is in everything around the model. The compaction so long chats don't OOM. The RAG fallbacks for queries that don't fit the textbook similarity-search assumption. The native channels for actual device actions. The Gradle excludes for plugin libs you don't need. The prompt structure that gets a 2B model to follow instructions reliably. The error handling for when the model file is half downloaded and the user opens the app anyway.

What surprised me about Gemma 4 E2B, coming from cloud models, is how capable a 2B model can be when it's running fast on your own device. Vision captioning is genuinely useful. Intent classification across 8 tool categories works well enough to ship. There's no perceived "AI thinking" delay because there's no network. The model speaks at the speed of your phone.

For a 1.5 GB download, that's a real deal.

Repo: github.com/ManojRakshu/pocketclaw

License: MIT

Tested on: OnePlus Nord CE 4 (Snapdragon 7s Gen 3, 8 GB RAM, Android 14)

Demo recorded: May 24, 2026

If you want to look at the design system, RAG service, or compaction engine, the source is fully open. PRs welcome.

Editorial assistance from Claude. All code, decisions, bugs, and engineering choices in this post are mine.