Choosing a Model for On-Device Healthcare AI

Introduction

"What is the best open language model?" is one of the most common questions in AI right now, and it is the wrong question to start with, at least if what you are building is a mobile application.

Leaderboards answer a different question than the one a product team actually needs answered. A model can top every public benchmark and still be the wrong choice for your app, because benchmarks are run on servers with abundant RAM, multiple GPUs, and no battery to drain. Your users' phones have none of those luxuries.

The better question, once AI moves from the cloud onto a device you don't control and can't upgrade, is: what is the best model for my constraints? That question has a different answer for every product, because every product has a different shape, different latency tolerance, different memory budget, different tolerance for being wrong.

Everything attributed to FarmakoMed in this article was checked against the current app codebase, not assumed. Where the implementation is still in progress or unverified on real hardware, that is stated explicitly, because an engineering journal that overstates its own product isn't a trustworthy one.

Cloud Thinking vs. Mobile Thinking

Most of the intuition engineers have about language models was built on cloud infrastructure. It's worth naming those assumptions explicitly, because they quietly stop being true the moment inference moves onto a phone.

In the cloud, RAM is elastic, if a model needs more, you provision a bigger instance. GPUs are plentiful and shared across requests. Scaling is a matter of adding nodes. The model can be updated by redeploying a service, with no user action required. And the assumption of continuous connectivity is so deep it's rarely stated at all.

None of that holds on a phone. RAM is fixed at the moment the user bought the device, and the operating system will reclaim it from your process without asking. There is one battery, and every millisecond of GPU or NPU time draws it down. There is no horizontal scaling, only the silicon in this one pocket. Updating the model means the user's device has to download it, which takes time, uses their data plan, and can fail halfway through. And connectivity is not continuous; it is intermittent, and sometimes absent entirely.

FarmakoMed's own device compatibility rules make this concrete. The app's LiteRT-LM runtime dependency itself requires Android 9 or newer, but FarmakoMed's local-AI gate is stricter than that: a minimum of 8 GB of RAM (12 GB+ recommended), free storage equal to the model size plus a 2 GB buffer before download is even offered, and, rather than trying to probabilistically detect "is this device fast enough", an explicit allowlist of validated flagship devices. A device that isn't on that list can still install the app and use every non-AI feature; it simply doesn't get offered local AI. That is a mobile-thinking decision a cloud-thinking team wouldn't need to make at all, because in the cloud, every request runs on the same hardware.

The Dimensions That Matter

"Best model" collapses a dozen independent variables into one word. Pulling them apart doesn't tell you which model to pick. It tells you which questions to ask before you pick one.

Quality and latency: two different clocks

Model quality, how often the output is correct, how well it follows instructions, how it handles ambiguous input, is the dimension everyone starts with, and the one benchmarks measure. But quality only matters if the answer arrives inside the window a user will tolerate. A model that's 3% more accurate but takes four times longer to respond is not obviously a better product decision; it depends entirely on what the feature is for.

FarmakoMed's own performance targets separate these explicitly: a hard 60-second timeout on local inference, with a target of under 20 seconds on flagship devices for the full extraction pipeline. Those numbers weren't derived from a model card. They came from deciding what a user standing in a pharmacy is willing to wait for.

Memory, storage, and the anti-pattern of bundling

A vision-language model resident in memory has a real cost, during FarmakoMed's own evaluation of using its on-device model for document text extraction, that engine's peak memory footprint was measured at roughly 2–4 GB, a number large enough on its own to explain why the team ultimately chose a different, purpose-built engine for that job (more on that below). Multiply that by whatever else is running, camera preview, image preprocessing, the JavaScript runtime, and memory pressure becomes the thing that decides whether inference completes or the OS kills the process mid-scan.

Storage is a related but separate constraint. A multi-hundred-megabyte-to-multi-gigabyte model artifact cannot reasonably ship inside an app bundle without inflating install size for every user, including the ones on unsupported devices who will never use it. That's why FarmakoMed's engineering constraints list "bundle large model into APK" as an explicit anti-pattern, and why the model is instead fetched at runtime through a versioned manifest, verified by SHA-256 checksum, over a resumable HTTP download, the same category of engineering that a podcast app uses for downloading episodes, applied to a neural network.

Battery, startup time, and thermal behaviour

Inference is computationally expensive, and computation costs battery and generates heat. A feature that's technically correct but leaves a phone noticeably warmer after a thirty-second scan will register to the user as something being wrong, even if the output was accurate. Startup time compounds this: loading a multi-gigabyte model into memory from cold storage is not instantaneous, and a user who opens a medication-scanning feature expecting a camera viewfinder and instead gets a multi-second blocking wait will interpret that as a bug.

Quantization and hardware acceleration

Quantization, reducing the numerical precision of a model's weights, is what makes multi-billion-parameter models feasible on a phone at all. FarmakoMed's active model ships as a .litertlm artifact, Google AI Edge's packaged, pre-quantized container format built for the LiteRT-LM runtime; an earlier candidate the team evaluated (Gemma 3n, since retired, see below) shipped as an explicitly int4-quantized .litertlm build, which gives a concrete sense of the precision trade-offs this artifact format is designed around.

Hardware acceleration is where the gap between "supported in theory" and "verified in practice" is widest. FarmakoMed's runtime can request four backends, CPU, GPU, NPU, or Google's AICore, but as of this writing, AICore is defined in the abstraction and not implemented in the current bridge, and NPU delegation remains opt-in pending a dispatch-provider library that isn't yet bundled. When GPU or NPU delegate initialisation fails, the runtime falls back to CPU and records why. That's a deliberately unglamorous but honest position: request acceleration, but don't assume it happened just because you asked.

Platform reality: Android and iOS are not the same runtime

"Cross-platform" is a UI-layer promise more often than a hardware-layer one. FarmakoMed's Android inference bridge uses the litertlm-android Gradle package (version 0.11.0 at the time of writing) with GPU acceleration through AICore or NNAPI. iOS uses a separate Swift package, the same underlying LiteRT-LM C runtime and the same .litertlm model file, which meant no model conversion or duplicate hosting was needed, but accelerated instead through Apple's Metal Performance Shaders, and, as of the team's iOS SDK evaluation, still labelled Early Preview rather than Stable. The team also evaluated and rejected Google's MediaPipe Tasks GenAI for iOS: it doesn't support the model family FarmakoMed uses, it's text-only with no image input (which rules out a medication-packaging camera feature outright), and Google's own documentation now recommends migrating away from it toward LiteRT-LM. Two platforms, two SDKs, two acceleration paths, one model file, and a genuine maturity gap between them at any given point in time.

Licensing and long-term maintainability

A model license and a software license are not the same kind of commitment. FarmakoMed's supporting tooling, its OCR engine, its PDF text extraction library, is Apache 2.0, a license that imposes no copyleft or commercial restriction. The model weights themselves carry Google's Gemma model license, which permits commercial use but is a separate document from the code license, worth re-checking with each model version upgrade rather than assumed to carry forward automatically.

Maintainability shows up architecturally, not just legally. FarmakoMed keeps a multi-model registry abstraction in place even though, at any given time, only one model profile is actually active, because the team has already needed to swap the active model once, and designing for that possibility in advance turned a potential rewrite into a configuration change.

Healthcare Changes Everything

A general-purpose assistant is judged partly on how interesting or creative its answers are. A healthcare app is judged almost entirely on whether it got the medication name right. That difference should change how you configure a model, not just which one you pick.

FarmakoMed's default inference preset for medication extraction sets temperature to zero and top-k to one, the decoding parameters that make output as deterministic as a language model can produce, rather than sampling for variety. That is the opposite tuning direction from a creative-writing or brainstorming assistant, and it's a direct consequence of the product domain: for "what is this medication and what strength is it," predictability is worth more than fluency.

Explainability matters here in a specific, narrow sense: not that the model can explain its own reasoning, but that the app can always show its work. Every AI-generated summary in FarmakoMed's document-processing pipeline carries a non-dismissible disclaimer and sits alongside, never in place of, the original extracted text and the original document image. The product design assumes the model will sometimes be wrong, and builds the escape hatch to the source material in from the start, rather than treating model output as a replacement for it.

There's a subtler consistency requirement too: production UI copy in FarmakoMed never surfaces the words "Gemma," "model," "inference," or "AI" to the end user directly, those terms are reserved for logs, code, and debug builds. What the user sees instead is "Checking medication information" or "Processed on your device." That's not obfuscation; it's a recognition that a patient looking at a medication result needs to trust the outcome, not audit the implementation. The technical honesty lives in the engineering documentation and the diagnostics, like this article, not in the product surface.

There Is No Perfect Model

Every model family makes trade-offs along the dimensions above, and every choice closes off some options while opening others. FarmakoMed's own model-selection history is a useful, unglamorous illustration of this, because it involves a model that was tried and then removed.

An earlier candidate, Gemma 3n E2B, was in FarmakoMed's active model registry before the team's own reliability testing led to its removal in favour of Gemma 4 E2B, with an automatic migration path for any user who had already installed the older profile. The multi-model registry abstraction wasn't theoretical scaffolding; it's the exact mechanism that made that swap possible without a full rewrite.

Model size within a single family is its own trade-off, not a strictly-better-if-bigger ladder. Gemma also ships a larger E4B variant, and FarmakoMed's tooling can resolve its release metadata, but the active production profile is the smaller E2B. Larger doesn't automatically mean "not yet used because of resource cost" either: as of the iOS SDK's Early Preview release the team evaluated, E4B's multimodal path was documented as broken on iOS specifically, while E2B was unaffected. Sometimes the smaller model in the family isn't just cheaper to run. It's the only one that currently works on the platform you need to ship on.

Lessons from FarmakoMed

These are engineering lessons the FarmakoMed team has direct evidence for, not aspirational best practices. Where the implementation is still incomplete, that's stated rather than smoothed over.

OCR is not an LLM problem

The most concrete lesson in this codebase concerns a decision not to use the vision-language model already resident in the app. When FarmakoMed needed to extract text from scanned medical documents, the obvious-seeming option was to reuse the same Gemma multimodal model already loaded for medication-package recognition. The team spiked it, measured it, and rejected it as the primary engine: roughly 2–4 GB peak RAM and multi-second-per-page latency, against a purpose-built alternative, Google's ML Kit text recognition, that added roughly 12 MB to a production Android app bundle and returned results in 242–305 milliseconds on real device hardware (a Galaxy S24 Ultra), fully offline, with zero network calls even when Google Play Services itself had no internet reach.

Reliability beats capability

The Gemma 3n → Gemma 4 swap described above wasn't driven by a benchmark score. It was driven by real-world reliability behaviour the team observed while testing, a reminder that a model's published capabilities describe what it can do under evaluation conditions, not what it will do in your specific pipeline, on your specific input distribution, running through your specific runtime bridge.

Warmup is a lifecycle decision, not a spinner

FarmakoMed triggers a model "warmup" step after the user has accepted the disclaimer, the device has passed compatibility checks, and the model is confirmed installed, but that warmup currently loads the runtime only; it deliberately does not run a real inference pass, specifically to avoid generating synthetic medication-like output during startup. Camera-based scan input stays locked until warmup finishes, and if warmup fails or times out, the app unlocks scanning anyway and shows a recoverable warning rather than leaving the user stuck. That's a small piece of UX, but it encodes a real lifecycle decision: never let model-loading mechanics block a user from a workflow they can still complete a different way.

Diagnostics that refuse to guess

FarmakoMed's own runtime documentation states the policy directly: don't claim performance without real-device evidence. That shows up in the diagnostics payload every inference run returns, fields like whether acceleration was supported, available, and actually used are each allowed to be true, false, or null, where null means the bridge genuinely cannot determine the answer from the current runtime API surface. It would be easy to default an unknown value to "false" and move on. Preserving the third state, "we don't know", is a small design choice that keeps the diagnostics honest instead of confidently wrong.

The Future of Mobile AI

Every constraint described in this article is a snapshot of late-2026 mobile hardware and software, not a law of nature. Dedicated NPUs are already shipping in flagship phones, but the software layer that lets an app reliably dispatch work to them is visibly still catching up, FarmakoMed's own runtime treats NPU delegation as opt-in and gated behind tooling that isn't fully in place yet, and Google's own AICore backend is defined in the abstraction layer before being implemented in the bridge. That gap between "the silicon exists" and "an app can reliably use the silicon" is exactly where the next couple of years of mobile AI engineering will happen.

On iOS, Google's LiteRT-LM team has reported roughly 56 tokens per second of decode throughput via Metal acceleration, a figure FarmakoMed's own engineering documentation cites from Google's public benchmarks, not from in-house measurement, and worth treating with exactly that provenance in mind until independently verified on target hardware.

The practical implication for engineers is not "wait for the hardware to mature". It's "build the abstraction layer now that lets you adopt better hardware paths later without a rewrite." A multi-model registry, a backend-selection layer that can request CPU, GPU, NPU, or a not-yet-implemented AICore path and gracefully fall back, and diagnostics that report what actually happened rather than what was requested, none of that requires the hardware to be ready today. It requires deciding, today, that the hardware situation two years from now will not look like it does now.

Engineering Decision Matrix

None of this reduces to a single "best" model. It reduces to a set of questions a product's constraints answer for you. This isn't a scorecard for choosing between named models. It's a starting framework for figuring out which constraint your product can least afford to compromise on.

Notice what's missing from that table: a "winner" column. That's intentional. The right-hand side of each row is a direction, not a destination, the actual model that satisfies it will differ by product, by platform, and by the month you're reading this.