Diagnostics Without Telemetry

Introduction

A privacy-first, on-device product makes a promise that sounds simple: your data stays on your device, nothing phones home. Living up to that promise is one kind of engineering discipline. Debugging a product that lives up to it is a different one, and it is the one nobody warns you about.

Somewhere out there, on a phone you will never see, running an OS version and a chipset combination you did not test, something goes wrong with an on-device AI feature. In a conventional architecture, this is a Tuesday: a crash reporter phones home with a stack trace, a session replay tool shows the exact taps that preceded it, an analytics event tells you it happened to four hundred other people this week too. You get a dashboard. You get a number. You get evidence.

None of that exists here, on purpose. There is no crash SDK quietly attaching a device snapshot to a report. There is no session replay, because a session replay of a medication list is a data breach with a nicer name. There is no background analytics stream telling you what the model just did with someone's health information. The architecture that makes the product trustworthy is the same architecture that makes it hard to see inside from the outside. This article is about what fills that gap, not a workaround for the absence of telemetry, but a different discipline built for a system that was never going to have it.

The Gap Between Private and Invisible

"We don't collect telemetry" and "we can't debug this" get conflated constantly, and they are not the same sentence. The first is a policy, chosen on purpose, for good reasons. The second is an operational failure that policy should never be allowed to cause. Treating them as the same thing is a category error, and it is a comfortable one, because it lets a team quietly lower its own diagnostic bar and call the result a privacy feature.

The honest framing is narrower and harder: a privacy-first architecture removes one tool, the ability to observe a live system from outside it, and leaves every other diagnostic discipline exactly as necessary as it always was. The bugs do not become rarer because you stopped watching for them. They become quieter. The work that a cloud dashboard used to do for free now has to be designed, deliberately, into the device.

Four Kinds of Bugs Need Four Kinds of Evidence

The instinct behind most telemetry pipelines is to collect broadly and decide later what mattered. That instinct is exactly backwards for a system with a privacy boundary, because it treats every bug as if it needed the same evidence, when in practice the bugs that show up around on-device AI split into at least four categories, and they have almost nothing in common:

• Crash and hard-failure bugs. The process dies, an exception escapes, a native library aborts. What you need is a stack trace, an error code, and device context, none of which requires a single byte of the user's actual data.
• Correctness bugs. The model runs fine and produces an answer that looks wrong. What you would need to fully diagnose this is the exact input and output pair, which is simultaneously the single most useful piece of evidence and the single most sensitive one a health app can hold.
• Performance bugs. Inference is slow, memory climbs, the battery drains faster than it should. What you need is timing and resource traces, mostly free of content, but still not something that should leave a device by default.
• Environment-specific bugs. A feature fails only on one chipset, one OS version, one manufacturer's background-process policy. What you need here is device diversity, not user data at all, this is a coverage problem, not an observability one.

Lumping these four together into one "collect everything, sort it out later" stream is precisely the design that a privacy-first product cannot afford, and it is worth noticing that it was never actually the right design, it was just the convenient one. A generic crash-and-analytics SDK does not ask which category a given event belongs to before it uploads it. It invites exactly the over-collection this architecture exists to avoid, because the category that most needs restraint, correctness bugs carrying real content, is the one hardest to keep out of a stream built to catch everything else.

A Bounded, Local, Structured Log

Most of what a crash-reporting SDK gives you is not, in fact, off-limits. Model load time, a state-machine transition, an error code, a coarse device class, these are operational facts about the software, not facts about the person using it. The problem with conventional logging is not that logs are inherently unsafe, it is that they are usually unbounded, unstructured, and eventually contain whatever a developer typed into a debug statement at 2 a.m., which is exactly the moment nobody is thinking about what belongs in a permanent record.

The fix is a log designed with the same discipline as any other outbound-adjacent artifact: bounded, so it cannot quietly become a shadow database of everything the app has ever seen, a fixed-size ring buffer that overwrites its oldest entries is enough. Structured, typed events with a fixed shape, not free-text strings, so a field cannot silently start carrying a symptom name or a drug name because a string got interpolated into a debug message. And local by default, existing to answer "what state was this device in a moment ago," not to accumulate into anything that resembles a permanent history.

What's Safe to Record, and What Never Is

A structured log still needs a firm line between fields that describe the software and fields that could describe the person. The test is not "does this feel sensitive," it is a simple, mechanical question: could this field, ever, under any circumstance, hold something the user typed or a value derived from their health data? If yes, it does not go in the log, full stop, not truncated, not summarised.

Determinism Is a Diagnostic Feature

The less raw evidence you are willing to collect, the more your ability to reproduce a bug depends on the software behaving the same way twice given the same shape of input. That is a deliberate case for determinism, not as a code-quality nicety but as a diagnostic requirement: if a feature's behavior is a predictable function of an explicit state machine, a bug report reading "it happened after backgrounding for over two minutes, on the third scan" is often enough to reproduce it, on your own device, with your own synthetic data, no field data required.

This is where a log built from typed state transitions earns its keep. It does not need to record what the user typed, only the shape of it, empty string, unusually long input, a duplicate entry, non-Latin characters, because the shape is usually what triggers the bug, not the specific content. A codebase held together by implicit control flow and ad-hoc callbacks cannot make this trade, its behavior is not a clean function of state, so nothing short of the original data will reproduce what happened. An explicit state machine can, and that is the diagnostic dividend of the lifecycle discipline this Journal has argued for elsewhere.

Self-Tests: Asking the Device to Check Itself

The other half of local diagnostics is not waiting for a bug report at all. A small suite of self-checks, run at safe, predictable moments, app start, returning from the background, immediately before invoking the model, can assert a handful of narrow invariants: is the model instance still valid, is available memory above the threshold this feature needs, did a resource that should have been released after the last use still show as held.

Each of these is a bounded yes-or-no question, and the answer is a code, not a document. That reframing matters. "Did anything go wrong" is an open-ended question that seems to demand rich, unbounded telemetry to answer well. "Is invariant #12 currently true" is not. A self-test suite turns an open diagnostic problem into a small, enumerable one, and every invariant you can name in advance is one less thing you need field data to discover after the fact.

graph LR
    T[Self-test runs] --> L[Bounded local log]
    L --> A[Anomaly detected]
    A --> U[User prompted]
    U --> E[Consented export]

One Door for Diagnostics, Too

A privacy-first architecture should already treat every outbound path as a deliberate, singular pipeline rather than a collection of one-off exits, backups, reports, and shares each earning the same scrutiny at the same gateway. A diagnostic bundle is exactly this kind of exit, and it does not deserve a side door of its own just because an engineer is on the receiving end instead of a doctor or an insurer.

In practice that means a diagnostic export is assembled only on an explicit action, "something's wrong, help me report it," never automatically and never silently. It is built exclusively from the structured local log and self-test results, the same bounded, content-free record described above, never from raw model input or output. And critically, it is shown to the user before it is sent, in full, so "here is literally everything this report contains" is a sentence the interface can make true, not just claim.

Reproducing What You Can't See: Device Labs and Fuzzing

Restricted field visibility only works if more of the burden shifts left, into the time before a release goes out, because that is the one window where you still have full access to real hardware and real data you generated yourself. Three practices do most of that work.

A physical device lab, covering the chipset classes, OS versions, and RAM tiers that actually matter for the install base, exists to catch category four from the taxonomy above, the environment-specific bug, before it ever reaches a device you cannot inspect. This is precisely the class of bug a cloud crash dashboard would otherwise have surfaced cheaply and by accident; without one, it has to be found on purpose, in a lab, on hardware you own.

Property-based and fuzz testing of the state machine's input space stands in for category two, correctness bugs, which can never be reproduced from a field report containing real content because that content is exactly what must not be collected. Generating a wide, synthetic spread of inputs, empty, oversized, malformed, adversarial, against your own deterministic state machine finds the same class of failure a real user's odd input would have triggered, without ever needing that user's data.

Chaos-style tests in CI, simulated memory pressure, forced backgrounding, killed and resumed processes, are cheap to run before release and expensive to diagnose blind after one. Every scenario reproduced this way in a lab is one less mystery that has to be pieced together later from a handful of state codes and a cooperative user's willingness to help.

None of this replaces the field entirely. A user's own bug report remains a real, valuable signal, it just arrives as a qualitative account cross-checked against a local log the user can inspect and choose to share, rather than as a stream the device was already sending before anyone asked.

The Cost of Not Knowing

It would be dishonest to present any of this as free. There is a real class of bug, the one-in-fifty-thousand-devices issue, the interaction that only shows up after eleven days of uptime on one specific firmware build, that a cloud crash dashboard would have surfaced in aggregate by lunchtime. Without that aggregate view, the same bug can sit unnoticed for a long time, discovered only when a user happens to hit it, happens to notice, and happens to be willing to help reproduce it.

A Note From Practice

This is a live constraint on FarmakoMed, a local-first health platform where medication data, documents, and AI-assisted features run primarily on the user's own device. There is no server-side copy of what the model was asked and no crash dashboard quietly filling in with a stranger's health data whenever something goes wrong. That absence is not a gap in the engineering, it is the point of the engineering, and it means the discipline in this article, bounded local logs, self-tests, deterministic reproduction, a consented export, is not an alternative to real diagnostics. It is what real diagnostics has to look like when the alternative was never actually on the table.