From early 2023 until the end of 2025 I built the ingest path for a smart indoor-farming platform: a thousand or so devices reporting over MQTT into a realtime pipeline on Google Cloud. Write path, retention, rollups, the lot.
The bugs I remember were not about the readings that arrived. Those are easy: you validate them, key them, store them. The bugs I remember were all about reasoning over readings that never arrived at all.
A gap in the data is two different facts
At 09:00 a device has no reading in the table. There are two entirely different explanations and they look identical.
The device was connected and had nothing to report. Or the device was on a flaky link, went dark, and buffered locally. Ours did exactly that: a device that lost connectivity would keep sampling, hold the readings, and replay them when it reconnected, sometimes hours later.
So a gap is not information. A gap is the absence of information, and the absence has at least two causes with opposite meanings. One is a fact about the world worth showing a user. The other is a fault worth alerting on. If your pipeline treats them the same, you will eventually show a user a confident chart of a period during which you knew nothing.
The tempting fix is a timeout: no reading for N minutes means offline. That is a guess dressed as a mechanism, and it is the same guess I have watched people make with dedupe caches. Too short and every slow poll looks like an outage. Too long and a genuinely dead device stays green for an hour.
Liveness belongs on its own channel
The fix is not a better threshold. It is a second channel.
A heartbeat is small, fixed size, on its own topic, at a cadence you choose rather than one the sensor dictates. It says nothing about temperature or position. It says only “I am here, and my clock reads this.”
With two channels the ambiguity collapses:
- Heartbeat present, payload absent. The device is healthy and has nothing to say. That is a real observation and you can render it.
- Heartbeat absent. The device is dark. That is a fault, and you know it without waiting for a payload that may never come.
- Heartbeat absent then a flood of payloads with old timestamps. The device was dark and is now replaying its buffer.
One signal that meant two things becomes two signals that each mean one thing. It costs a topic and a few bytes per interval, and it removes an entire category of wrong conclusion.
Every derived state needs an “as of”
The second thing I got wrong early was serving last-known state without saying when it was last known.
A dashboard that shows 21 degrees has told the user almost nothing. A dashboard that shows 21 degrees, as of four seconds ago, has told them something they can act on. The same number, six hours stale, is a different claim entirely, and the interface has no way to distinguish them unless the state record carries the time it was true.
So every state row carries two timestamps, not one. Reading time is when the device observed it. Ingestion time is when you received it. They are usually close and occasionally hours apart, and conflating them is how buffered replays quietly corrupt a timeline.
Staleness budgets then belong per field, not per device. A control loop watching temperature cares about seconds. A firmware version field is fine if it is a week old. Attaching one global freshness rule to a device makes both of those wrong.
What changes when the asset is allowed to move
Everything above I learned on devices that stayed where I put them. A grow room does not drive away. It is worth being precise about which parts transfer and which parts get harder, because I think the delta is the interesting bit.
On a fixed installation, location is a property you already know. It is in the config, not the telemetry. A gap hides readings, and that is all it hides.
Once the asset moves, position becomes the telemetry, and a gap hides displacement. That changes three things:
You cannot interpolate through a gap and call it a path. Drawing a straight line between the last point before a dark period and the first point after it produces a route the asset did not take, rendered with exactly the same confidence as a route it did. If the gap is unexplained, the honest render is a discontinuity, not a line.
A trip reconstructed across a dark period is a hypothesis. Trip boundaries, distance, and duration all derive from a sequence you know to be incomplete. Those derived values need to carry the same provenance as the state they came from: which segments are observed and which are inferred.
The buffered replay stops being merely inconvenient. On the farm, late readings meant a rollup had to be recomputed. On a moving asset, the replayed points have to be inserted into a sequence that has already been read, charted, and possibly acted on. The insert is not appending to the end of a log. It is editing the middle of a story someone has already been told.
Late data reopens conclusions, not just rows
This is the part that generalises furthest, and it is the same lesson I hit doing retention and rollups: if input can arrive after the window closes, then nothing derived from that window is ever final.
Suppose you emit an event: asset idle from 09:00 to 15:00. Then a device reconnects and delivers buffered points showing movement at 11:00. The event you emitted was wrong. It was reasonable given what you knew, and it was still wrong.
Append-only derived events cannot survive this. What works is treating a derived event the same way you treat a rollup: keyed on the thing it describes plus the window it covers, written as an upsert, recomputable from the raw sequence at any time. When late data lands you mark the affected windows dirty and recompute exactly those.
That has a consequence people skip. You cannot drop raw points for a window until you are confident nothing more is coming for it. Raw retention is not a storage decision, it is a bet on your worst-case outage, and the only way to size it is to observe your fleet rather than guess.
What I would tell someone building this
Separate liveness from payload before you write any alerting. One channel that means two things will generate a threshold you tune forever. Two channels that each mean one thing need no tuning at all.
Store reading time and ingestion time on every row. They are usually the same, they are occasionally hours apart, and every silent late-data bug I have chased lived in the gap between them.
Never interpolate across an unexplained gap. A rendered line is a claim. If you did not observe the middle, do not draw it with the same confidence as the parts you did.
Assume every derived event will be reopened. If a device can buffer, no window is final, and anything you emitted append-only will eventually be quietly wrong.