Telemetry where the link drops for hours

A dropped link is a normal state

The first decision, on which everything else depends: the site must operate on its own, without the server. The terminal takes the payment, the controller starts the bay, events are written locally. The server is where the data arrives, not where it is born. If the site cannot work without the server, no telemetry will help; it will simply record the downtime.

In the car wash network we support, the terminals are built exactly this way: they take payments without a link and send the transactions on when the channel returns. 14 million payment transactions have passed through them, a share of them by exactly this route.

The second decision: data must be complete, not fresh. Five missing minutes of readings do not matter. One missing payment is money that will not reconcile. So the event queue must be reliable for payments, and may be lossy for readings that are refreshed every minute anyway.

The buffer: what to keep and how much

The local buffer is not a structure in memory that disappears on reboot, but storage on disk that survives a power cut. On site, power fails more often than the link, and precisely at the moment when the link is down too.

The buffer size is calculated from the longest outage you are prepared to survive without loss, not from the average one. If the link can vanish for a day, the buffer must hold a day of events of every type on the busiest day.

When the buffer is full, a rule is needed: what to discard. The right answer is old readings, never payments. And that rule must be in the code, not in someone’s head. Separately: a buffer that never drains because the server returns an error on one event and the queue stops on it is the most common way to lose a week of data. The problem event is set aside; the rest moves on.

The clock that lies

A controller without time synchronisation drifts, and in a month it can be minutes away from the server. While the link is up, that goes unnoticed: the server stamps its own time on receipt. When events are sent on after an outage, the receipt time means nothing any more, and only the site’s timestamp remains — the one that drifts.

So every event must carry two timestamps: the site’s time and the server’s time of receipt. The difference between them while the link is up is the measured drift, and it can be applied to events sent later from the buffer. Time is stored in UTC; local time is for display only.

A special case is a site that rebooted without a clock battery and woke up in 1970. Such events are visible immediately if the system looks for them, and lost for ever if it does not. The rule is simple: a timestamp earlier than the system’s installation date is an error, not an event.

Deduplication

Sending on after an outage almost guarantees duplicates: the site sent, the server received, the acknowledgement did not arrive, the site sent again. If the server counts both, the day’s revenue doubles precisely on the days when the link was poor — the worst possible moment for an error, because those days are already hard to reconcile.

Every event must have an identifier created by the site at the moment of the event, not by the server at the moment of receipt. The server, receiving an event with a known identifier, replies “accepted” and does nothing. The site, receiving “accepted”, removes the event from the buffer. That is the whole mechanism, and it has to be there from day one, because adding it later means recomputing history.

The identifier must be unique across sites, not only within one. Bay number plus a counter that resets on reboot is not unique. Site identifier, time and a random part is.

What the dashboard must show

The most important number on a site dashboard is not revenue and not temperature but the age of the latest data. “Bay 3: updated 4 minutes ago” and “Bay 3: updated 6 hours ago” are two different states, and a dashboard that shows the same green value in both cases is lying.

Every reading has an age, and the dashboard shows it explicitly — by colour, by label, however, but visibly. A value older than a threshold is shown as stale, not as current. That applies to totals too: today’s revenue across a network where one site has been offline for six hours is not revenue but a lower bound, and the dashboard should say so.

The second thing is the buffer size on every site. A growing buffer means an outage still in progress. A buffer that has not drained after the link came back means a fault in the sending. Both states must be visible without anyone logging in to the site over SSH.

Alerts that do not cry wolf, and choosing the channel

An alert about a dropped link that arrives on every drop stops being read within a week. The rule that works: alert not on the fact of the outage but on its duration beyond a threshold, and set the threshold from how long the site can operate on its own. A ten-minute drop is nothing. A drop longer than half the buffer is.

The second rule: alert on what requires action, and stay silent about the rest. “Link restored” requires no action; it belongs in the log. “Buffer not draining after restore” requires action. “Bay produced no events for an hour during opening hours” requires action, and it is often more useful than any link alert, because it catches an outage, a broken bay and a broken terminal alike.

On the channel. Mobile is cheap, installed in a day, and drops. Wired is expensive to lay and stable. Our experience: if a wired channel is possible on site, it pays for itself; if not, mobile with a backup from a second operator and telemetry built as described above. The mistake is to treat the mobile channel as permanent and build a system that stands still without it.

Frequently asked