Every device built for a mine gets tested somewhere other than a mine first. Underground time is scarce and site access is expensive, so the bench and the office network carry most of the early work. The risk lies in the assumptions that test environment makes for you without anyone noticing, which the mine may or may not share. You usually find out on site.
The clearest example I have lived through cost us two weeks and a second site visit. It came down to a filter in our firmware that did exactly what we had written it to do.
The Tag That Kept Picking Up Jobs
The device started life as the LTE-enabled anchor described in the Level 4 article: a fixed unit meant to be installed along the drifts, chirping and listening so the system knows where each machine is and what is close to it. When the client proposed mounting them on vehicles, we modified the firmware so the same unit could serve both roles, fixed anchor or vehicle tag for tracking and proximity detection.
Most of that work happens machine to machine, with no network involved. The LTE modem gave the unit a path to the wider mine network wherever LTE coverage extended, which underground is often further than the fixed network goes. It could connect over Wi-Fi as well.
That link quickly carried more than positions. The unit sent remote I/O from the vehicle over LTE, which the ventilation system used to fine-tune Ventilation on Demand, and it relayed the emergency signal to the cap lamps of anyone working near the vehicle. A tracking tag had become part of the ventilation control loop and part of the evacuation chain.
This client had a habit of finding new uses for our hardware. The modem’s configuration was open, and they saw that every equipped vehicle now carried a cellular link, so they started sending the machine’s third-party telemetry through it, data unrelated to tracking or proximity detection.
For them it made sense: the machine already had a modem, so why install a second one. For us, it meant the device’s network configuration would now be shaped by systems we had never looked at. We did not register that at the time.
Testing Without the Real Network
A private LTE network is expensive, and we did not have one to test on. The modem spoke Wi-Fi, so we ran our tests on the office network: one network, one address for the device, straightforward routing.
Every test passed. Those tests answered the question we asked them. The mine was about to ask a different one.
Silent Anchors
On site, the SIM card went in and the client’s configuration was loaded. The modem attached to the private LTE network without complaint, the tag was powered and running, and the anchors a few metres away received nothing from it.
Everything we had access to checked out. The client’s private LTE configuration, its addressing and its routing were outside our view, so we had a symptom with no way to trace it, and we left site without a diagnosis.
One Device, Two Networks
The answer came after the visit, once the client gave us access to their private network from our side.
Most mines segregate their networks, usually along the lines of the Purdue model. Field devices and control systems sit on an OT network, business systems sit on the IT network, and an industrial DMZ sits between them so data can cross without the two being directly connected. The client’s telemetry flowed to a server on the IT side and from there to the cloud. Our application lived on the OT side and reached its own server through the DMZ. To serve both, the modem carried one address on each side.
Our firmware had been written with a single address in mind. A tag like this receives a lot of traffic meant for other devices, so to keep processing to what mattered, we had added an optimization: ignore any packet not addressed to the tag’s own IP. On the office Wi-Fi, where the device had exactly one address, the filter worked as intended. On site, traffic meant for the tag arrived on the second address, and the filter discarded it, exactly as written.
The code did its job. The assumption behind it came from the office network. Two weeks of testing and development went into understanding the behaviour, reworking the filter and validating it, followed by a second visit, which went smoothly.
Why It Stayed Cheap
The cost stayed low here, mostly thanks to circumstances. The client was a close R&D partner, iterating together was normal in that relationship, and once access was configured we could work against their network from our own office.
Most deployments come with fewer advantages. At a remote site, or with a client who is less of a partner, the same bug costs travel, a missed commissioning window and some of the client’s confidence in the product before it has delivered anything.
What a Scoping Phase Would Have Caught
At the time, we tended to jump into development and were less rigorous about the research that should come first. A proper scoping phase would most likely have caught this before a line of firmware was written.
That phase can be light. It needs a few questions answered by the people who own the network. What network will this device actually run on? How does this mine separate OT and IT, and where will our traffic cross between them? Who else is going to use this device once it is installed? And can we test on something that behaves like that network, even partially?
Any optimization that depends on an assumption about the environment deserves to be written down as an assumption, so someone can check it against the real site before the trip.
What This Means If You Own the Network
The same story carries a lesson for the mine’s side of the table.
A vendor’s device has almost always been tested on someone else’s network. Asking what it was tested on, and sharing how your OT and IT networks are separated before the first visit, takes an afternoon. Debugging without that information costs both sides far more.
Reusing a vendor’s device for another purpose is often a good idea, and in this case it was. It works best as a conversation with the vendor. A shared modem is convenient, and it also makes two systems depend on one configuration that neither team fully owns.
What Scoping Does Not Solve
Scoping catches the assumptions you know to ask about. The ones you make without realising slip through: nobody on our team saw that filter as an assumption about the network. It looked like ordinary housekeeping.
On-site testing stays necessary too. Some behaviours only appear under the mine’s real coverage, real traffic and real handoffs between access points, and the office can only approximate them.
And the network keeps changing after commissioning. A new firewall rule, a reorganized address range or a change on the IT side can break a device that has worked for a year. Someone has to own that interface after the project team leaves, otherwise the next silent anchor gets discovered by an operator on shift.
The fix was small. The question that would have prevented it fit in one email to the client’s network team.
Planning a device deployment on a mine’s OT network, or chasing a problem that only shows up on site? Get in touch. No commitment, no sales pitch.
Leave a Reply