p99log

Aug 25, 2026 / Systems, Integrations, Reliability

Capability Discovery Is a Distributed Systems Problem

What should your software do when it can't fully trust what a device says it can do?

Here’s a question that looks simple but isn’t: what should your software do when it can’t fully trust what a device says it can do?

Most teams treat “supporting device X” as a matter of detecting the vendor and model, then branching code based on that. This works fine - until you meet a device that says it can do something it can’t, or that can do something it never mentioned. At that point, knowing the vendor name doesn’t help. The real question was never “what is this device.” It was “what can I actually get this device to do, right now, reliably.”

This is p99 behavior in a different form. Most devices will report capabilities that are close enough to reality. The systems problem starts with the ones that do not.

A real example

Take IP security cameras. There’s an open standard called ONVIF, built specifically so different camera brands can be controlled the same way. A camera is supposed to advertise a “profile” listing what it supports - video streaming, pan-tilt-zoom control, and so on. In theory, you check the profile and you know what you’re working with.

In practice, this isn’t fully reliable. Even ONVIF’s own documentation acknowledges that conformance to the standard doesn’t guarantee full interoperability: optional features can be implemented differently by different manufacturers, even when both devices technically pass certification. So a camera might claim to support streaming through ONVIF, but the specific channel you actually need doesn’t come through cleanly. Meanwhile, that same camera might expose a full, working list of channels through its own manufacturer-specific interface - one that never showed up in the standard one.

Neither side is lying, exactly. The standard is just partially implemented in practice, in ways that vary from device to device. The moment you hit this, “automatic setup” stops being a simple lookup. It becomes something closer to a negotiation: try one source, check if it actually works, and have a plan for what to do if it doesn’t.

Check what it does, not what it claims to be

The fix isn’t a longer list of special cases for more vendors. It’s a different starting question. Instead of asking “what brand and model is this,” ask “what does this thing actually do when I test it right now.”

In practice, that means:

Checking the label is a snapshot. Devices get firmware updates that quietly change what they support, and manufacturers don’t always document it. Testing live is the only approach that keeps working as the ground shifts under it.

flowchart TD
    A[Device reports capability] --> B[Try standard interface]
    B --> C{Live check passes?}
    C -- Yes --> D[Use standard capability]
    C -- No --> E[Try vendor interface]
    E --> F{Live check passes?}
    F -- Yes --> G[Use fallback and record it]
    F -- No --> H[Return explicit unsupported result]
Capability discovery flow: validate the standard interface, then attempt a vendor fallback or return an explicit unsupported result.

Make your fallback plan a real, visible decision

Once you accept that any single source of information might be wrong, you need a backup plan. But that backup plan should be something you decided on purpose - not just whatever order the code happened to be written in.

Three things worth deciding clearly, ahead of time:

What do you try first, and why? In the camera example: try the open standard first, since it’s the cheapest to support across many brands. Fall back to the manufacturer-specific option only if the standard’s answer looks incomplete or fails the live check. This order is a judgment call about which source is more likely to actually be true - not just which one is more “official.”

When do you give up? This is the part teams often skip, and then discover the hard way - usually as a hang or an endless retry loop. If you’ve tried every option and none of them gave you a working result, that should be a clear, defined outcome - “nothing worked, here’s why” - not a silent, endless wait.

Can someone see it happening? When your software quietly falls back to a second or third option, that should show up somewhere - a log, a dashboard, something. If your camera setup is secretly relying on the manufacturer-specific method for 4 out of 10 devices because the standard method keeps failing for them, someone should be able to see that number without reading the code. A fallback that works silently today is a fallback nobody remembers exists the day it also breaks.

A useful comparison: this is the same kind of thinking that goes into designing a circuit breaker for a web service - you wouldn’t ship one without clearly defined states (working, failing, recovering) that you can actually see. Figuring out what a device can do deserves the same care. It’s the same kind of problem, just applied to hardware instead of a web service.

Don’t assume “standard” means “the same everywhere”

The biggest mistake here isn’t forgetting a backup plan. It’s an assumption made earlier: that because a standard protocol exists and a device claims to support it, every device that “supports” it behaves the same way. That’s rarely true. This isn’t unique to security cameras - it’s true of almost any standard that spans multiple companies making their own hardware or software.

The practical takeaway: “we support the standard” can’t be your whole plan. It should be the first thing you try, checked live, with a real backup plan behind it and a clear answer for when nothing works. The standard cuts down how many special cases you have to deal with. It doesn’t remove them - and treating it like it does is exactly how a “supported” device ends up being debugged in production instead of caught early.

The systems that hold up aren’t the ones that trusted the standard correctly. They’re the ones that never fully trusted it in the first place.