How Beni Actually Follows You: The Tech Behind the Lock

Anyone can build a camera that points at a person. The hard part is not losing them when they turn away, when someone else walks between you, when the sun drops behind a building. That gap — between "tracks a subject" and "keeps tracking the right subject" — is where most follow-me cameras fall apart. It's also where Beni's engineering team spent most of their year.

The problem isn't seeing. It's remembering.

Modern object detection is a solved problem. A phone from three years ago can identify a person in a frame in milliseconds. What's not solved is telling which person is the one you were following two seconds ago when they turned around, walked behind a tree, and re-emerged half-occluded.

Naive trackers use bounding-box similarity — if a person-shape appears close to where the last one was, assume it's the same person. This fails the moment someone crosses your path. Beni's system layers three signals on top of that.

Signal 1: Gait, not face

Faces are unreliable outdoors. They're small in the frame, lit from every direction, half-occluded by hats and sunglasses. So Beni doesn't rely on them for the primary lock — it looks at how you walk.

Every person has a distinct gait signature: cadence, stride width, weight distribution, the tiny bob of shoulders. Beni builds this signature during the first 20 seconds of following. From then on, even if you turn your back, wear a coat, or step behind a bush and out the other side, the gait pattern re-identifies you.

Signal 2: Silhouette color histogram

Once Beni has your gait, it also samples what you're wearing — not as a photo, but as a 32-bin color histogram of the region around your body. It's the visual equivalent of "green top, black pants, red shoes" but compressed into 96 numbers that survive lighting changes far better than raw pixels.

If two people with similar gait cross paths (siblings, teammates in uniforms), the color histogram breaks the tie. If both signals agree, lock confidence goes up. If they disagree, Beni slows down and re-scans instead of confidently following the wrong person.

Signal 3: Kalman-filtered spatial prediction

The third signal is where you should be. Beni maintains a running estimate of your velocity and acceleration and predicts where you'll be 300ms from now. When a lock candidate appears near that prediction, its score is boosted. When someone appears somewhere physically impossible (you can't teleport 4 meters in 0.3 seconds), that candidate is rejected outright.

"The trick isn't in any one signal. It's that when one signal fails, the other two are usually right — and Beni is doing the vote thirty times a second." — Dr. Sofia Kellett, Perception Lead

What happens when it does lose you

No system is perfect. When lock confidence drops below threshold — usually because you disappeared for more than 4 seconds behind full occlusion — Beni does something most trackers don't: it stops guessing and asks.

• The status ring turns amber and Beni holds position instead of chasing a wrong guess.
• It scans a 180° arc for anyone matching your gait+silhouette signature.
• If a match appears with ≥85% confidence, it re-acquires silently.
• If not, it waits. Your app buzzes gently — one tap on your live preview re-locks in under a second.

Compare this to trackers that will confidently follow the wrong stranger for 30 seconds before giving up. Slower to re-lock, but you never end up with 45 seconds of footage of a person you don't know.

Watch: subject-lock stress test

Six volunteers, one park, one Beni. We had them cross paths, swap jackets mid-shot, walk behind trees, and even hand each other a phone to try to confuse it. Full uncut test below.

https://www.youtube.com/watch?v=VIDEO_ID

The stuff we're still working on

We're not going to pretend this is solved. Two known failure modes we're actively improving:

• Extreme low light — Below about 8 lux, gait signal degrades because we can't reliably segment the body silhouette. Firmware 2.1 (shipping next quarter) adds a low-light mode that increases exposure and lowers frame rate to compensate.
• Cyclists moving toward the camera — Faster-than-walking approach speeds throw off the Kalman filter's calibration. Currently, if you're biking straight at Beni, it treats you like a fast walker and can misjudge your distance. Fix in progress.

Curious about the rest of the stack? We're publishing a longer read on the mapping and obstacle-avoidance system next month. Get on the engineering list if you want it in your inbox.