I gave my DJI Mini 3 a brain
I have a DJI Mini 3. It is a lovely little camera that flies, but out of the box it only does what the sticks and the DJI Fly app tell it to. I kept wondering how far I could push it with code and a decent vision model. This is the story of turning it into something that watches, thinks, and eventually flies itself.
Fair warning: parts of this were fiddly. I left the messy bits in because that is the fun part.
https://youtube.com/shorts/92wYNbN6g-I?feature=share
The two open doors
The Mini 3 is not a developer drone on paper. But it has two doors left open, and that turned out to be enough.
Door one is video out. The DJI Fly app can push an RTMP stream to any server you point it at. So I can get the live feed onto my laptop.
Door two is control in. DJI's Mobile SDK v5 supports the Mini 3 through something called Virtual Stick. No fancy onboard waypoints, but you can send it velocity and yaw commands from an Android app. That is enough to make it move on its own.
One brain reading the video, one path to send commands back. That is a full loop.
The idea
I did not want a pile of one off scripts. I wanted one clean pipeline where every stage is swappable:
video ~> perception ~> world model ~> brain ~> controller ~> drone
Read a frame, figure out what is in it, track it over time, decide what to do, act. The nice thing about drawing it this way is that the same code runs three ways without changing the middle. It can read a recorded clip on my desk, read a live stream while I fly manually, or actually command the aircraft. Only the ends change.
Step one, give it eyes
The first version does not touch the drone at all. It just watches.
The feed comes in over RTMP, YOLO runs on each frame, and a tiny tracker keeps a stable id on each object so the thing knows that the person in this frame is the same person as last frame. On top of that sits a small world model that works out useful facts. Is the subject centered. Is it drifting toward the edge. Did something new just walk in.
Then an advisory brain turns those facts into plain guidance. Stuff like "person entered the frame" or "subject drifting left, pan right." It can even say it out loud, though I turned the voice off pretty quickly because it got chatty.
First time I ran it on the real feed it locked onto a cup and a TV on my desk and started coaching my framing. Silly, but it worked, and that was the moment it stopped being a diagram.
Step two, give it hands
This is where it got real. Watching is safe. Flying is not.
The control path has to live in an Android app because that is where the Mobile SDK runs. So I wrote a thin bridge app. Its whole job is to connect to the drone, expose a tiny local API, and be a dumb, safe executor for whatever the laptop asks. The laptop keeps all the thinking. The phone just does what it is told and refuses anything unsafe.
The API is small on purpose:
GET /telemetry what is the drone doing right now
POST /arm enable virtual stick
POST /command go this direction for one tick
POST /takeoff lift off
POST /land come down
POST /rth abort, come home
Virtual Stick has a quirk that shaped the whole design: a command only applies for a fraction of a second, then the drone hovers again. So the bridge re-sends the current command about ten times a second. If the laptop goes quiet, it stops moving within a beat and comes home. That deadman behaviour is not a feature I added later, it is baked into the loop.
Making it follow me
Once the loop closed, follow mode almost wrote itself. Pick a target, in my case a person, then push three things at once. Yaw to keep them centered left to right. A little up or down to keep them centered vertically. Forward or back based on how big they look in frame, which keeps a roughly constant distance.
I tested it against a simulator first, a fake drone that just integrates the commands so I could watch the numbers without anything spinning. Then props off. Then a low hover in an open space with my hand on the real controller the whole time. Every command still passes through a safety layer that clamps speed, holds an altitude ceiling, keeps a geofence, and returns home on low battery.
The dumb problems that ate the most time
The blog would be lying if I skipped these.
I let the tool pick a package name for the Android app and it used my work domain. Renaming that across the whole project was the first yak I shaved.
The RTMP address on the phone had a typo. I was streaming to live/mini and reading from live/mini3. Ten minutes of confusion for one missing character.
The SDK's native library helper got renamed between versions, so the app compiled and then crashed on launch until I swapped one import.
And the classic: on the terrace there is no wifi. I had a small panic about whether any of this would work off the grid. Turns out you just make your own network with a phone hotspot. No internet needed, the two devices only have to see each other. The one gotcha is that the laptop IP changes on a new network, so I made that editable right on the phone instead of baked into the build. Lesson learned the annoying way.
Where it is now
It watches, it coaches, and it can follow a person on its own with a pile of safety rails. It also quietly saves highlight clips while it runs, and there is a command to stitch those clips into one video afterward, which is a nice way to get an edit for free.
https://youtube.com/shorts/Vh5u3aFcTFc?si=tnccCdG8n2cgk5pl
What is next
Natural language goals are the big one. I want to say "orbit that tree" or "follow the red car" and have a model turn that into flight. The plumbing for it is already there, it just needs the planner wired in. Orbit mode is close. On device inference would cut the latency. And I would like a proper web dashboard instead of an OpenCV window.
If you want the technical version, the README has the architecture and how to run it. If you build something with it, tell me.
