Building Physical Agentic AI
Planting seeds for the enchanted forest
Hello readers! It’s been a long time since my last post. I’ve missed you.
The past two years have been a whirlwind of personal and professional changes. It’s been frankly overwhelming, and it’s given me a great excuse to fall behind on writing.
One of the big events has been the acquisition of Edge Impulse by Qualcomm. I was the first full-time employee, so it’s been an unbelievable six year journey and one I am very grateful for.
After a period of adjustment things are getting back into a predictable rhythm, and I plan to post regularly again.
Anyway, this is an important one. I’ve been feeling a change in the air, like the end of one era and the beginning of another. So let’s head together down the rabbit hole.
Seven years ago I was working at Google on the TensorFlow Lite team. I had fooled around with IoT data processing in my Tiny Farms days, but at Google Brain, Pete Warden and Andy Selle introduced me to the nascent field of TinyML: deep learning models on embedded hardware. At the time, this meant tiny keyword spotting and image classification models running on cheap Cortex-M microcontrollers. It was mind blowing stuff that had never been seen, and a whole new world to explore.
TinyML blossomed into this big, diverse field that we now call edge AI. It has infiltrated into every corner of life, from consumer hardware to heavy industry. There are deep learning models everywhere now, turning raw sensor signals into data that’s useful for business logic.
At the same time, the rise of generative AI models has given us another giant playground of technologies. But these models require a ton of compute, so they’ve existed in a different world to most of edge AI.
One major challenge with edge AI was accessibility. Building an application using AI required not just domain knowledge in the application area (the prerequisite to solving any problem) but a ton of experience in deep learning, signal processing, embedded engineering, and statistics. This near-vertical learning curve meant that our original TinyML guidebook—representing the absolute fundamentals required to build something useful—was a couple of inches thick.
We tried to tackle this problem when creating Edge Impulse: a set of tools that make it much simpler to train and deploy perception models to edge devices. In 2026, any developer can train a model to do something useful in their domain of expertise. This is only true because of how the tooling has advanced.
The progression of technology often follows this pattern. First, fundamental advances lead to new theoretical capabilities. Next, improvements in tooling bring these capabilities within reach of the people who build things. Finally, they are applied to solve the long tail of real world problems.
TinyML was possible because microcontrollers happened to become capable enough to run small deep learning models. Today, the newest generation of edge AI silicon has been designed specifically around transformer-based models, like those used in LLMs. It can run small genAI models in real time.
This creates a foothold for AI agents to move to the edge. That’s the big change that’s coming. But it’s not about chatbots, or virtual assistants. These are the default use case for LLMs, but to me they’re pretty boring.
The development I find fascinating is chain-of-thought reasoning applied to physical problems: the idea that a device can take some input from sensors, have a little think about it, then decide on what to do.
The goal is to give AI agents the ability to sense, reason about, and influence the physical world. We can call this concept “Physical Agentic AI”.
In Reaching the enchanted forest, I describe a world of small, responsive systems that react like living things:
Instead of automating our world through centralized intelligence, we can place a tiny spark of insight into every object we make. We can add contextual understanding to every interaction, so your tools know what you’re trying to build and can help you along the way.
With Physical Agentic AI, I think we are almost there.
I think of Edge Impulse as our stack for edge AI perception. It allows developers to turn raw sensor data—for example, images, audio, or time series of accelerometer readings—into simple predictions. For example, a keyword spotting model might turn raw audio samples into a probability distribution over a set of keywords:
raw audio samples -> { “lights on”: 0.8, “lights off”: 0.1, “no detection”: 0.1 }An embedded developer can then write a simple algorithm to act upon this. For example:
threshold = 0.75
if (result[“lights on”] > threshold) {
// turn the lights on
}This takes care of the perception part, but we left the “understanding” to the developer. Our job was to turn the raw data into a useful data structure containing information that is pertinent to the application’s business logic (in this case, the probability of the user having said one of two keywords). Writing the business logic that makes decisions with that data was the user’s job.
There are tons of ways to make decisions on-device, and until recently they have all required deep domain expertise and worked in very different ways. For some applications a simple if statement is enough. For others you need more complex mechanics: state machines, Kalman filters, dynamic programming, graph algorithms, or whatever combination of things.
Every application is different, which makes it very tricky to have a general approach. Instead you need a software engineer with domain expertise (or with a friend who has it) to design the correct set of algorithms.
The application engineering process is basically an attempt to crystallize domain expertise in code. Instead of a person making decisions, we have a software algorithm that uses heuristics to efficiently approximate the same result. This is highly effective, but it takes a lot of work, and it can be brittle: if you didn’t anticipate a state during development then you may not design your algorithm to handle it well.
But since it’s 2026, we no longer have to choose between a real person and a bunch of hardcoded C++. Today, we have AI agents: magical computer programs that try to act like people do. What if we could slot them in instead?
For whatever reason, the large scale pre-training that we use to create LLMs bakes in a sort of elementary understanding of the world and its rules. It’s not perfect: models struggle with basic math, have a tendency to make things up, and can be just plain dumb. But they are the closest thing we have to a general purpose “decisionmaking engine”, however flawed, that can work in all sorts of applications.
The physical world has an infinite variety of situations and events. Businesses want to build systems that can handle them. Traditional software systems lack world knowledge, and require every situation to be accounted for in advance. But agentic AI systems can make reasonable decisions in novel situations, based on information from diverse, multimodal sources.
So how does this work? Instead of writing super-precise application code, we write a much looser prompt that tells the agent what sorts of decisions to make. And we can format inputs and outputs in a structured way, and expose “tools” to the LLM (which it calls using structured arguments) that enable it to do things (like use a calculator for basic math, or switch on a light).
There are some cool side effects to this approach. By creating a structured and constrained format for I/O and tool calls, we turn the agent into a standardized “black box”. For certain inputs, and under certain conditions, we expect certain outputs. This is something we can write tests for. We can then evaluate different agents, or test them against hard-coded business logic.
We can also build safety into our tool calls: hand coded guard rails, like rate limits, that protect against faulty behavior. We can build external validators to monitor the system for issues. And we can use tools like simulation and adversarial agents to help test, monitor, and inspect.
It turns out the testing piece is vital. This scrutiny is how we elevate a system from a clever tech demo to a real piece of engineering. We must be able to prove an agentic system works just as well as a traditional one. This is how we can get away with using small, simple language models. The models may be dumb, but if they can be proven to do the job then it simply doesn’t matter!
It’s time for some examples. I have a soft spot for agtech, so let’s start there. Imagine a greenhouse management system: it watches plant growth and leaf color via camera, interprets signs of stress or disease, adjusts irrigation valves, opens/closes roof vents, and toggles shade covers — adapting to conditions a timer system never could. Its prompt includes some expertise about expected plant diseases, along with information about how to adjust the physical conditions via tool calls.
A similar system could be used for monitoring beehives: it watches hive entrances via camera to assess colony activity levels and detect swarming behavior, controls ventilation flaps based on hive temperature, closes entrance reducers at night, and sends the beekeeper a weekly health assessment with visual evidence.
Or how about in your own home: understanding household activity, and gently adjusting the climate and lighting as you go about your day?
The cool thing about these examples is they use basically the same hardware: a camera and some environmental sensors. The drivers for the hardware are the same. The only thing that differs is the prompt. And to add functionality, you can just change it! For example, a gardener could enable the greenhouse to take care of a new type of plant by describing its healthy state and possible diseases. A nice suite of tests will protect them from breaking the existing behavior. Super fun stuff: it’s like low-code/no-code for the real world.
It may feel like there’s a hidden cost here: we’re replacing hand-optimized code with crazy expensive LLM inference. Isn’t that overkill? Well, it depends. There’s always a tradeoff between cost (chips and energy) and benefit (dramatically improved flexibility and reduced development time). For some applications, the tradeoff will be worth it. For others, not so much.
Right now, the sweet spot is applications where energy use is less important, and where a BoM cost in the $100s is OK. Chips are rapidly improving, so the design space will expand—but it’s already good enough to support applications like smart vehicles and industrial equipment.
So there’s a whole new world opening up! It’s even bigger, weirder, and even more exciting than what happened with TinyML. I can’t wait to see developers using these technologies to build crazy stuff that has never been envisioned—from handling mundane tasks in industry to creating completely new experiences in consumer tech.
This all sounds great—but there’s a huge barrier. As with TinyML, the first generation of tooling is incredibly hard to use. It’s difficult to even run an LLM on an IoT board, let alone integrate it with sensors, and test and deploy it. And there’s a whole universe of best practices, testing protocols, and development processes that need to be figured out. Generative AI is notoriously fuzzy, and without the right guardrails it is useless: just a new set of ways to build systems that don’t work.
But for people who build tools—like me!—this is the exciting part. It’s an extraordinarily interesting time. People are going to build physical systems that feel genuinely intelligent. To make sure everyone can take part in this revolution, we need to make it as easy and safe as possible.
I want to make it beautifully simple for any developer to summon the magic I describe in the enchanted forest—“a tiny spark of insight in every object we make”—while ensuring they’re doing real engineering, not building flashy demos that fall apart in actual use.
So here we are. Work continues on Edge Impulse, which provides the perceptual tools that these systems will rely on. But there’s a new frontier opening up, and I’m thrilled to be exploring it together.
Subscribe to this newsletter to stay in touch, email if you want to collaborate, or leave a comment below to share your thoughts.
Warmly,
Dan