What is Edge AI? From a $1 chip to a rack in the basement
Edge AI means running a model where the data is made. No round trip to a datacentre, no waiting for the answer to come back.
That is the whole definition. Everything else is size, and the range is wider than people expect. A $1 chip in a washing machine and a $20,000 server in a hospital basement are both edge AI. Five orders of magnitude of power separate them, and very little of the engineering carries across.
The box is fixed: its watts, its memory, its price. Nothing bigger is coming. You size the model to the box, never the box to the model.
The spectrum
Four tiers, and the interesting column is the last one: what each tier charges you for the privilege of fitting.
| Tier | What fits | What it costs you to fit |
|---|---|---|
| Microcontroller | a few hundred KB, 8-bit weights | quantize hard, compile into the firmware, live without an OS |
| Mobile SoC | tens of MB | the vendor’s runtime, and a power budget shared with the whole phone |
| Edge accelerator | hundreds of MB, a small LLM | a precision choice, and a compile for every chip you ship on |
| Edge server | several models at once | scheduling, cooling, and someone to own the machine |
No operating system
A microcontroller has a few hundred kilobytes of memory and no GPU. Often no operating system either. The model gets quantized to 8-bit and compiled into the firmware. In return, the device runs a year on one battery.
At this tier the cloud is not an option you weighed and rejected. Three separate things rule it out, and any one of them would be enough on its own.
- Latency. A hearing aid pulling one voice out of a restaurant has a few milliseconds. Any more and the sound stops matching the lips in front of you.
- Power. Transmitting costs far more energy than thinking, so a soil probe’s real job is deciding whether the reading is worth sending at all.
- Connectivity. A vibration sensor on a pump in a basement has no signal, and the bearing fails regardless.
So the engineering is not about choosing a model. Getting a useful one into a few hundred kilobytes is the whole job.
In everyone’s pocket
The mobile SoC is the tier you already carry. Phones and modern cameras have a neural accelerator, a few watts to spend, and gigabytes of shared memory. You used it today without noticing: face unlock, live captions, keyboard prediction, most of what happens between the shutter and the photo.
What separates it from the tiers either side is that none of the budget is yours alone. You are a guest on a device already doing a dozen other things, and every watt you spend comes out of somebody’s afternoon. A model that benchmarks well and warms the case in normal use has failed, whatever its accuracy.
It also explains why this work stays put rather than moving to a server that could run it faster. A face template and a keyboard history are precisely the things nobody wants making a round trip.
The workhorse
The edge accelerator, ten to thirty watts, is where industrial edge AI lives: NVIDIA Jetson, Rockchip RK3588, NXP i.MX 8, TI AM68. Enough memory for a real vision model, sometimes two. Little enough heat for a fanless enclosure. A warehouse robot, an inspection cell, a barrier camera, or a $249 box reading a chest X-ray and writing the report to go with it, fully offline.
This is the tier where the work stops being about the model at all. Fitting one means choosing a precision and compiling it for the chip, and both cost engineering time the cloud never charges you for. That work is per model and per silicon family, so shipping the same product on three boards means doing it three times.
The model is rarely the hard part. The budget around it is, and the per-chip work sets the schedule.
Still the edge
The edge server sits at the top: a machine that looks like a server, in a factory cupboard rather than a datacentre. A Jetson AGX rack, or an on-prem GPU box. Hundreds of watts. Several models at once.
This tier is the one that shows what the word actually means. A hospital running every study taken that day, a plant putting forty camera streams into one box, a law firm with a language model on premise because the documents cannot travel: none of that is small, and all of it is edge. Move the same machine into a datacentre and not one thing about the hardware changes. Only the category does.
Edge is a statement about where the data is, not about how much power you are allowed to spend near it.
What does not change
From the dollar chip to the machine in the cupboard, the hardware changes by five orders of magnitude and three things do not.
- The budget is fixed. In the cloud a slow model costs money. At the edge it costs you the product, because there is no bigger instance to move to.
- The data stays put. That is usually why the project exists, whether the driver is latency, connectivity, cost per inference, or a rule about where personal data may go. The EU is turning several of those into obligations.
- The device is not yours. It sits in someone else’s building, on someone else’s network, and the model on it can be extracted if you ship it unprotected.
The boundary
Edge is not the answer to everything. Pretending otherwise is how projects fail late.
Training belongs in the cloud. So does a model too large to quantize into your power budget, and no compiler work will change that. So does anything rare and bursty, where a box idle all week is worse value than an API call. So does anything needing a live view across every site at once.
The honest version is a line, not a slogan. Run at the edge what has to sit near the data, and send the rest away. Most real systems straddle that line. The arithmetic of owning the box against renting the cloud decides where you draw it.
That is the work we do: taking a model down to the tier you actually ship on, and keeping it fast, accurate and sealed once it gets there. Building on one of these tiers? We are looking for design partners. Talk to us.