Bringing it back inside is an engineering problem, not a hardware one

Ask a team why the AI features still run in someone else's data centre and they usually point at the graphics card. Then they try it. Dozens of call sites are scattered through the client, the software kit, the parsing of a streaming reply, the retry logic - all of it written around a hosted interface, and swapping vendors means writing it again. LocalAI turns that around: do not touch the code, move the address. It offers one interface that accepts the same requests as the large hosted providers, so an existing client keeps sending, keeps parsing, and the model runs on your own machine. The data never leaves the building, and no application was rewritten to make that true.

LocalAI is an open-source runtime you host yourself: one binary, one compatible interface, and whatever engine sits behind it is your decision.

A small core, with the engines arriving when called

What irritates people about local inference tools is the bulk: several gigabytes of dependencies installed to run a modest model, most of which will never be touched. LocalAI makes each inference engine a separate backend, shipped as a standard container image and pulled only when a model actually needs it. The backends run isolated from one another, can each be installed, upgraded or removed on their own, and a faulty one does not drag the rest down - it can even live on a different machine.

That arrangement brings a direct benefit: changing engines does not mean changing interfaces. Under the same endpoint one model may run on one engine and another on a quite different one, and the client notices nothing - the request format is unchanged, only a line of configuration underneath. More than sixty backends are available today, covering text generation, speech recognition and synthesis, understanding of images, image and video generation, embeddings and reranking. On the hardware side there are acceleration paths for the three big chip vendors, for the in-house silicon of the portable line, for the portable graphics standard, and for small embedded boards.

Without a graphics card, you can start today

The project states one principle plainly: every feature ships a processor path first, and that path is exercised in continuous integration on hardware ordinary people already own. It is not a degraded fallback that merely avoids crashing. A card makes the thing faster; it does not make it usable. For a team that wants to prove the workflow before signing for hardware, that order matters - get it running on the machines you have, argue about capacity afterwards.

If the model itself is too large to fit, there is quantisation. The project's own scheme picks a precision per tensor and per layer, and on a mixture-of-experts model of about 35 billion parameters it took the footprint from roughly 65 GB down to 12 GB while generation went from 30 to 74 tokens a second; perplexity moved from six point five to seven point one. What comes out is a standard file of the common quantised format, which the original engine reads without a patch.

When there is more than one machine, the scheduling is somebody else's job

A single machine has a ceiling. The answer here is simply to add another: request routing, placing a model where the video memory allows, affinity for cached prefixes and failover are all taken over by the runtime, work that would otherwise be a scheduler of your own. It supports a production shape that scales out over worker nodes, and a peer-to-peer federation as well - a few idle machines in an office joined together make tasks feasible that used to demand a dedicated card. Load rises, machines are added, and the cluster works out how to use them.

It also listens, and it looks

Local models are often dismissed as something that only types. This layer carries a good deal more: live transcription with speaker labels and timestamps, fast enough to keep up with a meeting in progress; recognition of several hundred classes of sound event, so a door, breaking glass or a smoke alarm is noticed; voiceprint and face recognition with liveness detection, which tells a person from somebody holding up a photograph; a question about the red cup on the left, answered with coordinates rather than a sentence; depth estimated from a single ordinary photograph, and a three-dimensional scene rebuilt from a handful of them, with no camera calibration and no graphics card. Names, addresses and card numbers are found and blanked out on the machine, before anything would leave it.

One security change to read before upgrading

The current release switched authentication to deny by default. The old middleware decided which endpoints needed protection by matching path prefixes, so anything not on the list stayed public; some routes carried aliases without that prefix, the global check did not apply to them, and security researchers reported the bypass. The middleware now looks at the request method as well and refuses by default, letting through only the methods and paths explicitly registered as public. Anyone running database authentication or an older key has two changes to prepare for: the endpoint that reports version information now wants credentials, and the addresses returned for generated audio, images and video also need the key before the file can be fetched.

If all you want is the model back on your own machine, ideally without rewriting anything, LocalAI offers the shortest road: change one address, and let the engines it assembles on demand do the rest.