# local_ai
Local coding AI for [aider](https://aider.chat): an Ollama container with a model
chosen for *this* machine. Aider runs on the host and talks to the container.
```bash
./setup.sh # analyse machine, pick model, generate Dockerfile + RUN_AIDER.md
./build-ai-image.sh # build image (downloads the model)
./run-ai-image.sh # start container (port 11434, or the next free one)
# then follow RUN_AIDER.md to start aider
```
- `models.tsv` – model catalogue (edit to add models)
- `Dockerfile.template` – container template; `setup.sh` fills in the model
- `Dockerfile`, `.ai-config`, `RUN_AIDER.md` – generated per machine (git-ignored)
- `./setup.sh --model --port --gpu|--no-gpu --yes` for non-interactive use
## What decides which model fits
- **Memory is the limit.** A model must fit in VRAM (GPU) or RAM (CPU). `setup.sh` keeps
about 4 GB of RAM free for the OS, aider and your editor, and about 10% of VRAM.
- **Fits in VRAM = fast.** The whole model on the GPU is by far the quickest option
(`fast (GPU)` in the menu).
- **Bigger than VRAM = hybrid.** Ollama puts as many layers as fit on the GPU and runs the
rest on the CPU (`hybrid ~N% GPU`, a rough estimate). Faster than CPU-only, but the CPU
part sets the pace. Check the real split with `docker exec local-ai ollama ps`.
- **CPU-only** works, at a few tokens/second for a 7B model. Mixture-of-Experts (MoE) models
such as `qwen3-coder:30b` only activate a few billion parameters per token, so they are
much faster on CPU than dense models of the same size, but still need the RAM for all of it.
- **Context window** (4k to 32k tokens) is chosen from the memory budget and costs RAM/VRAM
too. Aider needs room for the repo map plus your files, so 8k is a practical minimum.
- **Status in the menu:** `fits`, `tight` (fits with almost no headroom: close other apps),
`slow` (dense model without a GPU), `too big` (will swap or crash).
- **Port:** the container uses 11434 like a normal Ollama. If that is taken (for example by a
host Ollama), `setup.sh` picks the next free port and writes it into `RUN_AIDER.md`.
### Using an NVIDIA GPU
Docker needs both the **NVIDIA driver** and **nvidia-container-toolkit**. `setup.sh` detects
a GPU that is not usable yet and prints distro-specific install commands (also in
`RUN_AIDER.md`), but only when a driver would actually help. Be careful with driver installs:
- Take a snapshot first (Timeshift) and know how to reach the GRUB menu.
- Prefer the distro's recommended driver (`ubuntu-drivers devices`), not an arbitrary version.
- With Secure Boot on you must enrol the MOK key at the first reboot (`mokutil --sb-state`).
- If the machine stops booting: boot an older kernel or add `nomodeset` in GRUB, then
remove the packages (`dpkg -l | awk '/^ii/ && /nvidia/ && !/linux-firmware-nvidia/ {print $2}' | xargs sudo dpkg --purge --force-depends`,
then `sudo update-initramfs -u`). No network is needed for that.
- After a successful install, verify with `docker run --rm --gpus all ubuntu nvidia-smi` and
re-run `./setup.sh`.
## Example configurations
Speeds are rough orders of magnitude, not benchmarks. Your hardware will differ.
### 1. CPU-only laptop, 16 GB RAM, no usable GPU
`qwen2.5-coder:7b`, context 8k (this is what `setup.sh` picks on such a machine).
| Pros | Cons |
| --- | --- |
| Works anywhere with Docker, no driver risk | A few tokens/second; long answers take a while |
| Good quality for its size on everyday edits | First response is slow while the model loads into RAM |
| Leaves ~9 GB RAM for the rest of the system | Keep prompts small and add only the files aider needs |
### 2. Small GPU (about 4 GB VRAM, e.g. GTX 1650) with 16 GB RAM
Either `qwen2.5-coder:3b` (fully on the GPU) or `qwen2.5-coder:7b` (hybrid, ~55% on the GPU).
| Pros | Cons |
| --- | --- |
| 3B: fully in VRAM, fast and responsive | 3B: noticeably weaker; use `--edit-format whole` in aider |
| 7B hybrid: better answers, still sped up by the GPU | 7B hybrid: speedup is modest, the CPU part dominates |
| Both fit comfortably in 16 GB RAM | Needs the NVIDIA driver + container toolkit (driver install can break boot) |
A reasonable approach: start with the 7B hybrid for quality and switch to the 3B for quick,
simple edits. Re-run `./setup.sh` and `./build-ai-image.sh` to switch.
### 3. Workstation with a 24 GB GPU and 32+ GB RAM
`qwen2.5-coder:32b` (dense, best quality) or `qwen3-coder:30b` (MoE, faster).
| Pros | Cons |
| --- | --- |
| Close to cloud-assistant quality for code | Large image (~20 GB) and long first build |
| Fully in VRAM: fast, with room for a 16k+ context | The GPU is busy while the model is loaded |
| MoE variant stays usable even if it spills into RAM | Needs the most expensive hardware; dense 32B is slow if it does not fit |