local_ai

Local coding AI for aider: an Ollama container with a model chosen for this machine. Aider runs on the host and talks to the container.

./setup.sh            # analyse machine, pick model, generate Dockerfile + RUN_AIDER.md
./build-ai-image.sh   # build image (downloads the model)
./run-ai-image.sh     # start container (port 11434, or the next free one)
# then follow RUN_AIDER.md to start aider

models.tsv – model catalogue (edit to add models)
Dockerfile.template – container template; setup.sh fills in the model
Dockerfile, .ai-config, RUN_AIDER.md – generated per machine (git-ignored)
./setup.sh --model <tag> --port <n> --gpu|--no-gpu --yes for non-interactive use

What decides which model fits

Memory is the limit. A model must fit in VRAM (GPU) or RAM (CPU). setup.sh keeps about 4 GB of RAM free for the OS, aider and your editor, and about 10% of VRAM.
Fits in VRAM = fast. The whole model on the GPU is by far the quickest option (fast (GPU) in the menu).
Bigger than VRAM = hybrid. Ollama puts as many layers as fit on the GPU and runs the rest on the CPU (hybrid ~N% GPU, a rough estimate). Faster than CPU-only, but the CPU part sets the pace. Check the real split with docker exec local-ai ollama ps.
CPU-only works, at a few tokens/second for a 7B model. Mixture-of-Experts (MoE) models such as qwen3-coder:30b only activate a few billion parameters per token, so they are much faster on CPU than dense models of the same size, but still need the RAM for all of it.
Context window (4k to 32k tokens) is chosen from the memory budget and costs RAM/VRAM too. Aider needs room for the repo map plus your files, so 8k is a practical minimum.
Status in the menu: fits, tight (fits with almost no headroom: close other apps), slow (dense model without a GPU), too big (will swap or crash).
Port: the container uses 11434 like a normal Ollama. If that is taken (for example by a host Ollama), setup.sh picks the next free port and writes it into RUN_AIDER.md.

Using an NVIDIA GPU

Docker needs both the NVIDIA driver and nvidia-container-toolkit. setup.sh detects a GPU that is not usable yet and prints distro-specific install commands (also in RUN_AIDER.md), but only when a driver would actually help. Be careful with driver installs:

Take a snapshot first (Timeshift) and know how to reach the GRUB menu.
Prefer the distro's recommended driver (ubuntu-drivers devices), not an arbitrary version.
With Secure Boot on you must enrol the MOK key at the first reboot (mokutil --sb-state).
If the machine stops booting: boot an older kernel or add nomodeset in GRUB, then remove the packages (dpkg -l | awk '/^ii/ && /nvidia/ && !/linux-firmware-nvidia/ {print $2}' | xargs sudo dpkg --purge --force-depends, then sudo update-initramfs -u). No network is needed for that.
After a successful install, verify with docker run --rm --gpus all ubuntu nvidia-smi and re-run ./setup.sh.

Example configurations

Speeds are rough orders of magnitude, not benchmarks. Your hardware will differ.

1. CPU-only laptop, 16 GB RAM, no usable GPU

qwen2.5-coder:7b, context 8k (this is what setup.sh picks on such a machine).

Pros	Cons
Works anywhere with Docker, no driver risk	A few tokens/second; long answers take a while
Good quality for its size on everyday edits	First response is slow while the model loads into RAM
Leaves ~9 GB RAM for the rest of the system	Keep prompts small and add only the files aider needs

2. Small GPU (about 4 GB VRAM, e.g. GTX 1650) with 16 GB RAM

Either qwen2.5-coder:3b (fully on the GPU) or qwen2.5-coder:7b (hybrid, ~55% on the GPU).

Pros	Cons
3B: fully in VRAM, fast and responsive	3B: noticeably weaker; use `--edit-format whole` in aider
7B hybrid: better answers, still sped up by the GPU	7B hybrid: speedup is modest, the CPU part dominates
Both fit comfortably in 16 GB RAM	Needs the NVIDIA driver + container toolkit (driver install can break boot)

A reasonable approach: start with the 7B hybrid for quality and switch to the 3B for quick, simple edits. Re-run ./setup.sh and ./build-ai-image.sh to switch.

3. Workstation with a 24 GB GPU and 32+ GB RAM

qwen2.5-coder:32b (dense, best quality) or qwen3-coder:30b (MoE, faster).

Pros	Cons
Close to cloud-assistant quality for code	Large image (~20 GB) and long first build
Fully in VRAM: fast, with room for a 16k+ context	The GPU is busy while the model is loaded
MoE variant stays usable even if it spills into RAM	Needs the most expensive hardware; dense 32B is slow if it does not fit

4.9 KiB Raw Permalink Blame History Unescape Escape