local_ai

Local coding AI for aider: an Ollama container with a model chosen for this machine. Aider runs on the host and talks to the container.

./setup.sh            # analyse machine, pick model, generate Dockerfile + RUN_AIDER.md
./build-ai-image.sh   # build image (downloads the model)
./run-ai-image.sh     # start container (port 11434, or the next free one)
# then follow RUN_AIDER.md to start aider

• models.tsv – model catalogue (edit to add models)
• Dockerfile.template – container template; setup.sh fills in the model
• Dockerfile, .ai-config, RUN_AIDER.md – generated per machine (git-ignored)
• ./setup.sh --model <tag> --port <n> --gpu|--no-gpu --yes for non-interactive use

What decides which model fits

• Memory is the limit. A model must fit in VRAM (GPU) or RAM (CPU). setup.sh keeps about 4 GB of RAM free for the OS, aider and your editor, and about 10% of VRAM.
• Fits in VRAM = fast. The whole model on the GPU is by far the quickest option (fast (GPU) in the menu).
• Bigger than VRAM = hybrid. Ollama puts as many layers as fit on the GPU and runs the rest on the CPU (hybrid ~N% GPU, a rough estimate). Faster than CPU-only, but the CPU part sets the pace. Check the real split with docker exec local-ai ollama ps.
• CPU-only works, at a few tokens/second for a 7B model. Mixture-of-Experts (MoE) models such as qwen3-coder:30b only activate a few billion parameters per token, so they are much faster on CPU than dense models of the same size, but still need the RAM for all of it.
• Context window (4k to 32k tokens) is chosen from the memory budget and costs RAM/VRAM too. Aider needs room for the repo map plus your files, so 8k is a practical minimum.
• Status in the menu: fits, tight (fits with almost no headroom: close other apps), slow (dense model without a GPU), too big (will swap or crash).
• Port: the container uses 11434 like a normal Ollama. If that is taken (for example by a host Ollama), setup.sh picks the next free port and writes it into RUN_AIDER.md.

Using an NVIDIA GPU

Docker needs both the NVIDIA driver and nvidia-container-toolkit. setup.sh detects a GPU that is not usable yet and prints distro-specific install commands (also in RUN_AIDER.md), but only when a driver would actually help. Be careful with driver installs:

• Take a snapshot first (Timeshift) and know how to reach the GRUB menu.
• Prefer the distro's recommended driver (ubuntu-drivers devices), not an arbitrary version.
• With Secure Boot on you must enrol the MOK key at the first reboot (mokutil --sb-state).
• If the machine stops booting: boot an older kernel or add nomodeset in GRUB, then remove the packages (dpkg -l | awk '/^ii/ && /nvidia/ && !/linux-firmware-nvidia/ {print $2}' | xargs sudo dpkg --purge --force-depends, then sudo update-initramfs -u). No network is needed for that.
• After a successful install, verify with docker run --rm --gpus all ubuntu nvidia-smi and re-run ./setup.sh.

Example configurations

Speeds are rough orders of magnitude, not benchmarks. Your hardware will differ.

1. CPU-only laptop, 16 GB RAM, no usable GPU

qwen2.5-coder:7b, context 8k (this is what setup.sh picks on such a machine).

Pros	Cons
Works anywhere with Docker, no driver risk	A few tokens/second; long answers take a while
Good quality for its size on everyday edits	First response is slow while the model loads into RAM
Leaves ~9 GB RAM for the rest of the system	Keep prompts small and add only the files aider needs

2. Small GPU (about 4 GB VRAM, e.g. GTX 1650) with 16 GB RAM

Either qwen2.5-coder:3b (fully on the GPU) or qwen2.5-coder:7b (hybrid, ~55% on the GPU).

Pros	Cons
3B: fully in VRAM, fast and responsive	3B: noticeably weaker; use --edit-format whole in aider
7B hybrid: better answers, still sped up by the GPU	7B hybrid: speedup is modest, the CPU part dominates
Both fit comfortably in 16 GB RAM	Needs the NVIDIA driver + container toolkit (driver install can break boot)

A reasonable approach: start with the 7B hybrid for quality and switch to the 3B for quick, simple edits. Re-run ./setup.sh and ./build-ai-image.sh to switch.

3. Workstation with a 24 GB GPU and 32+ GB RAM

qwen2.5-coder:32b (dense, best quality) or qwen3-coder:30b (MoE, faster).

Pros	Cons
Close to cloud-assistant quality for code	Large image (~20 GB) and long first build
Fully in VRAM: fast, with room for a 16k+ context	The GPU is busy while the model is loaded
MoE variant stays usable even if it spills into RAM	Needs the most expensive hardware; dense 32B is slow if it does not fit