ml
92 lines
4.9 KiB
Markdown
92 lines
4.9 KiB
Markdown
# local_ai
|
||
|
||
<p align="center">
|
||
<img src="logo.png" alt="local_ai: a laptop running Qwen, DeepSeek and Granite coding models" width="320">
|
||
</p>
|
||
|
||
Local coding AI for [aider](https://aider.chat): an Ollama container with a model
|
||
chosen for *this* machine. Aider runs on the host and talks to the container.
|
||
|
||
```bash
|
||
./setup.sh # analyse machine, pick model, generate Dockerfile + RUN_AIDER.md
|
||
./build-ai-image.sh # build image (downloads the model)
|
||
./run-ai-image.sh # start container (port 11434, or the next free one)
|
||
# then follow RUN_AIDER.md to start aider
|
||
```
|
||
|
||
- `models.tsv` – model catalogue (edit to add models)
|
||
- `Dockerfile.template` – container template; `setup.sh` fills in the model
|
||
- `Dockerfile`, `.ai-config`, `RUN_AIDER.md` – generated per machine (git-ignored)
|
||
- `./setup.sh --model <tag> --port <n> --gpu|--no-gpu --yes` for non-interactive use
|
||
|
||
## What decides which model fits
|
||
|
||
- **Memory is the limit.** A model must fit in VRAM (GPU) or RAM (CPU). `setup.sh` keeps
|
||
about 4 GB of RAM free for the OS, aider and your editor, and about 10% of VRAM.
|
||
- **Fits in VRAM = fast.** The whole model on the GPU is by far the quickest option
|
||
(`fast (GPU)` in the menu).
|
||
- **Bigger than VRAM = hybrid.** Ollama puts as many layers as fit on the GPU and runs the
|
||
rest on the CPU (`hybrid ~N% GPU`, a rough estimate). Faster than CPU-only, but the CPU
|
||
part sets the pace. Check the real split with `docker exec local-ai ollama ps`.
|
||
- **CPU-only** works, at a few tokens/second for a 7B model. Mixture-of-Experts (MoE) models
|
||
such as `qwen3-coder:30b` only activate a few billion parameters per token, so they are
|
||
much faster on CPU than dense models of the same size, but still need the RAM for all of it.
|
||
- **Context window** (4k to 32k tokens) is chosen from the memory budget and costs RAM/VRAM
|
||
too. Aider needs room for the repo map plus your files, so 8k is a practical minimum.
|
||
- **Status in the menu:** `fits`, `tight` (fits with almost no headroom: close other apps),
|
||
`slow` (dense model without a GPU), `too big` (will swap or crash).
|
||
- **Port:** the container uses 11434 like a normal Ollama. If that is taken (for example by a
|
||
host Ollama), `setup.sh` picks the next free port and writes it into `RUN_AIDER.md`.
|
||
|
||
### Using an NVIDIA GPU
|
||
|
||
Docker needs both the **NVIDIA driver** and **nvidia-container-toolkit**. `setup.sh` detects
|
||
a GPU that is not usable yet and prints distro-specific install commands (also in
|
||
`RUN_AIDER.md`), but only when a driver would actually help. Be careful with driver installs:
|
||
|
||
- Take a snapshot first (Timeshift) and know how to reach the GRUB menu.
|
||
- Prefer the distro's recommended driver (`ubuntu-drivers devices`), not an arbitrary version.
|
||
- With Secure Boot on you must enrol the MOK key at the first reboot (`mokutil --sb-state`).
|
||
- If the machine stops booting: boot an older kernel or add `nomodeset` in GRUB, then
|
||
remove the packages (`dpkg -l | awk '/^ii/ && /nvidia/ && !/linux-firmware-nvidia/ {print $2}' | xargs sudo dpkg --purge --force-depends`,
|
||
then `sudo update-initramfs -u`). No network is needed for that.
|
||
- After a successful install, verify with `docker run --rm --gpus all ubuntu nvidia-smi` and
|
||
re-run `./setup.sh`.
|
||
|
||
## Example configurations
|
||
|
||
Speeds are rough orders of magnitude, not benchmarks. Your hardware will differ.
|
||
|
||
### 1. CPU-only laptop, 16 GB RAM, no usable GPU
|
||
|
||
`qwen2.5-coder:7b`, context 8k (this is what `setup.sh` picks on such a machine).
|
||
|
||
| Pros | Cons |
|
||
| --- | --- |
|
||
| Works anywhere with Docker, no driver risk | A few tokens/second; long answers take a while |
|
||
| Good quality for its size on everyday edits | First response is slow while the model loads into RAM |
|
||
| Leaves ~9 GB RAM for the rest of the system | Keep prompts small and add only the files aider needs |
|
||
|
||
### 2. Small GPU (about 4 GB VRAM, e.g. GTX 1650) with 16 GB RAM
|
||
|
||
Either `qwen2.5-coder:3b` (fully on the GPU) or `qwen2.5-coder:7b` (hybrid, ~55% on the GPU).
|
||
|
||
| Pros | Cons |
|
||
| --- | --- |
|
||
| 3B: fully in VRAM, fast and responsive | 3B: noticeably weaker; use `--edit-format whole` in aider |
|
||
| 7B hybrid: better answers, still sped up by the GPU | 7B hybrid: speedup is modest, the CPU part dominates |
|
||
| Both fit comfortably in 16 GB RAM | Needs the NVIDIA driver + container toolkit (driver install can break boot) |
|
||
|
||
A reasonable approach: start with the 7B hybrid for quality and switch to the 3B for quick,
|
||
simple edits. Re-run `./setup.sh` and `./build-ai-image.sh` to switch.
|
||
|
||
### 3. Workstation with a 24 GB GPU and 32+ GB RAM
|
||
|
||
`qwen2.5-coder:32b` (dense, best quality) or `qwen3-coder:30b` (MoE, faster).
|
||
|
||
| Pros | Cons |
|
||
| --- | --- |
|
||
| Close to cloud-assistant quality for code | Large image (~20 GB) and long first build |
|
||
| Fully in VRAM: fast, with room for a 16k+ context | The GPU is busy while the model is loaded |
|
||
| MoE variant stays usable even if it spills into RAM | Needs the most expensive hardware; dense 32B is slow if it does not fit |
|