## Why run a local model

- **Privacy.** Soul, Skills, and tool I/O never leave your network. Critical when an employee touches customer data, a Postgres replica, or finance records.
- **Cost.** No per-token bill. Routines that run on a tight cron can become uneconomical on the big closed models; locally hosted Llama / Qwen / Mistral don't care how often you call them.
- **Model choice.** The open ecosystem ships specialized weights — coding-tuned, retrieval-tuned, JSON-tuned — that the big labs don't. You can also stay on a stable model forever; nobody deprecates a checkpoint you downloaded.

## The shape of the integration

Genosyn's in-process agent loop talks to your model over an OpenAI-compatible HTTP API — no CLI to install, nothing to spawn. The runtime path is always:

```
Genosyn agent loop (in-process, runner + chat)
   └─ HTTP to an OpenAI-compatible /v1/chat/completions endpoint
        └─ your local server (Ollama / vLLM / llama.cpp / LM Studio)
             └─ the model weights on your GPU or Mac
```

So you need two pieces wired up:

1. A local server that exposes an OpenAI-compatible API. Most popular runtimes do this out of the box.
2. An AI Model on the employee, configured from Settings → AI Model with the provider kind set to `Custom`. Paste the base URL + model id (+ an optional key). Done.

## Step 1 — Run a local server

Pick one of these. They all expose `/v1/chat/completions` and `/v1/models` so any OpenAI-compatible client just works.

### Ollama (easiest)

Best for a Mac, a single GPU, or just kicking the tires. Ships a model registry and serves on `http://localhost:11434`.

```bash
# install
curl -fsSL https://ollama.com/install.sh | sh

# pull a tool-capable model
ollama pull qwen2.5-coder:32b

# serve (auto-starts on macOS; this is for Linux/manual)
ollama serve
```

Ollama exposes the OpenAI-compatible API at `http://localhost:11434/v1`.

### vLLM (best throughput)

For a real GPU box. Highest tokens/sec, batches concurrent requests across multiple employees. Native OpenAI-compatible server.

Prefer to run it by hand instead? The bare server is one pip install:

```bash
pip install vllm

vllm serve Qwen/Qwen2.5-Coder-32B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes
```

Endpoint: `http://<host>:8000/v1`. The two tool-call flags are required for MCP tool use to work — without them, vLLM will return tool calls as raw text and the agent will treat them as a normal message.

### llama.cpp (most portable)

Smallest dependency surface. CPU works; with a GPU it's fast too. Ships `llama-server` as its OpenAI-compatible endpoint.

```bash
# from a release binary, or build from source
llama-server \
  -m ./qwen2.5-coder-32b-instruct-q5_k_m.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --jinja \
  --chat-template-file qwen2.5-coder.jinja
```

Endpoint: `http://<host>:8080/v1`. `--jinja` enables the chat template required for tool calls.

### LM Studio (GUI-friendly)

Native macOS / Windows / Linux app. Browse models in a UI, flip the local server on in one click. Endpoint: `http://localhost:1234/v1`. Good for Mac users who don't want to live in the terminal.

## Step 2 — Wire it into Genosyn

From the app, either during the hire wizard (Step 2: Model) or afterwards at `Settings → AI Model`:

1. Pick the `Custom` provider kind — any OpenAI-compatible endpoint.
2. Base URL: `http://host.docker.internal:11434/v1` for Ollama on the same Mac/Windows host, or the full LAN URL of your GPU box.
3. Model id: the raw model name your server exposes — e.g. `qwen2.5-coder:32b`.
4. API key: leave blank for Ollama / vLLM / llama.cpp. Most local servers ignore the key entirely.
5. Click `Continue`. That's it — the next chat or routine run hits your endpoint.
6. Check the context window on the model card. Genosyn asks your server for it on save; if it reads Unknown, set it by hand — see below.

### Tell Genosyn your context window

A run budgets its history against the model's window, dropping the oldest tool results when the next prompt wouldn't fit. It can only do that if it knows the number, and self-hosted servers disagree about whether to publish one on `/v1/models`:

- vLLM Reports `max_model_len` — whatever you passed to `--max-model-len`. Detected automatically.
- LM Studio Reports `max_context_length`. Detected automatically.
- llama.cpp Reports `n_ctx` — what you passed to `-c`. Detected automatically.
- Ollama Reports nothing. Set it by hand: it defaults to a `num_ctx` of 4096 unless your Modelfile or `OLLAMA_CONTEXT_LENGTH` says otherwise — far smaller than the weights allow, and a common surprise.

Use Set manually on the model card for anything not detected. A number you type always wins over the probe, so it survives key rotations and re-saves; Clear hands the field back.

For everything Genosyn does detect, it re-asks your server every three hours — so restarting vLLM with a longer `--max-model-len`, or swapping the weights behind the same model id, lands on the card without you touching anything. A check that can't reach the box keeps the last known number, and Ask the provider runs it immediately when you don't want to wait.

## Docker networking

If you installed Genosyn through `genosyn install`, the app runs inside a Docker container. `localhost` inside the container is not the same as on your host — the LLM server on the host won't be reachable as `http://localhost:11434` from the employee.

macOS / Win Use `http://host.docker.internal:11434`. Docker Desktop wires this magic hostname to the host automatically.

Linux Add `--add-host=host.docker.internal:host-gateway` to your `docker run`, or run the LLM server bound to `0.0.0.0` and use the host's LAN IP.

Same machine, second container Easiest is a shared user-defined network (`docker network create genosyn-net`) and reference the LLM container by name.

## Hardware sizing

Rough rules of thumb for picking weights against your GPU memory. Quantized GGUFs at q4_k_m or q5_k_m are the practical baseline — the quality loss vs the original is small and the memory savings are large.

- **8–12 GB VRAM** 7B–8B models at q4. Good for chat. Light tool use; expect occasional plan errors on multi-step routines.
- **16–24 GB VRAM** 13B–14B at q5, or 32B at q4. The sweet spot for a single solid employee.
- **32–48 GB VRAM** 32B at q6/q8, or 70B at q4. Comparable to mid-tier closed models on most code/ops tasks.
- **80+ GB VRAM** (or M-series Mac with 64+ GB unified) 70B at q5+ or DeepSeek-V3 with offloading. State of the art for open weights.

## Troubleshooting

- **Model replies but never calls a tool.** Almost always the chat template / tool-call parser. Check that your server has tool-call support enabled (`--jinja` for llama.cpp, `--enable-auto-tool-choice` for vLLM, and a chat template that emits a function-call block).
- **Runs hang on the first message.** Network — the agent inside the container can't reach your host. See the Docker networking section above.
- **Model loops or hallucinates tool names.** Context window. Genosyn injects the Soul + every Skill + the MCP working set of tools at each turn; that's roughly 4k–5k tokens before the first user message. Run models with at least 32k context for serious work.
- **"This model's maximum context length is N tokens."** The prompt outgrew the window. Genosyn drops old tool results and retries once, so this shouldn't fail a run — but seeing `[compact]` with reason `overflow` in the log means it was caught late. Set the model's context window on its card and the next run budgets ahead instead of reacting.
- **Employee forgets what a tool told it earlier.** Look for `[compact]` in the run log: history was dropped to fit the window. Give the model a longer context, or trim the Skills and tools that ride along on every turn.
- **Slow.** Quantize down (q8 → q5), enable batching on vLLM, or pin the layers to GPU (`--n-gpu-layers` in llama.cpp). If your GPU is saturated, the answer is more hardware, not more tuning.