---
title: 'Model service (LLM)'
url: 'https://wiki.teletypegames.org/infrastructure/llm'
markdown: 'https://wiki.teletypegames.org/infrastructure/llm.md'
lang: en
date: '2026-10-21'
description: 'One place on the server where small language models run, for any stack that needs one. It speaks the OpenAI chat API, so any OpenAI-compatible client works with it. It is reachable only from the interstack docker network, not from the internet. Repository: git.teletypegames.org/infra/llm Address…'
taxonomy:
  tag:
    - howto
---

# Model service (LLM)

> One place on the server where small language models run, for any stack that needs one. It speaks the OpenAI chat API, so any OpenAI-compatible client works with it. It is reachable only from the `interstack` docker network, not from the internet.

- **Repository:** [git.teletypegames.org/infra/llm](https://git.teletypegames.org/infra/llm)
- **Address:** `http://llm:8080` on `interstack`

# Calling it

```
POST http://llm:8080/v1/chat/completions
Authorization: Bearer <key>
{"model": "qwen3.5-0.8b", "messages": [...], "stream": true}
```

Every request names its model: there is no default, a request without `model` gets a 400. `GET /v1/models` lists the models and whether each is loaded; `GET /health` needs no key.

| Model | `model` | Used by |
|---|---|---|
| Qwen3.5 0.8B | `qwen3.5-0.8b` | SERMO's Lapis-ACP in the [Teletype BBS](https://wiki.teletypegames.org/services/teletype-bbs), and any stack: a general small model, good at English and JSON, Hungarian only roughly; thinking off; loaded at start |
| Gemma 3 270M instruct | `gemma-3-270m` | Nobody now (SERMO's model until October 2026): the fastest, but it leaves a persona at any question beyond its examples; loaded on its first request |

# Models

The server is llama.cpp's `llama-server` in router mode: it starts a llama-server per model on demand. The models are the sections of `server/models.ini` in the repository, baked into the service's image; a section's name is the `model` of a request, and its keys are llama-server's long options (`hf-repo`, `c`, `parallel`, ...). The GGUF is downloaded from Hugging Face on the first load and kept in a volume.

- A model with `load-on-startup = true` is loaded when the service starts (Qwen3.5, so SERMO never waits). The others load on their first request.
- At most two models are loaded at once; a third unloads the least recently used.
- One memory limit (2.5 GB) covers all the loaded models (the two models above take about 2 GB together), and they share the server's CPU. A bigger model needs a bigger limit, and its long requests slow down the others.

Adding a model is a new section in `server/models.ini` and a push: the deploy builds a new image and restarts the service with it.

# Keys

Router mode can also load and unload models over HTTP, so the service asks for a key. `LLM_API_KEYS` in the stack's `.env` holds the accepted keys, one per caller, so one caller can be cut off alone.

| Caller | Its key |
|---|---|
| [Teletype BBS](https://wiki.teletypegames.org/services/teletype-bbs) | `SERMO_LLM_API_KEY` in the `teletype-bbs` stack's `.env` |

A new caller gets a new key (`openssl rand -hex 24`), appended to `LLM_API_KEYS`.

# Deploy

The stack lives in `/srv/stacks/llm`, which is not a git checkout. A push to master that changes `compose.yaml` or `server/` deploys it through Woodpecker, the same way as the [FTP server](https://wiki.teletypegames.org/infrastructure/ftp-server): the pipeline only calls the server over SSH with its own key, bound to one fixed command, `ci-deploy llm` (in `infra/host-config`). The server clones master itself, checks the compose file against its policy (no extra privileges, no docker socket, no public port), copies `compose.yaml` and `server/` in and runs `docker compose up -d --build`. `.env` and the model volume stay. The pipeline gets no host volumes.

'; var prevLevel = 2; for (var i = 0; i < headings.length; i++) { var h = headings\[i\]; var text = h.textContent.trim(); var slug = text.toLowerCase() .replace(/\[áàâä\]/g,'a').replace(/\[éèêë\]/g,'e').replace(/\[íìîï\]/g,'i') .replace(/\[óòôöő\]/g,'o').replace(/\[úùûüű\]/g,'u') .replace(/\[^a-z0-9\]+/g, '-').replace(/^-|-$/g, ''); if (!slug) slug = 'heading-' + i; if (used\[slug\]) { slug += '-' + i; } used\[slug\] = true; h.id = slug; var level = parseInt(h.tagName.charAt(1)); var cls = level > 2 ? ' class="toc-child"' : ''; html += '- [' + text + '](#'%20+%20slug%20+%20')
'; } html += '
'; toc.innerHTML = html; // Scroll-spy: highlight current heading var links = toc.querySelectorAll('a'); var ids = \[\]; for (var j = 0; j < links.length; j++) ids.push(links\[j\].getAttribute('href').slice(1)); function updateActive() { var current = ''; for (var k = 0; k < ids.length; k++) { var el = document.getElementById(ids\[k\]); if (el && el.getBoundingClientRect().top <= 80) current = ids\[k\]; } for (var m = 0; m < links.length; m++) { links\[m\].parentNode.classList.toggle('active', links\[m\].getAttribute('href') === '#' + current); } } window.addEventListener('scroll', updateActive, { passive: true }); updateActive(); })();

---

## Navigation

- Parent: [Infrastructure](https://wiki.teletypegames.org/infrastructure.md)
- Previous: [Central login (Authentik)](https://wiki.teletypegames.org/infrastructure/idp.md)
- Next: [Matomo](https://wiki.teletypegames.org/infrastructure/matomo.md)