Model service (LLM)

One place on the server where small language models run, for any stack that needs one. It speaks the OpenAI chat API, so any OpenAI-compatible client works with it. It is reachable only from the interstack docker network, not from the internet.

Repository: git.teletypegames.org/infra/llm
Address: http://llm:8080 on interstack

Calling it

TXT

POST http://llm:8080/v1/chat/completions
Authorization: Bearer <key>
{"model": "qwen3.5-0.8b", "messages": [...], "stream": true}

Every request names its model: there is no default, a request without model gets a 400. GET /v1/models lists the models and whether each is loaded; GET /health needs no key.

Model	`model`	Used by
Qwen3.5 0.8B	`qwen3.5-0.8b`	SERMO's Lapis-ACP in the Teletype BBS, and any stack: a general small model, good at English and JSON, Hungarian only roughly; thinking off; loaded at start
Gemma 3 270M instruct	`gemma-3-270m`	Nobody now (SERMO's model until October 2026): the fastest, but it leaves a persona at any question beyond its examples; loaded on its first request

Models

The server is llama.cpp's llama-server in router mode: it starts a llama-server per model on demand. The models are the sections of server/models.ini in the repository, baked into the service's image; a section's name is the model of a request, and its keys are llama-server's long options (hf-repo, c, parallel, ...). The GGUF is downloaded from Hugging Face on the first load and kept in a volume.

A model with load-on-startup = true is loaded when the service starts (Qwen3.5, so SERMO never waits). The others load on their first request.
At most two models are loaded at once; a third unloads the least recently used.
One memory limit (2.5 GB) covers all the loaded models (the two models above take about 2 GB together), and they share the server's CPU. A bigger model needs a bigger limit, and its long requests slow down the others.

Adding a model is a new section in server/models.ini and a push: the deploy builds a new image and restarts the service with it.

Keys

Router mode can also load and unload models over HTTP, so the service asks for a key. LLM_API_KEYS in the stack's .env holds the accepted keys, one per caller, so one caller can be cut off alone.

Caller	Its key
Teletype BBS	`SERMO_LLM_API_KEY` in the `teletype-bbs` stack's `.env`

A new caller gets a new key (openssl rand -hex 24), appended to LLM_API_KEYS.

Deploy

The stack lives in /srv/stacks/llm, which is not a git checkout. A push to master that changes compose.yaml or server/ deploys it through Woodpecker, the same way as the FTP server: the pipeline only calls the server over SSH with its own key, bound to one fixed command, ci-deploy llm (in infra/host-config). The server clones master itself, checks the compose file against its policy (no extra privileges, no docker socket, no public port), copies compose.yaml and server/ in and runs docker compose up -d --build. .env and the model volume stay. The pipeline gets no host volumes.