warpstore

A store engine reads a WarpEngine catalog and writes some host's own game library format. Only that last step differs per host. Talking to the API, picking a release, matching the machine and remembering what went where is the same work everywhere — warpstore is that work, in one file, shared by every store engine.

                    ┌──────────────────────────────┐
                    │  warpstore.py                │
                    │  catalog · releases · host   │
                    │  state · config · HTTP       │
                    └──────────────┬───────────────┘
                    ┌──────────────┴───────────────┐
                    ▼                              ▼
    warp-engine-batocera-store        warp-engine-retroarch-store
    ROM folders + gamelist.xml        .lpl playlists + thumbnails
                    │                              │
                    ▼                              ▼
             EmulationStation                  RetroArch

Repository: https://git.teletypegames.org/engines/warpstore Catalog side: WarpEngine Users: Batocera store engine · RetroArch store engine

Why it exists

The Batocera store was 1073 lines, and only about a third was about Batocera. The RetroArch store needed the other two thirds, so the shared part moved out — with a second consumer already in hand, so the seam sits where the two engines really differ.

Single file, Python 3 standard library only: the hosts range from a Batocera box (python3, no pip) to a desktop, and neither may be asked to install anything. An engine's installer drops warpstore.py next to the adapter script, and the adapter does import warpstore.

What an adapter supplies

1. DEFAULT_CONFIG — where things live on this host, which catalog platform maps to what.
2. accept(spec, sw) — which catalog entries this host can run, and how they are grouped.
3. The code that writes the host's library format.

import warpstore as ws

def accept(spec, sw):
    """Return (scope, extras), or raise ws.Skip(reason) to reject the title."""
    if spec["system"] not in available_systems:
        raise ws.Skip(f"no '{spec['system']}' ROM folder on this box")
    return spec["system"], {"system": spec["system"]}

cfg = ws.load_config(path, DEFAULT_CONFIG, required=("paths.subfolder",))
ws.init(path)
games, skipped = ws.select_games(cfg, ws.fetch_catalog(cfg, use_cache=True), accept)

scope is how the host groups its library — a Batocera system, a RetroArch playlist. state.json is keyed by <scope>:<name>, so the same software name in two scopes stays two entries.

What is in it

Five of them carry decisions worth knowing.

select_games vs. survey_catalog — select_games returns what the store can install. survey_catalog returns that plus one record per title this machine cannot install, with a reason code (platform_off, host_asset, no_asset, vetoed, and from 1.5.0 gated — a title the host sells, which an anonymous store cannot download; the record then carries its purchase_url) and a sentence. A client can grey those out instead of hiding them, which is what the WarpEngine Client does in its own port. Titles the store chooses not to offer are in neither list — a wrong status or an exclude list is editorial, not a limit of the machine.

fetch_catalog asks the host to filter by owner_id and catalog.statuses itself (status=released,archived), and follows nextPage if the host pages the answer anyway — so a host-side page size can never quietly cut a sync short. Both only shorten the download: survey_catalog applies the same filters again, so a host that ignores a parameter changes nothing.

download_url points at GET /api/download rather than /file/, so the host sees every download. Whether it counts them is the host's call: teletypegames.org deliberately leaves the warpstore/* user agent (user_agent) out of its download counts (ignored_user_agents), so installs by a store do not show up in the statistics.

resolve_for_host(value, host) — a config value that may depend on the machine. A plain string is the same everywhere, which is what a cartridge is: data for an emulator. Otherwise the value is a map and the most specific key wins:

{"linux-aarch64": …, "aarch64": …, "linux": …, "*": …}

No matching key and no * gives None, and the caller reports the title as skipped instead of installing something that cannot run. This generalises the Batocera engine's resolve_for_arch: Batocera only ever answered linux, but a RetroArch host may be Windows, macOS or Android — which decides what a libretro core file is called (.dll, .dylib, _android.so).

within(path, root) — guards every delete. A store may only remove files from the subtree it owns, never from the user's library and never from another store's.

prune_empty_dirs(dirs, root) — the uninstall side of the same rule. Only empty directories go, and only inside root: one stray file keeps a directory. This is what lets purge (and so uninstall.sh) leave the host as it was.

write_atomic — nothing is written in place. A store interrupted mid-sync would otherwise leave a half-written playlist or gamelist, which is worse than an old one.

Taking every store off a machine

curl -fsSL https://git.teletypegames.org/engines/warpstore/raw/branch/master/uninstall.sh | sh

Use this to strip the whole framework without remembering what is installed: it finds every store home under the known roots, has each store's own engine purge what it installed, then deletes the store, its launcher and the engine files. For a single store, prefer that engine's own uninstall.sh — it knows the host extras, like a Batocera Ports entry.

If an engine cannot finish (RetroArch running, ROMs root unmounted) the run stops there, rather than deleting the engine that knows what it installed.

The script is POSIX sh, not bash, so | sh works where /bin/sh is dash. The engines' own installers are too.

State

state.json, version 2, keyed <scope>:<name>:

{
  "version": 2,
  "installed": {
    "c64:blessingofra": { "name": "blessingofra", "scope": "c64", "asset": "blessingofra-2.0.0.prg", "…": "…" }
  }
}

record_scope() falls back to a record's system field, which is what the Batocera store wrote before this module existed. There the two were the same string, so an installed box keeps working with no migration and re-downloads nothing. A version: 1 file, keyed by bare software name, is re-keyed on first run.

Versioning

The engines fetch warpstore.py from wherever their installer's WARPSTORE_RAW_BASE points — master by default, so a change here reaches a store on its next install, which is also the reason to keep the surface small. A store repository may pin it instead: the TTG Batocera Store fetches both its engine and warpstore at fixed commits, and bumps them after a release. --version on either engine prints both numbers, e.g. 3.1.0 (warpstore 1.2.0).