warpstore

A store engine reads a WarpEngine catalog and writes some host's own game library format. Only that last step differs per host. Talking to the API, picking a release, matching the machine and remembering what went where is the same work everywhere — warpstore is that work, in one file, shared by every store engine.

                    ┌──────────────────────────────┐
                    │  warpstore.py                │
                    │  catalog · releases · host   │
                    │  state · config · HTTP       │
                    └──────────────┬───────────────┘
                    ┌──────────────┴───────────────┐
                    ▼                              ▼
    warp-engine-batocera-store        warp-engine-retroarch-store
    ROM folders + gamelist.xml        .lpl playlists + thumbnails
                    │                              │
                    ▼                              ▼
             EmulationStation                  RetroArch

Repository: https://git.teletypegames.org/engines/warpstore Catalog side: WarpEngine Users: Batocera store engine · RetroArch store engine

Why it exists

The Batocera store was 1073 lines, and only about a third was about Batocera. The RetroArch store needed the other two thirds, so the shared part moved out — with a second consumer already in hand, so the seam sits where the two engines really differ.

Single file, Python 3 standard library only: the hosts range from a Batocera box (python3, no pip) to a desktop, and neither may be asked to install anything. An engine's installer drops warpstore.py next to the adapter script, and the adapter does import warpstore.

What an adapter supplies

1. DEFAULT_CONFIG — where things live on this host, which catalog platform maps to what.
2. accept(spec, sw) — which catalog entries this host can run, and how they are grouped.
3. The code that writes the host's library format.

import warpstore as ws

def accept(spec, sw):
    """Return (scope, extras), or raise ws.Skip(reason) to reject the title."""
    if spec["system"] not in available_systems:
        raise ws.Skip(f"no '{spec['system']}' ROM folder on this box")
    return spec["system"], {"system": spec["system"]}

cfg = ws.load_config(path, DEFAULT_CONFIG, required=("paths.subfolder",))
ws.init(path)
games, skipped = ws.select_games(cfg, ws.fetch_catalog(cfg, use_cache=True), accept)

scope is how the host groups its library — a Batocera system, a RetroArch playlist. state.json is keyed by <scope>:<name>, so the same software name in two scopes stays two entries.

What is in it

Five of them carry decisions worth knowing.

select_games vs. survey_catalog — select_games returns what the store can install. survey_catalog returns that plus one record per title this machine cannot install, with a reason code (platform_off, host_asset, no_asset, vetoed) and a sentence. A client can grey those out instead of hiding them, which is what the WarpEngine Client does. Titles the store chooses not to offer are in neither list — a wrong status or an exclude list is editorial, not a limit of the machine.

resolve_for_host(value, host) — a config value that may depend on the machine. A plain string is the same everywhere, which is what a cartridge is: data for an emulator. Otherwise the value is a map and the most specific key wins:

{"linux-aarch64": …, "aarch64": …, "linux": …, "*": …}

No matching key and no * gives None, and the caller reports the title as skipped instead of installing something that cannot run. This generalises the Batocera engine's resolve_for_arch: Batocera only ever answered linux, but a RetroArch host may be Windows, macOS or Android — which decides what a libretro core file is called (.dll, .dylib, _android.so).

within(path, root) — guards every delete. A store may only remove files from the subtree it owns, never from the user's library and never from another store's.

prune_empty_dirs(dirs, root) — the uninstall side of the same rule. Only empty directories go, and only inside root: one stray file keeps a directory. This is what lets purge (and so uninstall.sh) leave the host as it was.

write_atomic — nothing is written in place. A store interrupted mid-sync would otherwise leave a half-written playlist or gamelist, which is worse than an old one.

Taking every store off a machine

curl -fsSL https://git.teletypegames.org/engines/warpstore/raw/branch/master/uninstall.sh | sh

Use this to strip the whole framework without remembering what is installed: it finds every store home under the known roots, has each store's own engine purge what it installed, then deletes the store, its launcher and the engine files. For a single store, prefer that engine's own uninstall.sh — it knows the host extras, like a Batocera Ports entry.

If an engine cannot finish (RetroArch running, ROMs root unmounted) the run stops there, rather than deleting the engine that knows what it installed.

The script is POSIX sh, not bash, so | sh works where /bin/sh is dash. The engines' own installers are too.

State

state.json, version 2, keyed <scope>:<name>:

{
  "version": 2,
  "installed": {
    "c64:blessingofra": { "name": "blessingofra", "scope": "c64", "asset": "blessingofra-2.0.0.prg", "…": "…" }
  }
}

record_scope() falls back to a record's system field, which is what the Batocera store wrote before this module existed. There the two were the same string, so an installed box keeps working with no migration and re-downloads nothing. A version: 1 file, keyed by bare software name, is re-keyed on first run.

Versioning

Neither engine pins a version: both fetch warpstore.py from master at install time, so a change here reaches every store on its next install. That is the point, and also the reason to keep the surface small. --version on either engine prints both numbers, e.g. 3.1.0 (warpstore 1.2.0).