EN /HU | Login

Backup

Two chains: a Makefile-based backup for database dumps and Gitea repository mirrors, followed by an rsync-based mirror of the whole file tree to an off-site NAS. This has been the setup since 2026-10-16. Before that the second chain was restic; we retired it: the backup is simpler now, at the cost of having no versioning.

Daily schedule

Time What Tool
04:00 Redmine (MariaDB), catalog and Gitea (MySQL) dumps → mirror every Gitea repository → delete dumps older than 7 days make backup (cron, infra/backup-scripts)
05:00 Matomo, ResourceSpace and Uptime Kuma dumps → Woodpecker and Grafana SQLite copies → package list → full file mirror to the NAS vps-backup-sync (systemd timer, infra/host-config)

The mirror runs an hour later so it carries the fresh dumps and repository mirrors too. The 04:00 chain does not sync on its own; everything reaches the NAS with the 05:00 mirror.

Chain 1: dumps and repository mirrors (Makefile)

Environment variables in the Makefile:

Variable Meaning
BACKUP_BASE Root of all backups (default /srv/backup)
BACKUP_KEEP_DAYS Retention for DB dumps (default 7)
DATABASE_BACKUP_BASE Where SQL dumps go
REPOSITORY_BACKUP_BASE Where mirrored repos go
GITEA_SERVER, GITEA_TOKEN Gitea API for listing repositories; the token comes from an .envrc file the Makefile reads itself (-include), so it works under cron too

Targets

BASH
make backup                          # db_redmine -> db_catalog -> db_gitea -> repos -> db_cleanup
make db_redmine db_catalog db_gitea  # databases only
make repos                           # update the git mirrors
make db_cleanup                      # drop expired dumps

Database backups

Each dump runs inside the service's container. Dumps are timestamped into /db_backup in the container, which is mounted to the host's DATABASE_BACKUP_BASE.

Service How Note
Redmine (MariaDB) database/mariadb_backup.sh MARIADB_USER, MARIADB_PASSWORD, MARIADB_DATABASE in the container
Catalog (MySQL, softwares) database/mysql_backup.sh MYSQL_ROOT_PASSWORD, MYSQL_DATABASE in the container
Gitea (MySQL, gitea) db_gitea in the Makefile: mysqldump --single-transaction --routines --triggers Since 2026-10-16. Before that the Gitea database (users, permissions, issues, PRs, OAuth apps, registry metadata) was not backed up anywhere.
Wiki (PostgreSQL) database/pg_backup.sh Retired 2026-09-11 – the Wiki.js database is gone; the script is legacy.

Repository backups

make repos lists every private and public repository from the Gitea API, then per repo: fetch --all --prune + pull when it exists locally, git clone when it does not.

Cleanup

make db_cleanup groups dumps by database prefix and deletes files older than BACKUP_KEEP_DAYS. A safety check keeps at least one backup per database, even when it is older than the retention window.

Chain 2: file mirror (vps-backup-sync)

The /usr/local/sbin/vps-backup-sync script (source in the infra/host-config repository) runs as root, started by vps-backup-sync.timer.

Dumps at the start of the script: Matomo (mariadb-dump), ResourceSpace (--all-databases), Uptime Kuma (sqlite3 .backup). For the live Woodpecker and Grafana SQLite databases, sqlite3 .backup writes a consistent copy next to the file (woodpecker-backup.sqlite, grafana-backup.db). The list of installed packages goes to /srv/backup/system/ (dpkg --get-selections, apt-mark showmanual).

What it mirrors: /srv/stacks, /srv/stacks_disabled, /srv/backup, /srv/host-config, /srv/presentation, /srv/dockge, the host's /etc, /root, /home, /usr/local/sbin, /var/spool/cron and /var/www, plus the ResourceSpace filestore and Matomo Docker volumes. About 27 GB in total.

What it skips: live database directories (the dumps stand in for them, since a running DB's files are not consistent), node_modules, tmp, log, Gitea search indexes, caches.

How: rsync -aH --numeric-ids --relative --delete into a mode-700 directory on the NAS, with absolute paths. A non-root user receives on the NAS, so owner, permissions and device data are stored in xattrs (-M--fake-super). Copy back with the same option, otherwise every file ends up owned by the NAS user.

No versioning. The NAS always holds the latest state only. If something is deleted or corrupted on the server, it is deleted or corrupted on the NAS the next night too. For databases, the 7-day dump rotation gives some history.

Off-site mirror

The target is an external Synology NAS (DS212j, DSM 6.2, ext4). It only accepts ssh-rsa keys, has no SFTP subsystem and cannot do Btrfs snapshots, so NAS-side versioning would only be possible with hardlinked copies.

Open question: the mirror is writable from the server, so a compromised host could delete the backup too. Secrets (/root, /etc/shadow, .env files, certificate keys) sit unencrypted on the NAS.

Restoring to a new server

The exact commands are in the infra/host-config README, section „Mentés és visszaállítás”. In short:

  1. Copy the whole tree back as root, with -M--fake-super, into a temporary directory.
  2. Install packages from /srv/backup/system/apt-manual.txt, then put /etc back selectively and restore /srv, /home, /root, /var/www, /var/spool/cron, /usr/local/sbin.
  3. sudo /srv/host-config/install.sh for the host's own files.
  4. Start the stacks with empty databases, then load the latest dumps (redmine_production, softwares, gitea, matomo, resourcespace).
  5. SQLite: move the *-backup.* copies into place with the stack stopped.
  6. Docker volumes: restore the ResourceSpace filestore and Matomo volume contents.

File structure

The backup repository (infra/backup-scripts):

TEXT
.
├── Makefile                # every chain-1 task
└── database/
    ├── mariadb_backup.sh
    ├── mysql_backup.sh
    └── pg_backup.sh        # legacy, no longer runs

Chain 2 lives in the infra/host-config repository: files/usr/local/sbin/vps-backup-sync, files/etc/systemd/system/vps-backup-sync.{service,timer}, files/etc/logrotate.d/vps-backup-sync. Install with sudo ./install.sh.

On the server (BACKUP_BASE=/srv/backup):

TEXT
/srv/backup/
├── database/                     # dumps, 7 days
│   ├── gitea-2026-10-16-02-00.sql
│   ├── redmine_production-2026-10-16-02-00.sql
│   ├── softwares-2026-10-16-02-00.sql
│   ├── matomo-2026-10-16-05-03.sql
│   └── resourcespace-2026-10-16-05-03.sql
├── repositories/                 # mirrors of every Gitea repository
├── system/                       # package list (dpkg, apt-mark)
└── archive-2026-10-15/           # tar.gz archives of volumes removed during cleanup

Verification

  • The mirror log on the host (/var/log/vps-backup-sync.log): every step ends with an OK or HIBA (error) line.
  • After an error-free run the script sends an Uptime Kuma push; if none arrives for 26 hours, an alert goes to the Discord Alert channel.
  • Grafana: the VPS: Backup dashboard shows the last run state, dump age and size, mirror size and the backup log lines. The script writes the metrics for node-exporter's textfile collector (/srv/stacks/observability/textfile/vps_backup.prom); the log lines are in Loki ({service_name="cron-jobs", program="vps-backup-sync"}). Grafana: https://grafana.teletypegames.org (sign in with your Gitea account).
  • If the newest dump of a database gets old, Grafana's Database dump stale alert goes to Discord.
  • Every database must have a fresh file for every day in the dump directory; if a prefix disappears or stops, the chain is stuck.

Repositories

  • https://git.teletypegames.org/infra/backup-scripts
  • https://git.teletypegames.org/infra/host-config