--- title: 'Data and network' url: 'https://wiki.teletypegames.org/engines/rubbs/data-and-network' markdown: 'https://wiki.teletypegames.org/engines/rubbs/data-and-network.md' lang: en date: '2026-11-02' description: 'What every content board of ours used to write for itself, in rubbs since 0.33.0 and 1.0.0: one log format, work in the background, an SQLite file with migrations, bounded HTTP, a cache, rate limits, a paced Wikipedia client, control-free text, and the rule of where a failure is handled. Each bloc…' taxonomy: tag: - reference --- # Data and network > What every content board of ours used to write for itself, in [rubbs](https://wiki.teletypegames.org/engines/rubbs) since 0.33.0 and 1.0.0: one log format, work in the background, an SQLite file with migrations, bounded HTTP, a cache, rate limits, a paced Wikipedia client, control-free text, and the rule of where a failure is handled. Each block is a few lines to adopt; the full signatures are in the [API reference](https://wiki.teletypegames.org/rdoc/bbs/latest/). # The config fields These fields of `BBS.configure` belong to this page (the rest are on [Getting started](https://wiki.teletypegames.org/engines/rubbs/getting-started) and [Telnet and the store](https://wiki.teletypegames.org/engines/rubbs/telnet-and-store)): | Field | Default | Meaning | |---|---|---| | `log_tag` | `ENV['BBS_LOG_TAG']` or `bbs` | What every log line starts with: the board's technical name, `feedia-bbs`. | | `user_agent` | `rubbs/VERSION (+https://teletypegames.org)` | The `User-Agent` of `BBS::HTTP` requests. | | `max_sessions_per_ip` | `ENV['BBS_MAX_SESSIONS_PER_IP']` or `6` | Calls at once from one public address; the next one is told so and closed (`BBS crowded` in the log). | | `ai_concurrency` | `2` | `BBS::AI::Client` requests running at once in the process; the next waits up to 5 s, then gets `BBS::AI::Busy` (see [AI characters](https://wiki.teletypegames.org/engines/rubbs/ai)). | ```ruby BBS.configure do |c| c.log_tag = 'feedia-bbs' c.user_agent = 'Feedia BBS (+https://teletypegames.org)' end ``` # Logging and background work ## Logging Every line the library and the board write to the log goes through `BBS.log(tag, message)`, to stderr, so the lines of eleven boards in one log read the same. An exception as the message is written as its class and message. ```ruby BBS.log('catalog refresh', 'done in 3.2 s') # feedia-bbs: catalog refresh: done in 3.2 s BBS.log('catalog refresh', error) # feedia-bbs: catalog refresh: SocketError: … ``` The library's own lines use the same format: `session`, `door …`, `gateway …`, `store`, `BBS busy`, `BBS crowded`. A `BBS::DiscordWebhook`'s `label:` is the log tag too, so it no longer needs the board's slug. ## Background work Work out of the callers' way (an index loaded at boot, a catalog refreshed every hour) runs in a `BBS::Background` thread. An error is logged and the block is tried again after `retry_after` seconds (5), the pause doubling up to `max_retry` (300): a service that is down costs a few log lines, not a crashed board or a busy loop. ```ruby BBS.background('index') { wiki.load_index } # until it succeeds once BBS.background('catalog refresh', every: 3600) { library.refresh } # every hour ``` Both return the `Thread`; `BBS::Background.new(name, …) { … }.run` runs the same work in the calling thread (a test, a command-line tool). # Storage ## `BBS::SQLite` One SQLite file and its one connection, shared by the callers' threads behind a lock: WAL mode (a command-line tool of the board can use the file meanwhile), a busy timeout, foreign keys on. The schema is a list of migrations numbered by `PRAGMA user_version`: a file runs only the ones it has not had yet, each in a transaction. A migration is an SQL string or a lambda; never edit one that has run, append a new one. ```ruby DB = BBS::SQLite.new('data/board.sqlite3') # creates data/ when needed DB.migrate( 'CREATE TABLE posts (id INTEGER PRIMARY KEY, title TEXT NOT NULL)', 'ALTER TABLE posts ADD COLUMN body TEXT', ->(db) { db.insert('INSERT INTO posts (title) VALUES (?)', 'Welcome') } ) DB.query('SELECT * FROM posts') # => [{ 'id' => 1, 'title' => 'Welcome', 'body' => nil }] DB.get('SELECT * FROM posts WHERE id = ?', 1) # the first row or nil DB.insert('INSERT INTO posts (title) VALUES (?)', 'Hi') # => the new rowid DB.change('DELETE FROM posts WHERE id = ?', 2) # => rows changed DB.transaction { … } # a nested call joins the outer one ``` ## `BBS::Store` The one-table, upsert-by-key store of a Flow's `persist` is built on `BBS::SQLite` since 0.33.0. A `.csv` path means the `.sqlite3` file beside it, and the CSV file of an older board is imported on first use (broken rows set aside in a `.rejected` file); `store.open!` does it at boot. See [Telnet and the store](https://wiki.teletypegames.org/engines/rubbs/telnet-and-store). # Network ## `BBS::HTTP` Bounded requests to the services a board shows. Every failure raises `BBS::HTTP::Error`, whose message names the address without its query string (so no token reaches a log); a body over `max_bytes` raises its subclass `BBS::HTTP::TooLarge`. Statuses other than 2xx are returned, not raised. ```ruby response = BBS::HTTP.get('https://example.org/feed.xml', headers: { 'Accept' => 'application/rss+xml' }) response.ok? # => true for 2xx response.status # => 200 response.body # => binary String BBS::HTTP.post(url, body: URI.encode_www_form(query: q), headers: { 'Content-Type' => 'application/x-www-form-urlencoded' }) File.open(path, 'wb') { |file| BBS::HTTP.get(url, to: file, max_bytes: 64 * 1024 * 1024) } ``` | Option | Default | | |---|---|---| | `open_timeout` / `read_timeout` | 5 / 15 s | per connection / per read | | `deadline` | 30 s | the whole request, redirects included | | `max_bytes` | 4 MiB | the body is streamed and the request fails above it | | `redirects` | 3 (`post`: 0) | followed to `http`/`https` only | | `public_only` | `true` | refuses hosts that resolve to loopback, private, link-local, CGNAT, ULA and other non-public ranges; connects to the vetted address | | `to` | `nil` | an IO the 2xx body is written to as it comes (`body` is `nil` then) | ## `BBS::Cache` TTL, least-recently-used eviction, a byte cap, one computation per key (callers missing the same key at once wait for it and share its value), and the expired value served when the computation raises. ```ruby ARTICLES = BBS::Cache.new(ttl: 600, max_entries: 500, max_bytes: 8 * 1024 * 1024) ARTICLES.fetch([lang, title]) { repository.article(lang, title) } BBS.background('catalog', every: 3600) { CATALOG.refresh(:all) { repository.load } } ``` `refresh(key) { … }` replaces the kept value only when the block succeeds, so a background refresh never empties the cache. `delete(key)`, `clear`, `size`. ## `BBS::RateLimit` A sliding window per key, thread-safe; quiet keys are forgotten. `caller_key(session)` is the caller's IP address on the internet, and the line's session id behind a local proxy (bbs-web), whose address is shared by everyone. ```ruby TIPS = BBS::RateLimit.new(limit: 3, per: 3600) key = BBS::RateLimit.caller_key(session) TIPS.allow?(key) # => true, and counted; false when over the limit TIPS.wait(key) # => seconds until the next event is allowed PER_CALLER = BBS::RateLimit.new(limit: 5, per: 600) PER_BOARD = BBS::RateLimit.new(limit: 100, per: 600) BBS::RateLimit.allow_all?([[PER_CALLER, key], [PER_BOARD, :all]]) # counted in all or none LOGINS = BBS::RateLimit.new(limit: 10, per: 600, store: DB, name: 'logins') return locked_out unless LOGINS.room?(key) LOGINS.record(key) unless password_ok?(name, password) ``` - `allow_all?` checks several limits at once (the caller's and the whole board's): the event is counted in every limit or, when one has no room, in none. - `room?` checks and `record` counts apart, for an event that counts only when it went wrong (a bad password). - With `store:` (a `BBS::SQLite`) the events are kept in its `rate_limit_events` table under `name:`, so a login lockout survives a restart. ## `BBS::Wikipedia` Wikipedia's Action API as a board asks it: through `BBS::HTTP`, with the board's User-Agent (Wikipedia may answer 403 without one), paced so that all callers together send at most `rate` requests a second; a request that would wait more than 3 s for its turn is not sent. The answer comes parsed (`format=json`, `formatversion=2` added); a failure raises `BBS::Wikipedia::Error`, a `BBS::HTTP::Error`. Alexandria BBS and Commodex BBS use it. ```ruby WIKIPEDIA = BBS::Wikipedia.new(user_agent: 'Alexandria BBS (+https://teletypegames.org)', rate: 5) WIKIPEDIA.get('en', action: 'query', list: 'search', srsearch: 'Commodore 64') # => { "query" => { "search" => [...] } } ``` ## The gateway guard and translation `BBS::Gateway` (see [Doors](https://wiki.teletypegames.org/engines/rubbs/doors)) resolves the host once and, since 1.0.0, calls only a public address: a listing that names `127.0.0.1` or a private address cannot turn the board into a relay into its own network. A board that dials its own neighbours says so; a C64 can call a board made for ASCII and ANSI terminals with `translate: true` (since 0.34.0). ```ruby BBS::Gateway.new(host: 'host.docker.internal', port: 2327, name: 'Feedia', public_only: false) BBS::Gateway.new(host: board.host, port: board.port, translate: !board.petscii?).call(session) ``` # Text and addresses - `BBS::Text.clean(text)`: valid UTF-8 without C0 controls (tab and line feed kept), DEL, C1 and the bidirectional override characters. With `keep_newlines: false` each line end becomes a space, so the words of two lines stay apart. - `BBS::Text.line(text)`: one clean line, whitespace runs (line ends and tabs too) one space, no space at either end. For a title, a name, a summary. - `BBS::Text.printable?(char)`: the rule of every input field. - `BBS::Address.public?(ip)` (on the public internet) and `BBS::Address.local?(ip)` (private, loopback, link-local). The frame buffer stores no control character either, so text from a service cannot move or restyle the caller's screen. # Where an error is handled A board's layers handle a failure of a service in one place: - a **repository** (`lib/domain/repository`) talks to the outside world and lets its failures go up: `BBS::HTTP::Error` (and `BBS::Wikipedia::Error`) as it comes, or a domain error of the board for what is wrong with the answer itself. It does not turn them into `nil` or an empty list, and writes no text for the caller; - a **service** decides what the caller sees: it rescues those errors, writes one `BBS.log` line, and returns what the UI shows (a cached answer, an empty result, a message of its own); - the **UI** never rescues `BBS::HTTP::Error`; it shows what the service gave. So an error message with a URL or a token never reaches a caller, and each failure is logged once. ```ruby class Library def search(query) repository.search(query) rescue BBS::HTTP::Error => e BBS.log('search', e) [] end end ``` # See also - [rubbs](https://wiki.teletypegames.org/engines/rubbs) — the overview - [Telnet and the store](https://wiki.teletypegames.org/engines/rubbs/telnet-and-store) — the session, the limits and `BBS::Store` - [Doors](https://wiki.teletypegames.org/engines/rubbs/doors) — gateways, ZMODEM and XMODEM - [AI characters](https://wiki.teletypegames.org/engines/rubbs/ai) — the model client and its concurrency limit - [Changelog](https://wiki.teletypegames.org/engines/rubbs/changelog) — when each block arrived '; var prevLevel = 2; for (var i = 0; i < headings.length; i++) { var h = headings\[i\]; var text = h.textContent.trim(); var slug = text.toLowerCase() .replace(/\[áàâä\]/g,'a').replace(/\[éèêë\]/g,'e').replace(/\[íìîï\]/g,'i') .replace(/\[óòôöő\]/g,'o').replace(/\[úùûüű\]/g,'u') .replace(/\[^a-z0-9\]+/g, '-').replace(/^-|-$/g, ''); if (!slug) slug = 'heading-' + i; if (used\[slug\]) { slug += '-' + i; } used\[slug\] = true; h.id = slug; var level = parseInt(h.tagName.charAt(1)); var cls = level > 2 ? ' class="toc-child"' : ''; html += '- [' + text + '](#'%20+%20slug%20+%20') '; } html += ' '; toc.innerHTML = html; // Scroll-spy: highlight current heading var links = toc.querySelectorAll('a'); var ids = \[\]; for (var j = 0; j < links.length; j++) ids.push(links\[j\].getAttribute('href').slice(1)); function updateActive() { var current = ''; for (var k = 0; k < ids.length; k++) { var el = document.getElementById(ids\[k\]); if (el && el.getBoundingClientRect().top <= 80) current = ids\[k\]; } for (var m = 0; m < links.length; m++) { links\[m\].parentNode.classList.toggle('active', links\[m\].getAttribute('href') === '#' + current); } } window.addEventListener('scroll', updateActive, { passive: true }); updateActive(); })(); --- ## Navigation - Parent: [RuBBS](https://wiki.teletypegames.org/engines/rubbs.md) - Previous: [Commodore 64](https://wiki.teletypegames.org/engines/rubbs/commodore.md) - Next: [Doors](https://wiki.teletypegames.org/engines/rubbs/doors.md)