Data and network

What every content board of ours used to write for itself, in rubbs since 0.33.0 and 1.0.0: one log format, work in the background, an SQLite file with migrations, bounded HTTP, a cache, rate limits, a paced Wikipedia client, control-free text, and the rule of where a failure is handled. Each block is a few lines to adopt; the full signatures are in the API reference.

The config fields

These fields of BBS.configure belong to this page (the rest are on Getting started and Telnet and the store):

BBS.configure do |c|
  c.log_tag    = 'feedia-bbs'
  c.user_agent = 'Feedia BBS (+https://teletypegames.org)'
end

Logging and background work

Logging

Every line the library and the board write to the log goes through BBS.log(tag, message), to stderr, so the lines of eleven boards in one log read the same. An exception as the message is written as its class and message.

BBS.log('catalog refresh', 'done in 3.2 s')   # feedia-bbs: catalog refresh: done in 3.2 s
BBS.log('catalog refresh', error)             # feedia-bbs: catalog refresh: SocketError: …

The library's own lines use the same format: session, door …, gateway …, store, BBS busy, BBS crowded. A BBS::DiscordWebhook's label: is the log tag too, so it no longer needs the board's slug.

Background work

Work out of the callers' way (an index loaded at boot, a catalog refreshed every hour) runs in a BBS::Background thread. An error is logged and the block is tried again after retry_after seconds (5), the pause doubling up to max_retry (300): a service that is down costs a few log lines, not a crashed board or a busy loop.

BBS.background('index') { wiki.load_index }                         # until it succeeds once
BBS.background('catalog refresh', every: 3600) { library.refresh }  # every hour

Both return the Thread; BBS::Background.new(name, …) { … }.run runs the same work in the calling thread (a test, a command-line tool).

Storage

BBS::SQLite

One SQLite file and its one connection, shared by the callers' threads behind a lock: WAL mode (a command-line tool of the board can use the file meanwhile), a busy timeout, foreign keys on. The schema is a list of migrations numbered by PRAGMA user_version: a file runs only the ones it has not had yet, each in a transaction. A migration is an SQL string or a lambda; never edit one that has run, append a new one.

DB = BBS::SQLite.new('data/board.sqlite3')        # creates data/ when needed
DB.migrate(
  'CREATE TABLE posts (id INTEGER PRIMARY KEY, title TEXT NOT NULL)',
  'ALTER TABLE posts ADD COLUMN body TEXT',
  ->(db) { db.insert('INSERT INTO posts (title) VALUES (?)', 'Welcome') }
)
DB.query('SELECT * FROM posts')                    # => [{ 'id' => 1, 'title' => 'Welcome', 'body' => nil }]
DB.get('SELECT * FROM posts WHERE id = ?', 1)      # the first row or nil
DB.insert('INSERT INTO posts (title) VALUES (?)', 'Hi')   # => the new rowid
DB.change('DELETE FROM posts WHERE id = ?', 2)     # => rows changed
DB.transaction { … }                               # a nested call joins the outer one

BBS::Store

The one-table, upsert-by-key store of a Flow's persist is built on BBS::SQLite since 0.33.0. A .csv path means the .sqlite3 file beside it, and the CSV file of an older board is imported on first use (broken rows set aside in a .rejected file); store.open! does it at boot. See Telnet and the store.

Network

BBS::HTTP

Bounded requests to the services a board shows. Every failure raises BBS::HTTP::Error, whose message names the address without its query string (so no token reaches a log); a body over max_bytes raises its subclass BBS::HTTP::TooLarge. Statuses other than 2xx are returned, not raised.

response = BBS::HTTP.get('https://example.org/feed.xml', headers: { 'Accept' => 'application/rss+xml' })
response.ok?       # => true for 2xx
response.status    # => 200
response.body      # => binary String

BBS::HTTP.post(url, body: URI.encode_www_form(query: q),
               headers: { 'Content-Type' => 'application/x-www-form-urlencoded' })

File.open(path, 'wb') { |file| BBS::HTTP.get(url, to: file, max_bytes: 64 * 1024 * 1024) }

BBS::Cache

TTL, least-recently-used eviction, a byte cap, one computation per key (callers missing the same key at once wait for it and share its value), and the expired value served when the computation raises.

ARTICLES = BBS::Cache.new(ttl: 600, max_entries: 500, max_bytes: 8 * 1024 * 1024)
ARTICLES.fetch([lang, title]) { repository.article(lang, title) }

BBS.background('catalog', every: 3600) { CATALOG.refresh(:all) { repository.load } }

refresh(key) { … } replaces the kept value only when the block succeeds, so a background refresh never empties the cache. delete(key), clear, size.

BBS::RateLimit

A sliding window per key, thread-safe; quiet keys are forgotten. caller_key(session) is the caller's IP address on the internet, and the line's session id behind a local proxy (bbs-web), whose address is shared by everyone.

TIPS = BBS::RateLimit.new(limit: 3, per: 3600)
key  = BBS::RateLimit.caller_key(session)
TIPS.allow?(key)   # => true, and counted; false when over the limit
TIPS.wait(key)     # => seconds until the next event is allowed

PER_CALLER = BBS::RateLimit.new(limit: 5, per: 600)
PER_BOARD  = BBS::RateLimit.new(limit: 100, per: 600)
BBS::RateLimit.allow_all?([[PER_CALLER, key], [PER_BOARD, :all]])   # counted in all or none

LOGINS = BBS::RateLimit.new(limit: 10, per: 600, store: DB, name: 'logins')
return locked_out unless LOGINS.room?(key)
LOGINS.record(key) unless password_ok?(name, password)

• allow_all? checks several limits at once (the caller's and the whole board's): the event is counted in every limit or, when one has no room, in none.
• room? checks and record counts apart, for an event that counts only when it went wrong (a bad password).
• With store: (a BBS::SQLite) the events are kept in its rate_limit_events table under name:, so a login lockout survives a restart.

BBS::Wikipedia

Wikipedia's Action API as a board asks it: through BBS::HTTP, with the board's User-Agent (Wikipedia may answer 403 without one), paced so that all callers together send at most rate requests a second; a request that would wait more than 3 s for its turn is not sent. The answer comes parsed (format=json, formatversion=2 added); a failure raises BBS::Wikipedia::Error, a BBS::HTTP::Error. Alexandria BBS and Commodex BBS use it.

WIKIPEDIA = BBS::Wikipedia.new(user_agent: 'Alexandria BBS (+https://teletypegames.org)', rate: 5)
WIKIPEDIA.get('en', action: 'query', list: 'search', srsearch: 'Commodore 64')
# => { "query" => { "search" => [...] } }

The gateway guard and translation

BBS::Gateway (see Doors) resolves the host once and, since 1.0.0, calls only a public address: a listing that names 127.0.0.1 or a private address cannot turn the board into a relay into its own network. A board that dials its own neighbours says so; a C64 can call a board made for ASCII and ANSI terminals with translate: true (since 0.34.0).

BBS::Gateway.new(host: 'host.docker.internal', port: 2327, name: 'Feedia', public_only: false)
BBS::Gateway.new(host: board.host, port: board.port, translate: !board.petscii?).call(session)

Text and addresses

• BBS::Text.clean(text): valid UTF-8 without C0 controls (tab and line feed kept), DEL, C1 and the bidirectional override characters. With keep_newlines: false each line end becomes a space, so the words of two lines stay apart.
• BBS::Text.line(text): one clean line, whitespace runs (line ends and tabs too) one space, no space at either end. For a title, a name, a summary.
• BBS::Text.printable?(char): the rule of every input field.
• BBS::Address.public?(ip) (on the public internet) and BBS::Address.local?(ip) (private, loopback, link-local).

The frame buffer stores no control character either, so text from a service cannot move or restyle the caller's screen.

Where an error is handled

A board's layers handle a failure of a service in one place:

• a repository (lib/domain/repository) talks to the outside world and lets its failures go up: BBS::HTTP::Error (and BBS::Wikipedia::Error) as it comes, or a domain error of the board for what is wrong with the answer itself. It does not turn them into nil or an empty list, and writes no text for the caller;
• a service decides what the caller sees: it rescues those errors, writes one BBS.log line, and returns what the UI shows (a cached answer, an empty result, a message of its own);
• the UI never rescues BBS::HTTP::Error; it shows what the service gave.

So an error message with a URL or a token never reaches a caller, and each failure is logged once.

class Library
  def search(query)
    repository.search(query)
  rescue BBS::HTTP::Error => e
    BBS.log('search', e)
    []
  end
end

See also

• rubbs — the overview
• Telnet and the store — the session, the limits and BBS::Store
• Doors — gateways, ZMODEM and XMODEM
• AI characters — the model client and its concurrency limit
• Changelog — when each block arrived