Global Blocklist

Lumio refuses text through one matcher, the pure crate crates/lo-blocklist. Three lists use it:

List	Stored in	Options	Where it is checked
Global blocklist (Admin → Blocklist)	blocklist_entries (main Postgres, no account_id)	Options::STRICT	services::blocklist::GlobalBlocklist in apps/api
An account's blocklist for Shopify buyer names (Shopify → Buyer-name blocklist)	shopify_name_blocklist (main Postgres, one row per entry, account_id)	Options::STRICT	services::shopify_blocklist::CompiledNameBlocklist, compiled per order, no cache
An account's system-word-filter install	the install's config	Options::VERBATIM	lo_shopify::WordFilter (Shopify typed names)

The global list is never public: it is readable only through the admin API (/v1/admin/blocklist…, admin* GraphQL fields) and has deliberately no WebSocket channel. A consumer gets a yes/no answer, never the list.

The matcher (lo-blocklist)

No database, no network, no I/O. A caller loads its rules, compiles them once and asks per text.

use lo_blocklist::{Blocklist, Options, Rule, RuleKind};

let rules = vec![Rule::word("ass"), Rule::regex("b[a4]d"), Rule::new(RuleKind::Wildcard, "sp*m")];
let (list, rejected) = Blocklist::compile(&rules, Options::STRICT);
// `rejected`: Vec<RejectedRule { id: RuleId, reason: RejectReason }>, ordered by id.
assert!(list.matches("Ｂ4D"));
assert_eq!(list.first_hit("an ass"), Some(lo_blocklist::RuleId(0)));

• Rule { kind, pattern }, RuleKind = Substring (anywhere), Word (whole word only: ass does not hit Cassandra; a pattern edge that is punctuation needs no boundary), Wildcard (* = any run, the rest literal) and Regex (Rust regex, case-insensitive). RuleKind::as_str() / FromStr use the stored names substring, word, wildcard, regex.
• Blocklist::compile(&[Rule], Options) -> (Blocklist, Vec<RejectedRule>). Literal rules go into one Aho-Corasick automaton, wildcard and regex rules into one RegexSet. A rule that fails validation, or does not fit under the set cap with the rules before it (RejectReason::SetTooLarge), is left out and reported; the rest form the list. RuleId is the rule's index in the slice. Blocklist::default() matches nothing.
• matches(&str) -> bool for the hot path; first_hit(&str) -> Option<RuleId> returns the lowest matching id (the admin test field).
• Normalisation (Options::normalize, lo_blocklist::normalize): NFKC, lowercase, invisible characters removed, whitespace collapsed and trimmed. With it on, the text and substring / word / wildcard patterns are normalised; a regex pattern is never rewritten (that would turn \S into \s) but runs on the normalised text. No leetspeak folding.

Options

	Options::STRICT	Options::VERBATIM
Used by	the global blocklist and every new list	the system-word-filter module, so it behaves as on the streamer's chat
normalize	true	false (literals lowercased only)
max_input_chars	Some(MAX_INPUT_CHARS) = 256	None
rule_size_limit	STRICT_RULE_SIZE_LIMIT_BYTES = 64 KiB	RULE_SIZE_LIMIT_BYTES = 1 MiB

Limits

Constant	Value	Meaning
MAX_PATTERN_LEN	512	UTF-16 code units per pattern (JavaScript string.length); a wildcard's translated regex is held to it too
MAX_INPUT_CHARS	256	characters of text STRICT matches; cut again after normalisation
STRICT_RULE_SIZE_LIMIT_BYTES	64 KiB	compiled size per wildcard / regex rule under STRICT: room for one Unicode class like \w (about 50 KiB), not for \w{20} (about 1 MiB)
RULE_SIZE_LIMIT_BYTES	1 MiB	the same under VERBATIM
SET_SIZE_LIMIT_BYTES	32 MiB	the whole RegexSet, and the Aho-Corasick automaton
SET_DFA_SIZE_LIMIT_BYTES	4 MiB	lazy-DFA cache of the set
STRICT_SET_CAPACITY	512	SET_SIZE_LIMIT_BYTES / STRICT_RULE_SIZE_LIMIT_BYTES: a strict list with at most this many pattern rules can never reach the set cap

The Rust regex crate matches in linear time, so with these caps a hostile pattern or text cannot stall a caller.

Validation

• validate_rule(&Rule, Options) -> Result<(), RejectReason> is the one check every rule passes, on save as on load; compile runs it for each rule. Reasons: Empty, TooLong, Invalid(message) (does not compile or exceeds rule_size_limit).
• check_breadth(&Rule, Options) is the extra save-time check: valid; for a wildcard / regex rule, every match contains at least MIN_REQUIRED_LITERAL_CHARS (2) characters of literal text; and it hits neither empty text nor any name in BREADTH_PROBES. The literal check walks the case-folded regex_syntax HIR for the longest run of characters every match must contain (inside the match, not only as prefix or suffix, so *bad* passes): a character the pattern fixes up to case weighs one, one of at most MAX_SMALL_CLASS_CHARS (4) alternatives weighs half, anything optional or larger breaks the run, and an alternation counts its weakest branch. It closes the class the probe list cannot (a probe list is a denylist: \p{Thai}, [ğşı], ^.{11,15}$ miss every probe); the probes stay as the second line of defence and the only one for substring / word rules. The probe set covers short names and initials, accented Latin, Cyrillic, Greek, Han, Arabic, Hebrew, Devanagari, Hangul and kana names, long and hyphenated names up to 25 characters, apostrophes, underscores and digits. Otherwise RejectReason::TooBroad ("Pattern matches too broadly"). It is not part of validate_rule, so an existing word-filter list keeps its broad entries.
• dedupe_key(&Rule) -> String is the key under which a store treats two rules as one entry: normalize(pattern) for substring / word / wildcard, the pattern verbatim for regex (case is meaning there). Store it next to the pattern and make (kind, key) unique; never use SQL lower().
• RejectReason's Display is the user-facing message (Pattern is empty, Pattern is too long (max 512 characters), Invalid pattern: …, Pattern matches too broadly, Blocklist is too large to compile).

The global list in apps/api

apps/api/src/services/blocklist.rs owns both halves: the admin writes and reads that GraphQL (graphql/blocklist.rs) and REST (routes/blocklist.rs) share, and the in-process matcher.

Store and writes

blocklist_entries (20261104180000_blocklist_entries): id (UUID v7, so ORDER BY id is creation order), kind, pattern (1–512), pattern_key (dedupe_key, 1–1024, UNIQUE (kind, pattern_key) as uq_blocklist_entries_kind_pattern_key), category, locale, enabled, note (≤ 500), created_by / updated_by (ON DELETE SET NULL, indexed), created_at, updated_at. Hard delete.

Every write:

1. validates with check_rule (check_breadth + dedupe_key under Options::STRICT); an import validates all lines on the blocking pool before it opens the transaction;
2. takes pg_advisory_xact_lock(hashtext('blocklist_entries')) in its transaction and checks the [blocklist] caps (max_entries, max_pattern_entries, counted over every row, enabled or not);
3. writes, then — for a create, an import that added rows, or an update that switches an entry on or changes an enabled entry's kind or pattern — compiles the whole enabled list as the transaction sees it (ensure_compiles, on spawn_blocking) and refuses with 409 BLOCKLIST_SET_TOO_LARGE if any rule comes back SetTooLarge. Nothing is committed then. Switching off and deleting skip the check;
4. commits, emits one system:blocklist_* audit event and publishes an empty message on lumio:blocklist:changed (best effort: a failed publish is logged, the 60 s backstop catches up).

With max_pattern_entries at or below STRICT_SET_CAPACITY (512; default 500) step 3 cannot fire for rules that passed step 1, but it stays as the guarantee that a saved, enabled rule always loads.

The admin test field

test (adminBlocklistTest, POST /v1/admin/blocklist/test) answers from the database's list, not from the instance's GlobalBlocklist, so it reflects a save made a moment ago on any instance. It does not compile per call: a compile at the caps takes seconds (512 rules of one Unicode class each), and an admin with only blocklist:read must not be able to keep blocking-pool threads busy. A process-wide TestList keeps the last compiled list keyed by db::blocklist::enabled_revision — count(*) plus md5 over every enabled entry's id@updated_at — and compiles again only when that changes. The revision is read before the rows, so a concurrent write costs one more compile on the next call, never a stale answer. A tokio::sync::Mutex makes it single-flight: one compile per process at a time, the callers behind it reuse the result.

GlobalBlocklist and the reloader

AppState.global_blocklist: Arc<GlobalBlocklist> holds RwLock<Option<…>> around an Arc<CompiledBlocklist> (the compiled list plus the entry ids for first_hit).

• blocks(text) is what a consumer calls: true when the list hits, and also true while no list is loaded yet (cold start, first load failed) — fail closed. A loaded empty list passes everything. snapshot() returns the current Arc without holding the lock.
• run_reloader (spawned once in main.rs on every instance) is one serial task: subscribe to lumio:blocklist:changed → reload; then reload on each message (a burst is coalesced) and on a 60 s tick (BACKSTOP_INTERVAL); on a lost subscription it reconnects and reloads again. If subscribing fails it reloads anyway and retries after 5 s. No reader ever reloads.
• A reload reads SELECT id, kind, pattern FROM blocklist_entries WHERE enabled ORDER BY id and compiles on spawn_blocking. A DB or compile failure keeps the previous list. Rules the matcher leaves out are skipped with a warn; the rest still load.

Metrics

Registered in the API's process registry (see Metrics):

Metric	Meaning
blocklist_reload_failures_total{outcome}	db_error, compile_error, rule_rejected (entries left out per reload). Alert on any rule_rejected increase: rules saved under different limits no longer load
blocklist_age_seconds	seconds since the list in use was loaded, -1 before the first load; well above 60 means reloads fail
blocklist_rules_loaded	rules in the list in use

Consumer checklist

The worked example is the Shopify buyer name, services::shopify::order_event_raw (called from the orders/create / orders/paid webhook in routes/webhooks.rs):

let label = match lo_shopify::order_label_candidate(order, settings) {
    None => None,
    Some(name) if global.blocks(&name) => None, // global list first, every form, no query
    Some(name) if account_lists_block(db, account_id, &name).await => None, // account list + word filter, one query
    Some(name) => Some(name),
};

1. Check the exact string you will show. Shopify checks the finished label (after sanitize_stream_name), never a raw field, and the label never falls back to another form that skipped the check.
2. Take &GlobalBlocklist (or AppState.global_blocklist) and call blocks(). Never load blocklist_entries on the request path and never compile per request.
3. Treat true as "do not show" and drop silently, with no hint to the person who typed the text. blocks() already fails closed before the first load.
4. Run the global list before account-level lists, so a hit costs no query.
5. Never return the list or the matching entry to a non-admin caller; only the yes/no outcome leaves the service.
6. Another process (for example a future V8 host function in extension-runtime or bot-module-worker) needs its own GlobalBlocklist and its own run_reloader subscription; the loader and reloader are reusable functions for that.
7. A new per-account list stores dedupe_key next to its pattern, validates with check_breadth on save, compiles with Options::STRICT, and caps its pattern rules at STRICT_SET_CAPACITY or checks the compiled set on save. The Shopify buyer-name list (services::shopify_blocklist) is the worked example: UNIQUE (account_id, kind, pattern_key), the cap check and insert under a per-account pg_advisory_xact_lock, [shopify] name_blocklist_max_pattern_entries refused at startup above STRICT_SET_CAPACITY, and a stored rule that no longer compiles skipped and logged by entry id and a fixed reason class (empty, too_long, invalid, too_broad, set_too_large), never by its pattern.
8. A test field over an account list checks only that list. Running the global list there would let any account read the secret list one guess at a time.