Error Tracking (Sentry)

Lumio's four Next.js apps - apps/web, apps/admin, apps/id, apps/shopify - each ship with a @sentry/nextjs integration. apps/stats has none. The Rust backend is on sentry-rust via lo-telemetry, covered in Rust backend at the end of this page.

When no DSN is configured, the integration is a no-op - apps boot normally, no events are sent. Local dev typically leaves the DSN blank; staging and production supply it via the container environment at run time (LUMIO_PUBLIC_SENTRY_DSN, see Environment variables).

Browser Sentry depends on consent and route

With a DSN, apps/web uses three browser modes. Stored Functional consent enables full on every page: error events, browser sessions and performance tracing (10% of transactions in production). Without that consent, /dashboard, /account, /hub and /popout and their child paths use errors: error events only, without browser sessions, transactions, trace headers, client reports or device storage. Other routes use off and send nothing. Path matching uses whole segments, so /hubspot is not a dashboard route. /overlay, /widget and /api are not included. apps/id still starts browser Sentry only after Functional consent; apps/admin and apps/shopify have no browser Sentry.

In apps/web, src/lib/sentry-consent.ts selects the mode from getStoredConsent() and src/lib/sentry-dashboard-paths.ts. <SentryConsentSync /> reconciles it on consent changes, cross-tab storage events and client navigation. Switching between errors and full reinitializes the client because Sentry integrations cannot be removed. The transport filters envelopes by the current mode and trace propagation targets are cleared when leaving full; beforeSend also rejects events outside dashboard paths in errors mode. Thus a navigation from the dashboard to a public page without consent sends no error event even before the next sync. Withdrawing consent outside the dashboard closes a full client.

The DSN reaches the browser at request time: the root layout calls getBrowserSentryConfig() from src/lib/config.ts and passes dsn and environment to <SentryConsentSync />. The release images are built without a DSN, and a NEXT_PUBLIC_* value is frozen at next build, so a DSN read from process.env.NEXT_PUBLIC_SENTRY_DSN in browser code would be empty in a deployed image. The component is mounted inside ConsentProvider in apps/web/src/app/(main)/layout.tsx; the ID app mounts its own consent sync in its root layout. The stored choice in lumio_cookie_consent is the source of truth, including on routes where the consent banner is exempt.

<SentryConsentSync /> is the browser entry point under Next 16 Turbopack: sentry.client.config.ts is loaded only in webpack builds. The extension frames follow the web host's mode; see Extension Supervisor and Runtime.

apps/admin has no consent banner, so its sentry.client.config.ts is empty. Server-side capture in src/instrumentation.ts is unaffected. apps/shopify also has an empty browser config and no Sentry origin in its CSP connect-src; its HMAC check, server render and API forward routes report errors server-side.

Files per app

Each of apps/{web,admin,id,shopify}/ has the same five files (paths relative to the app):

File	Loaded by	Purpose
next.config.ts	build	Wraps withSentryConfig (source-map upload, build-time injection)
sentry.client.config.ts	browser (webpack builds only)	In apps/web and apps/id a delegate to syncSentryConsent() for webpack builds; empty in apps/admin and apps/shopify. Turbopack never loads it - see the caution above
src/sentry.server.config.ts	Node runtime	SSR / route-handler / Server-Action capture
src/sentry.edge.config.ts	Edge runtime	Edge route handlers
src/instrumentation.ts	server	register() imports the right runtime config and binds the logger (apps/shopify has no @lumio/logger, so nothing to bind); also exports onRequestError: Sentry.captureRequestError in id and shopify, and in web and admin a wrapper that skips an expired session first (see Suppressed noise)

instrumentation.ts belongs in src/

All four apps use the src/ layout (src/app, src/proxy.ts). Next.js looks up instrumentation.ts - like proxy.ts and instrumentation-client.ts - only in the folder that holds app/, so here in src/. A copy in the app root is ignored without a warning: register() never runs, and server-side Sentry, the logger binding and onRequestError are all off whatever DSN is set. apps/web/__tests__/next-convention-files.test.ts fails if a convention file sits in the app root of web, admin, id, stats or shopify, and checks that web, admin, id and shopify register their runtime configs from src/instrumentation.ts. After next build the server output contains .next/server/instrumentation.js; if that file is missing, the hook was not found.

Configuration is duplicated across the four apps on purpose - the apps evolve independently. The browser side differs on purpose: apps/web selects a mode from route and consent, apps/id is consent-gated, and apps/admin and apps/shopify have no browser Sentry. None uses Session Replay. The server and edge configs are identical in web, admin and ID; apps/shopify has the same settings with a stricter scrub (see What gets stripped).

Environment variables

Variable	Where	Purpose
LUMIO_PUBLIC_SENTRY_DSN	Run time (browser + server)	Public DSN. Read per request by the root layout and handed to the browser SDK (web according to route and consent; ID after Functional consent); also the server fallback. This is the name to set on deployed containers. apps/web also adds its ingest origin to the CSP connect-src.
NEXT_PUBLIC_SENTRY_DSN	Build time	Local-dev fallback for LUMIO_PUBLIC_SENTRY_DSN. Inlined at next build, so setting it on a container of a prebuilt image has no effect.
SENTRY_DSN	Server only	Optional - falls back to LUMIO_PUBLIC_SENTRY_DSN, then NEXT_PUBLIC_SENTRY_DSN. Useful when you want the SSR / Edge runtimes on a different project than the browser. In apps/admin and apps/shopify it is the only DSN that matters (no browser SDK).
SENTRY_RELEASE	Build time	Release name the SDK reports and the source maps are uploaded for. The release image build sets it to <app>@<version>. Unset locally: the release is then the git HEAD SHA.
SENTRY_URL	Build time	Base URL of the Sentry instance for the upload (self-hosted). Leave blank locally.
SENTRY_ORG	Build time	Sentry org slug - required for source-map upload. Leave blank locally.
SENTRY_PROJECT	Build time	Sentry project slug - same. The app's Dockerfile sets it to the app name.
SENTRY_AUTH_TOKEN	Build time	Auth token for source-map upload. In the release image build it is the SENTRY_RELEASE_TOKEN secret, passed as a BuildKit secret (see Web apps).
LUMIO_ENV / NEXT_PUBLIC_LUMIO_ENV	Both	Tags every event with the environment (development / staging / production). Falls back to NODE_ENV. The browser SDK gets LUMIO_ENV from the root layout at request time, like the DSN.

Source-map upload only runs when SENTRY_AUTH_TOKEN + SENTRY_ORG + SENTRY_PROJECT are all set. In practice only the release image build sets them.

Web apps: release and source maps

release-images.yml builds web, admin, id and shopify with docker buildx build -f docker/<app>.Dockerfile. Each of these Dockerfiles declares ARG SENTRY_RELEASE, and the workflow passes:

• --build-arg SENTRY_RELEASE=<app>@<version>, always. <version> is the version of apps/<app>/package.json, the product version in lockstep. withSentryConfig writes that name into the client, server and edge bundles, so every event of the image carries release = "<app>@<version>", with or without a token.
• --build-arg SENTRY_URL=…, --build-arg SENTRY_ORG=… and --secret id=sentry_token,env=SENTRY_RELEASE_TOKEN, only when the SENTRY_RELEASE_TOKEN secret and the SENTRY_URL / SENTRY_ORG variables are all set. The Dockerfile mounts the secret only for RUN pnpm build (--mount=type=secret,id=sentry_token,env=SENTRY_AUTH_TOKEN) and sets SENTRY_PROJECT=<app> there. The token is never an ARG or ENV, so it is in no layer and not in the image history.

During next build the Sentry plugin then creates the release in project <app>, uploads the browser and server source maps with debug IDs. deleteSourcemapsAfterUpload then deletes the browser maps (.next/static/**/*.map), so the public bundle never serves them; it does that with or without an upload. The server maps stay in the image under .next/server, which is not served. next.config.ts sets release.finalize: false, and the plugin's own set-commits (automatic, fail-silent) finds no .git in the build context (.dockerignore). Commits and finalize come from the workflow's Sentry step (next section), the same as for the Rust apps.

Fail-open. Without the secret the mount sets SENTRY_AUTH_TOKEN to an empty string; next.config.ts reads it with ||, so the plugin skips the upload and the image is built as before, with a ::warning in the log. A Sentry outage or a failed upload during the build only logs [sentry] source-map upload skipped: … (errorHandler); it never fails next build. telemetry: false keeps the plugin from reporting its own usage to sentry.io, which it otherwise does in every build without SENTRY_URL. apps/stats has no Sentry and gets none of these arguments.

Checking it. The build log does not show the upload: the plugin prints nothing, and the Build + Push <app> group only shows Running next.config.js provided runAfterProductionCompile … / Completed runAfterProductionCompile. A failed upload prints [sentry] source-map upload skipped: … there. Check the result in Sentry instead: Releases → <app>@<version> lists the release, and Settings → Projects → <app> → Source Maps → Artifact Bundles lists two bundles per release (browser and server). The same through the API: GET /api/0/projects/<org>/<app>/files/artifact-bundles/?query=<app>@<version>. A new event from that image has resolved frames with the original file names (src/…).

What gets captured

• Unhandled exceptions in browser code (React error boundaries, Promise rejections, sync throws) - web on dashboard paths even without Functional consent, and web and ID elsewhere after Functional consent. Never in admin or Shopify.
• SSR / route-handler errors through the onRequestError export in src/instrumentation.ts (Sentry's recommended hook).
• Anything emitted via logger.error(...) or logger.warn(...) - the logger calls Sentry.captureException / Sentry.captureMessage when bound. See Logging for the binding details.

Dashboard flash diagnostics

The web and admin apps report dashboard layout failures as Sentry messages at warning level, tagged flicker with the mechanism. Server reports are sent whenever the server has a Sentry DSN (SENTRY_DSN, or its configured fallback). Browser reports reach Sentry only when its SDK is running: on the web app a dashboard path sends it even without Functional consent, while another route requires consent. Otherwise it goes to the browser console. The admin app has no browser Sentry, so its browser reports remain in the console. Neither browser path starts Sentry just to send a report.

Tag	Where	Event	Fields
flicker: A	Server, from the web or admin layout	The me load renders the outage screen, or a GraphQL error is skipped because usable data.me is present.	app, layout_event (load-error-rendered or partial-error-skipped), status, gql_code, error_kind, route
flicker: A	Browser, when the outage screen is shown	Confirms the outage screen was painted.	kind (unavailable or rate-limited), route
flicker: B	Server, from /api/auth/refresh or proxy.ts	The API refuses a session refresh (401/403 or a GraphQL error). An unreachable API or a 5xx/429 is not reported.	source and reason for the refresh route; source, request, and route for the proxy
flicker: B	Browser, before a refused refresh causes a reload	Confirms the client reload path ran.	app, route
flicker: C	Browser, from SeasonProvider in the web app	Providers remount within five seconds after the seasonal and motion attributes were cleared from <html>.	route

The route field keeps at most three pathname segments, replaces numeric IDs, UUIDs and long token-like segments with :id, and omits the query string. The report does not attach a user ID, token or error-message text. status is the HTTP status (or none), gql_code is the GraphQL extension code (or none), and error_kind is network, http, graphql or other.

What gets stripped

beforeSend in every runtime config deletes the authorization and cookie keys from event.request.headers before the payload leaves the process. In the Next.js apps, sendDefaultPii: false keeps Sentry's automatic IP / user-agent collection off, so cookie-issued JWTs and popout tokens never end up in Sentry; the extension frames get the same from dataCollection (see Extension Supervisor and Runtime).

apps/shopify strips more, in src/lib/sentry-scrub.ts, through both beforeSend and beforeSendTransaction. The embedded page is opened with Shopify's launch query, and the forward routes carry Lumio and order data:

• the authorization and cookie headers (any casing), request.cookies and request.data (no request body is ever attached);
• the query params id_token, hmac, session, code and state are replaced with [Filtered] in request.url and request.query_string, and in the string values of span data and breadcrumb data (where transactions repeat the URL as url.full / http.query).

apps/shopify/__tests__/sentry-scrub.test.ts covers it with a URL that carries all five params.

If you want to enrich events with explicit user info (e.g. account ID), set it manually after login:

import * as Sentry from "@sentry/nextjs";

Sentry.setUser({ id: account.id, ip_address: undefined });

ip_address: undefined opts out of Sentry's IP capture even if sendDefaultPii ever flips on later.

Session Replay

Not enabled in any app. Replay was removed from apps/web under ZAF-91 and from apps/id / apps/admin under ZAF-2681 - it is the highest-PII component, and the privacy policy states that Lumio does not use it. Do not re-add it, even consent-gated, without a disclosed processing entry in the policy first.

Performance traces

tracesSampleRate is 0.1 when NODE_ENV === "production" (10% of transactions) and 1.0 elsewhere for server capture and for browsers in full mode. The web errors mode removes browserTracingIntegration entirely: setting the sample rate to zero alone would still allow sentry-trace and baggage headers.

Suppressed noise

denyUrls on the browser side is the same list in all three apps:

denyUrls: [/extensions\//i, /^chrome:\/\//i, /^moz-extension:\/\//i, /^safari-web-extension:\/\//i]

Note the second entry matches the chrome:// internal scheme, not chrome-extension:// - the generic /extensions\//i pattern is what catches Chrome extension frames. Add entries when you see recurring noise from a specific extension or third-party script.

Expired sessions in SSR (web, admin)

An expired or revoked session is an expected state, not a defect. When the API answers a Server Component's GraphQL read with extensions.code = UNAUTHENTICATED (or a bare HTTP 401), serverGqlSSR in apps/{web,admin}/src/lib/server-gql.ts throws an error with the fixed digest UNAUTHENTICATED. The onRequestError wrapper in src/instrumentation.ts drops exactly that digest (isExpectedAuthError in src/lib/error-digest.ts); every other render error still goes to Sentry, FORBIDDEN included.

The segment's error.tsx turns that error into a login: web logs out and reloads, and the proxy then redirects to the ID app's login; admin opens /login?error=expired&callbackUrl=<path>. Every login redirect carries a callbackUrl back to the page the user was on (only the app origin plus the path, never the query): the proxies build it from the request URL, the (app) and (admin) layouts from the x-lumio-pathname request header the proxy sets (src/lib/login-redirect.ts). The (app) layout sends only a refused session (401/403) to the login; any other failure of its me read (API down, 5xx, rate limit) renders layout-load-error.tsx in place, which retries on its own. On /login?error=session_invalid the ID app asks the API (GET /users/me) whether it accepts the session again and, if so, returns to the callbackUrl (apps/id/src/lib/session-invalid-return.ts); while the API is down its offline banner reloads the page until that check passes. The return address also survives a failed sign-in with any error code: a refusal in the NextAuth signIn callback returns /login?error=AccessDenied&redirect_uri=… (loginErrorUrl), and when NextAuth fails before the callback the login page reads redirect_uri back from the auth-state cookie (redirectUriFromAuthState, both in apps/id/src/lib/login-error-return.ts). Every use re-checks it with isAllowedRedirect. The boundary has to read error.digest, not error.message: in a production build (staging included) Next.js replaces the message of a Server Component error with a generic one and keeps only the digest. A message match only ever fires in next dev. The same holds for the other fixed digests, FORBIDDEN and API_UNREACHABLE; classifyBoundaryError() in src/lib/error-digest.ts is the one place that maps them.

Adding a new captured event manually

Most code should just throw / call logger.error. For business-event tracking (e.g. "user upgraded plan"), use Sentry's addBreadcrumb or captureMessage directly:

import * as Sentry from "@sentry/nextjs";

Sentry.captureMessage("user upgraded plan", {
  level: "info",
  extra: { account_id: accountId, plan_slug: plan.slug },
});

These don't surface as errors but let you search them in Sentry's "Issues" tab when triaging.

Debugging the integration locally

Set the DSN in .env.local:

NEXT_PUBLIC_SENTRY_DSN=https://<key>@o<org>.ingest.sentry.io/<project>
LUMIO_ENV=development

LUMIO_PUBLIC_SENTRY_DSN works the same way. In apps/web, an error on /dashboard, /account, /hub or /popout can be reported without Functional consent; accept that category to exercise browser sessions and tracing or to report errors from other web routes. apps/id requires Functional consent for browser events. To reset, clear the lumio_cookie_consent localStorage key and reload.

Then trigger an error from any client component:

"use client";

import { useEffect } from "react";

export default function DebugPage() {
  useEffect(() => {
    throw new Error("sentry smoketest");
  }, []);
  return null;
}

Reload the page, open Sentry → Issues, the event should land within a few seconds with environment: development. Filter that out before shipping or the dashboard fills with smoketest noise.

Extension Supervisor and Runtime

apps/extension-supervisor and apps/extension-runtime run in browser iframes inside the dashboard and inside OBS overlays and widgets. Each has @sentry/browser with a build-time DSN (SENTRY_DSN_SUPERVISOR / SENTRY_DSN_RUNTIME). The host decides the mode because the frames cannot read its stored consent:

• Dashboard hosts such as the widget editor and extension designer send errorTracking: true to the Supervisor when Functional consent is stored (full: errors, sessions and traces), or errorTracking: "errors" on a dashboard path without it (error events only, no sessions, tracing, client reports or device storage). OBS /overlay and /widget hosts send no flag, so both frames stay off.
• The Supervisor starts Sentry according to that flag. It drops any errorTracking parameter already in extensionUrl, then passes errorTracking=1 for full or errorTracking=errors for errors on the Runtime frame URL. The Runtime accepts only those two values. The string "errors" stays off on older Supervisors that only accept true.
• The decision is read when the frame starts; an existing frame keeps that mode until reloaded. Neither frame uses Session Replay.

Both frames are on @sentry/browser 11, which has no sendDefaultPii. Its replacement dataCollection collects user info, cookies and request/response bodies when left unset, so both Sentry.init calls pin it to the restrictive baseline: no user info, cookies, bodies, GenAI/MCP inputs and outputs, database query data, queue data or GraphQL documents, and IP-revealing headers and query parameters (forwarded, -ip, remote-, via, -user) denied. Do not drop or loosen that block. The Next.js apps are still on @sentry/nextjs 10 and keep sendDefaultPii: false.

Rust backend (lo-telemetry)

Every Rust service calls lo_telemetry::init_sentry(&config.sentry, "<service>") before init_tracing. Sentry stays off unless [sentry] enabled = true and dsn is set; config keys and ENV names are in Installation → Error tracking (Rust services). The service names are the same as the diagnostics file's: api, twitch-bot, kick-bot, youtube-bot, discord-bot, crawler-twitch, bot-module-worker, automation-worker and innertube-proxy. The proxy has no TOML config and builds its [sentry] / [logging] values from env vars, see InnerTube Proxy → Logging and error tracking.

Events and breadcrumbs

init_tracing installs three Sentry tracing layers from sentry::integrations::tracing, each behind the stdout filter (RUST_LOG / [logging] level) plus a filter of its own: BreadcrumbLayer (breadcrumbs), ErrorLayer (events) and SpanLayer (spans from info up become transactions, sampled by [sentry] traces_sample_rate). There is no LogLayer, so no Sentry structured logs are sent. For every line that passes the stdout filter, [sentry] event_level decides:

Level	event_level = "warn" (default)	event_level = "error"
error	event + breadcrumb	event + breadcrumb
warn	event + breadcrumb	breadcrumb
info	breadcrumb	breadcrumb
debug / trace	nothing	nothing

The breadcrumb layer runs before the error layer, so an event's own line is already its last breadcrumb. init_tracing reads event_level once, from the value init_sentry stored; that is why init_sentry must run first. With Sentry off the layers keep the library default (error → event, warn/info → breadcrumb).

Panics are captured by the default sentry panic integration.

Lines from the twitch_irc library (twitch_irc and its submodules) are breadcrumbs only, whatever their level (BREADCRUMB_ONLY_TARGETS in crates/lo-telemetry/src/sentry_events.rs). The library logs every dropped pool connection, missing PONG and server RECONNECT at error and rebuilds the connection itself. A lasting outage is reported by the silence watchdog in lo-twitch-irc (crates/lo-twitch-irc/src/silence.rs): when a client has joined channels and receives nothing for 5 minutes, it logs error "Twitch IRC receives nothing while channels are joined, connections keep failing" once per outage, under its own lo_twitch_irc target. twitch-irc pings every open connection every 30 s and forwards the PONG, so a healthy pool is never silent that long.

A malformed client request is not an event either (CLIENT_REQUEST_NOISE in the same file). actix-http answers an unparseable HTTP/1 request with 400 (431 for an oversized head) and then logs stream error: Request parse error: … at error on actix_http::h1::dispatcher. On a service with no reverse proxy in front, such as the innertube-proxy, every internet scanner would open an event. before_send drops exactly these events; the line stays on stdout, in the diagnostics file and as a breadcrumb. The match is on target and message, not on the target alone: the same call site reports the server's own failures (I/O error: …, Internal error, Body error: …), and those, plus a parse error that wraps an I/O error, still become events.

Reconnect loops

An upstream WebSocket that drops and reconnects is expected behaviour, not an error. A worker logs a single drop or failed reconnect at info, and error only when the reconnect keeps failing: at RECONNECT_ERROR_THRESHOLD (5) consecutive failures and every further 5, decided by lo_common::reconnect_failure_is_persistent. The escalation has its own fixed message, so it groups into one Sentry issue:

Worker	Drop / single failure (info)	Lasting failure (error)
StreamElements (api::workers::streamelements)	StreamElements connection lost, reconnecting	StreamElements reconnect keeps failing
Twitch EventSub (lo_twitch_api::eventsub, api::workers::twitch_eventsub)	EventSub WebSocket read error, reconnecting, EventSub keepalive timeout exceeded, reconnecting, Failed to connect to EventSub, retrying	EventSub reconnect keeps failing
Kick bot Pusher chat (kick_bot::bot)	Failed to connect to Kick WebSocket — retrying, Kick WebSocket error — reconnecting, Kick WebSocket closed by server — reconnecting	Kick WebSocket reconnect keeps failing
Kick chat listener (api::workers::kick_chat_moderation)	Kick chat listener connect failed, Kick chat listener connect timed out, Kick chat socket error, Kick chat socket unresponsive — reconnecting	Kick chat listener reconnect keeps failing
7TV EventAPI (api::workers::seventv_events)	7TV EventAPI connect failed, 7TV EventAPI connect timed out, 7TV EventAPI: no hello — reconnecting, 7TV EventAPI silent for 3 heartbeats — reconnecting	7TV EventAPI reconnect keeps failing
Twitch IRC tags (api::workers::twitch_irc_tags)	Twitch IRC stream ended; reconnecting	Twitch IRC reconnect keeps failing

The StreamElements, Kick bot and Twitch IRC tags counters reset when a connection stayed up for 60 s (they count failed connects and connections that ended sooner), the EventSub counter when a new session took its subscriptions. The Kick chat listener and 7TV count per shard: every failed connect and every connection that ended within 60 s adds one, a connection that lived 60 s sets the count back to 0. Every line carries consecutive_failures. New reconnect loops follow the same pattern.

A Pusher pusher:error frame on the Kick chat listener gets its level from the code (classify_protocol_error in apps/api/src/services/kick_chat_moderation.rs): 4200–4299 (reconnect now) and 4100–4199 (reconnect after a backoff) log info as Kick chat socket protocol error, and the close that follows counts towards consecutive_failures; 4000–4099 (app missing, disabled, over quota, unauthorized, so a reconnect cannot heal it) logs error as Kick chat socket refused by Pusher; any other code, or none, stays warn.

GraphQL request log levels

POST /v1/gql logs one line per request. A response with errors gets its level from the errors' extensions.code, not from their message (classify_error_log_level in apps/api/src/graphql/mod.rs):

Errors in the response	Level	Message
Every error is the uncoded Authentication required literal (unauthenticated popout pages)	debug	GraphQL auth-required (unauthenticated request)
Every error has an expected client code: NOT_FOUND, UNAUTHORIZED, UNAUTHENTICATED, VALIDATION_ERROR, BAD_REQUEST, PLAN_LIMIT_REACHED, FEATURE_DISABLED (the Authentication required literal also counts)	info	GraphQL request with expected client errors
Anything else: a server fault, FORBIDDEN, an error without a code, or a mix of expected and unexpected	warn	GraphQL request with errors

So a missing history report or a dead refresh token is a breadcrumb, not a Sentry issue. FORBIDDEN stays warn on purpose: a refused permission check is a security signal. A resolver error only gets an expected level if it carries a code, so new resolvers map their failures through ApiError + crate::graphql::errors::to_graphql_error. Resolvers that answer with a bare message such as Overlay not found or No active account use crate::graphql::errors::{not_found, unauthorized, bad_request} instead: they add the code of the REST twin's ApiError variant and leave the message unchanged. A bare async_graphql::Error::new(…) has no code and is logged at warn, so keep it for real faults (e.g. a row missing right after it was written). graphql_requests_total{outcome="error"} counts every response with errors, whatever its log level.

What every event carries

• release = "<service>@<version>", with the workspace product version, so a regression can be tied to a release. server_name is the service name. The release workflow fills that release with commits and debug files, see Releases, commits and debug files.
• Tags service and target (the log target, e.g. api::workers::twitch_eventsub), plus these fields when the log line has them: platform, sub_type, notice_type, event_type, error_kind. IDs (account_id, channel_id, user_id, …) stay in the Rust Tracing Fields context and are never tags, because tags must stay low-cardinality.
• Fingerprint [target, message template], where the template is the message with every digit run replaced by <n>. "Pool connection 3 has failed" and "Pool connection 7 has failed" are one issue. The stack trace stays on the event.

What gets stripped

before_send and before_breadcrumb always remove viewer names and message text from the event's fields and breadcrumb data. They use the same key lists as the diagnostics file's redact_user_fields (crates/lo-telemetry/src/redact.rs), including JSON documents logged as a field. This cannot be switched off. IDs, event types and numbers stay. The log message itself is not rewritten, so never interpolate a viewer name into the message string: log it as a field.

Volume guard

One bad token can log the same error dozens of times per start, e.g. one Failed to create EventSub subscription per subscription type. The guard sends at most max_events_per_fingerprint (default 10) events per fingerprint per fingerprint_window_secs (default 60). The rest is dropped and counted. The first event of the next window carries the count as suppressed_count in the Volume Guard context. The guard lives in the process (no Redis), remembers up to 1024 fingerprints (least recently seen is evicted) and fails open. max_events_per_fingerprint = 0 switches it off.

Releases, commits and debug files

release-images.yml (Build & Push job) has a step Sentry release, commits and debug files. It runs after every image of the run is pushed, signed and promoted, for each Rust app the run built and for each web app built with Sentry build args (see Web apps). Web apps go first, api last:

1. Release. It creates the release <app>@<version> in the Sentry project <app>, or reuses it if the SDK already created it with the first event (or, for a web app, the image build created it). For a Rust app <version> is [workspace.package] version from the root Cargo.toml: the CARGO_PKG_VERSION the binary reports. For a web app it is the version of apps/<app>/package.json, the name its image build passed as SENTRY_RELEASE. It is never the release tag or the workflow_dispatch input. The step also finalizes the release.
2. Commits. sentry-cli releases set-commits <release> --commit zaflun/lumio@<sha> uses the GitHub integration of the Sentry org. Sentry computes the range from the previous release; a release without a predecessor gets the last 20 commits. A shipped commit whose message contains Fixes <SHORT-ID> resolves that Sentry issue with the release. Sentry fetches the commits in the background, so the step waits (up to 120 s) until the release reports them before it moves to the next app. Back-to-back fetches for all apps of one run left api@2026.10.317 with 0 commits. Every app's release gets the same commits, and Sentry resolves a Fixes issue in the last release that received the commit. That is why api, which owns the issues, is always processed last.
3. Debug file (Rust apps only; a web app's source maps went up in its image build). It splits the debug info off the app's unstripped binary (objcopy --only-keep-debug --compress-debug-sections=zlib) and uploads that debug companion with sentry-cli debug-files upload. The companion has the binary's GNU build-id and therefore the same debug id. The unstripped api binary is about 1 GB, the companion only a fraction of it. Symbolication then shows file and line for Rust frames.

No deploy is reported. The workflow pushes images and moves the beta/latest channel tags; it does not deploy. Staging pulls :beta on the host (dev-stack/staging/verified-pull.sh). The SDK's environment already tells staging and production events apart.

Apps without a Sentry project are skipped with a warning. Every Rust app in RUST_APPS and each of web, admin, id and shopify has a project today. A new app needs a project named exactly like the app (the service name).

Upload size limit. sentry-cli uploads debug files in 8 MiB chunks, several per request; the server's GET /api/0/organizations/<org>/chunk-upload/ reports chunkSize and maxRequestSize (32 MiB). Every proxy in front of Sentry must accept request bodies of that size. A proxy with nginx's default client_max_body_size 1m answers 413 Request Entity Too Large, and the step logs debug-files upload failed.

Where the debug info comes from. --config=release (.bazelrc) adds -Cdebuginfo=line-tables-only -Cstrip=none through @rules_rust//rust/settings:extra_rustc_flag. rules_rust appends these after its opt defaults (-Cdebuginfo=0 -Cstrip=debuginfo), so they win. Line tables are the smallest level that still maps an address to file and line, inlined frames included. The rust_binary therefore keeps its line tables. Each app's image_layer packs :<app>_stripped instead: the strip_debug rule (misc/strip_debug.bzl) runs the C++ toolchain's objcopy --strip-debug. That removes the DWARF sections and keeps the symbol table, the GNU build-id note and .eh_frame. The image binary has the same Sentry debug id as the uploaded file, and in-process backtraces keep their function names. The debug info never enters the image. CI (fastbuild) is not affected: the flags only live in --config=release, and the _stripped targets are manual.

Token and settings. The step reads three repository settings:

Setting	Kind	Value
SENTRY_RELEASE_TOKEN	secret	A Sentry Organization Auth Token (organization settings, Organization Tokens). Its only scope is org:ci: create releases, set commits, upload debug files. It cannot read issues or events.
SENTRY_URL	variable	the base URL of the Sentry instance
SENTRY_ORG	variable	the org slug

The token reaches sentry-cli only as the SENTRY_AUTH_TOKEN environment variable of that step, never in argv or the log. The Build, push, sign & attest step holds it as SENTRY_RELEASE_TOKEN (deliberately not SENTRY_AUTH_TOKEN, so no other tool there picks it up) and hands it to the web image builds only as a BuildKit secret. The triage token the agents use (event:read and friends) has no release scope and is not used here.

Fail-open. The step has continue-on-error: true and always exits 0. A missing secret or variable, an unreachable Sentry, a missing project or a failed upload each log a ::warning and the release still publishes its images. sentry-cli is a pinned GitHub release binary checked against its SHA-256 (SENTRY_CLI_VERSION / SENTRY_CLI_SHA256 in the step). When you bump it, take the digest from the release asset (gh api repos/getsentry/sentry-cli/releases/tags/<version>, field digest).

Checking a release. The step log prints <release>: <n> commits per app. In Sentry, Releases → <app>@<version> shows the commit count. On a new event from that release, the Rust frames carry filename and lineno, and Settings → Projects → <app> → Debug Files lists the uploaded binary.

Sentry is telemetry: no audit event, no permission.