YouTube Live Chat Streaming

Overview

Lumio receives YouTube live chat messages via InnerTube — YouTube's own internal polling endpoint (youtubei/v1/live_chat/get_live_chat). This transport costs 0 Data API quota and delivers the full range of chat event types. gRPC streamList and REST polling are available as optional fallbacks but are disabled by default.

Architecture

Chat Transport

Mode	Cost	Default
InnerTube (primary)	0 quota	Always active
gRPC streamList (fallback)	~0 quota	Disabled (grpc_fallback_enabled = false)
REST Data API v3 (fallback)	5 units per poll	Disabled (rest_fallback_enabled = false)

The worker starts every stream task in InnerTube mode. gRPC is lazy-connected only if enabled and InnerTube fails. REST polling is a last resort.

Fallback cascade: InnerTube → gRPC (if enabled) → REST (if enabled). Each transition triggers after 3 consecutive failures within 60 seconds.

How InnerTube Chat Works

InnerTube uses YouTube's own live chat polling contract:

1. Bootstrap — The worker resolves the InnerTube API key and client version through the credential resolver, then fetches the live chat embed page (youtube.com/live_chat?v=\{video_id\}&is_popout=1) to extract the initial continuation token. Continuations are cached in Redis so restarts don't re-scrape the embed page.
2. Poll loop — get_live_chat is called with the continuation token. The response carries the next continuation token and a server-recommended polling interval. The worker honors that interval (minimum 1 s, default 3 s).
3. All event types flow through InnerTube — text messages, SuperChats, SuperStickers, memberships, gift memberships, message deletions, user bans, poll events, and members-only mode changes are all returned in the same response.

InnerTube is unauthenticated for chat reception — no OAuth token required.

Broadcast Discovery

Broadcast discovery runs independently of chat reception:

1. Primary: InnerTube browse endpoint — Queries youtubei.googleapis.com/youtubei/v1/browse to list live and upcoming broadcasts via the channel's Streams tab. 0 quota. Returns broadcast IDs, titles, viewer counts, and scheduled start times (but not liveChatId).
2. liveChatId resolution — For newly discovered broadcasts, liveChatId is resolved once via Data API v3 liveBroadcasts.list (5 quota units). The result is cached in the channel_status table and survives worker restarts. Unresolvable broadcasts are skipped until they disappear and reappear.
3. Fallback: Data API v3 — If InnerTube browse fails entirely, the worker falls back to Data API discovery. Gated by rest_fallback_enabled.

InnerTube Credential Rotation

InnerTube requests need YouTube's current WEB-client API key and client version. Lumio resolves both through the same centralized resolver:

1. Override — api_key_override / client_version_override, only when an operator intentionally pins a value.
2. Redis — cached values from the shared refresher worker.
3. Scrape — fresh youtube.com scrape; successful results are written back to Redis.
4. Compiled constant — cold-boot last resort from lo_youtube_api::innertube.

The override layer is a deliberate pin. When either override is non-empty, Redis and scrape-based auto-rotation are bypassed for that value until the override is removed.

Redis keys:

Key	Shape	TTL
lumio:yt:innertube_key	JSON string with the last scraped InnerTube API key	24 h
lumio:yt:innertube_version	JSON string with the last scraped InnerTube client version	24 h
lumio:yt:innertube_refresh:leader	SET NX leader lock for the shared refresher replica	~90% of refresh_interval_secs
lumio:yt:continuation:\{account_id\}:\{live_chat_id\}	next-poll continuation token for one active chat	1 h

The shared innertube_credentials worker runs once on API startup and then every refresh_interval_secs (default 21 600 s / 6 h). One API replica wins the Redis leader lock per interval, scrapes youtube.com, refreshes both credential keys, and logs value changes. The 24 h credential TTL is intentionally longer than the refresh interval, so normal account workers read warm Redis entries instead of scraping independently.

If Redis or scraping fails, the system fails open: account workers still try their own scrape and then fall back to the compiled constant. If an InnerTube poll returns a credential-rotation error (400 or 403, excluding quotaExceeded), the stream task bypasses Redis, force-scrapes both values, writes fresh hits back to Redis, and retries the poll exactly once. If the fresh scrape returns the same values that just failed, the retry is skipped and the task falls through to the normal InnerTube failure counter; there is no hot retry loop against youtube.com.

Discovery intervals:

State	Interval
Idle (no active broadcasts)	60 s
Active (broadcasts running)	60 s

InnerTube costs 0 quota, so frequent polling is safe. Each cycle refreshes viewer counts, likes, total views, and detects new or ended broadcasts.

Badge + Emote Enrichment

Member badge images and channel-custom emotes are extracted inline from InnerTube chat responses — no separate observer is needed:

• Member badges — Each message's authorBadges array is parsed for custom-thumbnail badges. The badge image URL and tier label are stored in the MembershipBadgeCache (LRU + Redis) and attached to rendered messages.
• Channel emotes — Custom emojis with a channelId/emoteId pattern are extracted from emoji renderers and merged into the channel emote registry via lo_chat::merge_channel_emotes.

Broadcast Statistics

The worker fetches like counts and total view counts via InnerTube endpoints at 0 quota cost, once per 60-second discovery cycle per live broadcast:

• updated_metadata endpoint — Current like count per broadcast
• player endpoint — Lifetime view count per broadcast

These are stored in channel_status (like_count, total_views) and displayed on the dashboard and popout viewer badge.

Channel-feed fallback

Both InnerTube endpoints are undocumented and fail silently: any change to their response shape yields None, which would write NULL and make the badge disappear. A second, quota-free source covers that gap — the public channel Atom feed at https://www.youtube.com/feeds/videos.xml?channel_id=\{YT_CHANNEL_ID\}, whose entries carry a media:community block:

<media:community>
  <media:starRating count="41" average="5.00" min="1" max="5"/>
  <media:statistics views="603"/>
</media:community>

starRating count is the like count; statistics views is the cumulative view count (not concurrent viewers — those keep coming from broadcast discovery). The sibling average has been a constant 5.00 since YouTube removed public dislikes and is ignored.

Rules the worker applies (ChannelFeedCache in apps/api/src/workers/youtube.rs):

• InnerTube stays primary. The feed is consulted only for a value InnerTube did not deliver; when both numbers are present no request is made at all.
• One GET per channel, at most every 15 minutes. YouTube serves the feed with cache-control: public, max-age=900, so a faster cadence would only re-read the same edge-cached bytes. It can never replace the 60-second InnerTube poll.
• Matched strictly on videoId. Uploads, Shorts and live broadcasts share the same 15-entry feed, so "newest entry" is not necessarily the stream.
• Also covers non-live broadcasts. Upcoming broadcasts and premieres never hit InnerTube, so the feed is their only statistics source.
• Early warning. A fill happening at all means InnerTube returned nothing where the public feed has a number — the earliest signal that an InnerTube path changed. The worker logs it per fill.

The feed needs no authentication and no PubSubHubbub subscription; it is a plain scheduled GET (see Webhooks). A live smoke test guards the assumptions:

cargo test -p lo-youtube-api --test channel_feed_smoke -- --ignored --nocapture

Multi-Stream Support

YouTube allows multiple simultaneous live streams per channel. The worker manages them all:

1. Discovery finds all active and upcoming broadcasts.
2. A separate stream task is spawned per broadcast (each with its own InnerTube continuation state).
3. Task lifecycle is managed per-broadcast: new → spawn, ended → cancel.
4. Active streams are stored in Redis at lumio:youtube:active_streams:\{account_id\} (TTL 120 s) for the frontend.

Messages from all streams appear together in the Multichat.

Configuration

All settings live in the [youtube] section of config/*.toml or as environment variables.

Setting	ENV override	Default	Description
grpc_fallback_enabled	LUMIO__YOUTUBE__GRPC_FALLBACK_ENABLED	false	Enable gRPC streamList as fallback when InnerTube fails
rest_fallback_enabled	LUMIO__YOUTUBE__REST_FALLBACK_ENABLED	false	Enable REST polling as fallback when gRPC also fails; also gates Data API broadcast discovery fallback

The [youtube.innertube_observer] subsection controls badge caching and InnerTube credentials:

Setting	ENV override	Default	Description
api_key_override	LUMIO__YOUTUBE__INNERTUBE_OBSERVER__API_KEY_OVERRIDE	(empty)	Emergency pin for the InnerTube API key; disables auto-rotation for the key while set
client_version_override	LUMIO__YOUTUBE__INNERTUBE_OBSERVER__CLIENT_VERSION_OVERRIDE	(empty)	Emergency pin for the InnerTube client version; disables auto-rotation for the version while set
refresh_interval_secs	LUMIO__YOUTUBE__INNERTUBE_OBSERVER__REFRESH_INTERVAL_SECS	21600	Shared credential refresher interval (6 h)
cache_ttl_seconds	LUMIO__YOUTUBE__INNERTUBE_OBSERVER__CACHE_TTL_SECONDS	1209600	Member + tier-badge Redis entry TTL (14 d)

The one-time liveChatId resolution via Data API (liveBroadcasts.list) is always permitted regardless of rest_fallback_enabled.

Events Supported

Chat + Moderation

Event type	Description
TEXT_MESSAGE_EVENT	Regular chat message
TOMBSTONE	Silent removal (no-op)
MESSAGE_DELETED_EVENT	Message deleted — marked in DB, broadcast via WebSocket
MESSAGE_RETRACTED_EVENT	Message retracted — same handling as deleted
USER_BANNED_EVENT	User banned — moderation log entry with ban type and duration

Monetization

Event	Lumio event type	Key fields
SUPER_CHAT_EVENT	youtube:superchat	amount_micros, currency, amount_display_string, user_comment, tier
SUPER_STICKER_EVENT	youtube:supersticker	Same + sticker_id, alt_text

Membership

Event	Lumio event type	Key fields
NEW_SPONSOR_EVENT	youtube:member	member_level_name, is_upgrade
MEMBER_MILESTONE_CHAT_EVENT	youtube:member	member_level_name, member_month, user_comment
MEMBERSHIP_GIFTING_EVENT	youtube:gift_membership	gift_memberships_count, gift_memberships_level_name
GIFT_MEMBERSHIP_RECEIVED_EVENT	youtube:gift_membership_received	member_level_name, gifter_channel_id

Gift memberships use the same bundling pattern as Twitch gift subs: the header gift_membership event collects recipient names from individual gift_membership_received events, then broadcasts with a giftRecipients list. Individual received events are stored in DB but excluded from the event panel via exclude_bundled.

Interactive

Event	Lumio event type	Key fields
POLL_EVENT	youtube:poll / youtube:poll_end	question_text, options[] (text + tally), status

Polls are displayed in the ChatAlerts component using the same UI as Twitch polls.

Chat Sending + Moderation

Chat sending and moderation use the user's login OAuth connection (not the channel connection). InnerTube is read-only.

Operation	Endpoint	Token source
Send message	POST /youtube/v3/liveChat/messages	get_provider_token("google")
Ban / timeout	POST /youtube/v3/liveChat/bans	get_provider_token("google")
Delete message	DELETE /youtube/v3/liveChat/messages?id=\{message_id\}	get_provider_token("google")

All OAuth tokens are managed by the centralized Token Refresh Worker.

Quota Impact

Operation	Units	Frequency
InnerTube chat polling	0	Per poll interval (~3 s per active stream)
InnerTube broadcast discovery	0	Every 60 s
InnerTube statistics (likes, views)	0	Every 60 s per broadcast
liveChatId resolution (Data API)	5	Once per broadcast, cached forever
REST broadcast discovery fallback	1	Every 60 s (only when InnerTube fails + rest_fallback_enabled)
Chat sending	200	Per message sent
Moderation action	200	Per action
Channel enrichment	1	On demand, cached

Key Files

File	Purpose
crates/lo-youtube-api/src/innertube/mod.rs	InnerTube get_live_chat polling + API key/continuation bootstrap
crates/lo-youtube-api/src/innertube/parser.rs	Parse get_live_chat responses into InnerTubeChatEvent variants
crates/lo-youtube-api/src/innertube/events.rs	InnerTubeChatEvent enum definition
crates/lo-youtube-api/src/innertube/browse.rs	InnerTube browse (broadcast discovery, viewer counts, likes)
crates/lo-youtube-api/src/innertube/scraper.rs	Scrape API key + client version from youtube.com
apps/api/src/services/innertube_credentials.rs	Central API key + client version resolver, Redis persistence, one-shot rotation retry
apps/api/src/workers/innertube_credentials.rs	Shared 6 h credential refresher worker
crates/lo-youtube-api/src/streaming.rs	gRPC streamList client (fallback)
crates/lo-youtube-api/src/client.rs	REST Data API v3 client (fallback + moderation)
apps/api/src/workers/youtube.rs	Multi-stream worker: broadcast discovery, per-stream task lifecycle, InnerTube chat loop
apps/api/src/graphql/youtube.rs	youtubeActiveStreams GraphQL query
apps/api/src/routes/youtube_streams.rs	GET /v1/youtube/active-streams REST endpoint
apps/api/src/graphql/channel_status.rs	channelStatus query (includes broadcastStatus, likeCount, totalViews, scheduledStart, liveChatId)
apps/web/src/hooks/use-youtube-streams.ts	Frontend hook polling active streams
apps/web/src/app/(app)/dashboard/chat/multichat.tsx	Broadcast sub-menu + reply context