Fix video codec, aspect and placement

Three problems visible at once on a running instance: the picture
stuttered and drifted behind the sound, it was letterboxed oddly, and it
appeared as a separate tile instead of coming from the bot.

- YouTube handed us AV1 (format 398). Software-decoding AV1 at 720p does
  not sustain real time on a small server, which explains both the
  stutter and the drift; H.264 is now requested first, VP9 second.
- The frame was padded into a fixed box, so a clip whose proportions
  differed got black bars baked in and then more from the client. Size is
  now a bounding box and the frame keeps the clip's own proportions.
- Video was published as a screen share, which every client renders as
  its own tile. VIDEO_SOURCE=camera (the new default) puts it inside the
  bot's tile; "screen" keeps the old behaviour.

VIDEO_SYNC_OFFSET_MS is there for the residual drift, since audio and
video travel as two separately published tracks. Measured loudnorm first
to rule it out as the cause of the desync: it adds 0 ms.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Leonid Pershin
2026-09-09 02:29:29 +03:00
co-authored by Claude Opus 5
parent 9285fd7fbd
commit 0b17b880d6
7 changed files with 113 additions and 25 deletions
+5
View File
@@ -69,9 +69,14 @@ const schema = z.object({
.transform((value) => value === "true"),
// YouTube only serves 360p as a single progressive file, and only such files
// can be streamed, so that is the realistic default.
// Treated as a bounding box: the frame keeps the clip's own proportions.
VIDEO_WIDTH: z.coerce.number().int().positive().default(640),
VIDEO_HEIGHT: z.coerce.number().int().positive().default(360),
VIDEO_FPS: z.coerce.number().int().min(1).max(60).default(24),
/** "camera" shows up inside the bot's tile, "screen" as a separate one. */
VIDEO_SOURCE: z.enum(["camera", "screen"]).default("camera"),
/** Nudge the picture against the sound, in milliseconds; positive delays video. */
VIDEO_SYNC_OFFSET_MS: z.coerce.number().int().default(0),
DEFAULT_VOLUME: z.coerce.number().min(0).max(200).default(60),
MAX_QUEUE_SIZE: z.coerce.number().int().positive().default(500),