Release video frames against the audio clock

With the codec fixed the picture stopped stuttering but ran ahead of the
sound, and for a structural reason: the audio takes a longer road to the
call — our PCM goes through revoice's own ffmpeg and its buffer — while
frames went out the moment they were decoded.

Each frame now carries its position in the stream and waits until the
audio that belongs with it has actually played, using the player's own
playback position as the shared clock. Frames that fall more than 250 ms
behind are dropped rather than shown late, so the picture recovers by
itself instead of accumulating drift.

The queue is bounded by bytes and never by pausing the stream: one ffmpeg
feeds both outputs, so blocking the video pipe would stop the audio whose
clock we are waiting on — a deadlock.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Leonid Pershin
2026-09-09 02:56:04 +03:00
co-authored by Claude Opus 5
parent ec50e4c4df
commit bc567e953e
3 changed files with 235 additions and 147 deletions
+8 -3
View File
@@ -169,8 +169,8 @@ export interface PlaybackInput {
cleanup(): void;
/** Resolves with a reason if the downloader died on its own, for reporting. */
failure?: Promise<string | null>;
/** Raw I420 frames to publish as a screen share, when video is on. */
video?: { stream: Readable; width: number; height: number };
/** Raw I420 frames to publish alongside the sound, when video is on. */
video?: { stream: Readable; width: number; height: number; fps: number };
}
export interface PlaybackOptions {
@@ -252,7 +252,12 @@ export async function openPlayback(
input: pipeline.audio,
inputOptions: pipeline.audioInputOptions,
cleanup: () => pipeline.kill(),
video: { stream: pipeline.video, width: pipeline.width, height: pipeline.height },
video: {
stream: pipeline.video,
width: pipeline.width,
height: pipeline.height,
fps: config.VIDEO_FPS,
},
};
}