Release video frames against the audio clock
With the codec fixed the picture stopped stuttering but ran ahead of the sound, and for a structural reason: the audio takes a longer road to the call — our PCM goes through revoice's own ffmpeg and its buffer — while frames went out the moment they were decoded. Each frame now carries its position in the stream and waits until the audio that belongs with it has actually played, using the player's own playback position as the shared clock. Frames that fall more than 250 ms behind are dropped rather than shown late, so the picture recovers by itself instead of accumulating drift. The queue is bounded by bytes and never by pausing the stream: one ffmpeg feeds both outputs, so blocking the video pipe would stop the audio whose clock we are waiting on — a deadlock. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
ec50e4c4df
commit
bc567e953e
@@ -169,8 +169,8 @@ export interface PlaybackInput {
|
||||
cleanup(): void;
|
||||
/** Resolves with a reason if the downloader died on its own, for reporting. */
|
||||
failure?: Promise<string | null>;
|
||||
/** Raw I420 frames to publish as a screen share, when video is on. */
|
||||
video?: { stream: Readable; width: number; height: number };
|
||||
/** Raw I420 frames to publish alongside the sound, when video is on. */
|
||||
video?: { stream: Readable; width: number; height: number; fps: number };
|
||||
}
|
||||
|
||||
export interface PlaybackOptions {
|
||||
@@ -252,7 +252,12 @@ export async function openPlayback(
|
||||
input: pipeline.audio,
|
||||
inputOptions: pipeline.audioInputOptions,
|
||||
cleanup: () => pipeline.kill(),
|
||||
video: { stream: pipeline.video, width: pipeline.width, height: pipeline.height },
|
||||
video: {
|
||||
stream: pipeline.video,
|
||||
width: pipeline.width,
|
||||
height: pipeline.height,
|
||||
fps: config.VIDEO_FPS,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user