Sync the picture to the sound people hear, not the sound queued

The diagnostics came back clean — aheadBy steady at 3.3 s, nothing
dropped, nothing re-pegged — so the remaining lip-sync error was a
constant, and the cause is the clock: media.seconds counts audio handed
to LiveKit, whose audio source holds up to a second (its default queue)
before anyone hears it. Frames released against that ran a queue-length
early, every time.

The clock now subtracts what is still queued, which the source reports
directly, so the compensation follows the real buffer instead of a
guessed constant.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Leonid Pershin
2026-09-09 04:12:48 +03:00
co-authored by Claude Opus 5
parent 9f024f0415
commit 7a369cf9f2
2 changed files with 18 additions and 4 deletions
+12 -4
View File
@@ -383,10 +383,10 @@ export class GuildPlayer extends EventEmitter<GuildPlayerEvents> {
width: video.width, width: video.width,
height: video.height, height: video.height,
fps: video.fps, fps: video.fps,
// Frames are released against the sound that has actually played, which is // Frames are released against the sound people have actually heard, not
// the only clock both sides share: the audio path runs through revoice's // the sound handed to LiveKit: its audio source buffers up to a second,
// own ffmpeg and buffer, so it always trails the raw frames. // and gating on the queued position showed every frame that much early.
audioClock: () => this.media?.seconds ?? 0, audioClock: () => this.heardSeconds(),
}); });
try { try {
await publisher.start(room, video.stream, (reason) => { await publisher.start(room, video.stream, (reason) => {
@@ -402,6 +402,14 @@ export class GuildPlayer extends EventEmitter<GuildPlayerEvents> {
} }
} }
/** Playback position as heard, i.e. minus whatever is still queued in LiveKit. */
private heardSeconds(): number {
const media = this.media;
if (!media) return 0;
const queuedMs = media.source?.queuedDuration ?? 0;
return Math.max(0, media.seconds - queuedMs / 1000);
}
private stopScreenShare(): void { private stopScreenShare(): void {
const publisher = this.videoPublisher; const publisher = this.videoPublisher;
this.videoPublisher = null; this.videoPublisher = null;
+6
View File
@@ -9,6 +9,12 @@ const require = createRequire(import.meta.url);
export interface MediaPlayerLike { export interface MediaPlayerLike {
readonly seconds: number; readonly seconds: number;
/**
* LiveKit's audio source. `seconds` counts what has been handed to it, and it
* holds up to a second before anyone hears it — `queuedDuration` (in ms) is
* that gap, which matters when lining a picture up with the sound.
*/
source?: { readonly queuedDuration: number } | null;
readonly duration: number; readonly duration: number;
codecData?: { duration?: string } | null; codecData?: { duration?: string } | null;
paused: boolean; paused: boolean;