Audio-Visual Highlight · ablation · shared vs independent bridge time
Does each modality need its own diffusion time?
The bridge model draws a time t and asks the network to move audio and
video that far toward the edited target. The baseline run draws two independent times
(ta, tv) with a second timestep embedder;
the ablation draws one shared t for both and is otherwise byte-identical — same data, architecture,
bridge noise σ = 1, optimiser and seed. Both are decoded here on the same 84 held-out clips with the
same sampler (Euler, t₀ = 0.02, 10 steps), so every difference comes from that one flag.
The ablation, at the matched checkpoint
Both runs saved a checkpoint at optimiser step 12,000. This is the only comparison in which nothing but the flag
differs; everything below it is context. Edit size = ‖generated − input‖ / ‖target − input‖
on the waveform — 1.00 means the model moved the audio exactly as far as the ground truth did.
step 12,000
audio PSNR ↑
LSD ↓
clips > floor
edit size
cos envelope
video PSNR ↑
SSIM ↑
LPIPS ↓
FVD ↓
pass-throughreturn the input unchanged
21.20
9.77
–
0.00
–
16.87
0.600
0.302
362
independent (ta, tv)baseline, 2 embedders
21.68
9.54
42/84
0.71
0.40
18.12
0.626
0.269
289
shared tablation, 1 embedder
22.11
9.53
42/84
0.71
0.43
17.60
0.611
0.270
310
shared − independentthe ablation effect
+0.43
-0.01
+0
+0.01
+0.03
-0.51
-0.015
+0.001
+21
Every checkpoint scored on these 84 clips
Pass-through is the floor: return the input untouched. Because the target differs from the input by only one
amplified object, a model that does nothing already scores 21.20 dB, so PSNR alone cannot tell you whether an
edit happened — read it together with edit size.
audio
video
checkpoint
steps
PSNR ↑
LSD ↓
clips > floor
edit size
PSNR ↑
SSIM ↑
LPIPS ↓
FVD ↓
pass-throughreturn the input unchanged
–
21.20
9.77
–
0.00
16.87
0.600
0.302
362
independent (ta, tv) — baseline run
epoch 895.6k steps
5.6k
21.31
9.78
33/84
0.93
18.15
0.621
0.270
317
step 8kscores only
8k
21.07
9.37
28/84
0.89
18.01
0.635
0.265
327
epoch 1529.5k steps
9.5k
21.84
9.32
42/84
0.75
18.16
0.630
0.267
306
epoch 161val-best, 10.0k
10k
20.87
10.09
32/84
0.82
18.38
0.638
0.262
303
step 12kmatched point
12k
21.68
9.54
42/84
0.71
18.12
0.626
0.269
289
step 16kscores only
16k
22.00
9.26
51/84
0.52
18.09
0.632
0.267
297
epoch 32720.3k steps
20.3k
22.07
9.14
51/84
0.55
18.19
0.630
0.269
292
step 24kscores only
24k
21.67
9.70
49/84
0.62
18.15
0.625
0.268
319
shared t — ablation run
step 12kmatched point
12k
22.11
9.53
42/84
0.71
17.60
0.611
0.270
310
epoch 230latest, 14.3k
14.3k
21.82
9.76
44/84
0.64
18.19
0.628
0.270
302
earlier run, before the noise fix
earlier runsigma 0.01, epoch 985
61.1k
19.81
10.54
26/84
1.14
17.85
0.612
0.282
666
The ablation is a null result so far. At step 12k the shared-t run is
+0.43 dB on audio PSNR and −0.52 dB on video PSNR. Both are inside the baseline’s own
checkpoint-to-checkpoint spread on this test set (audio PSNR moves 21.07 → 21.68 → 22.00 → 21.67
across 8k/12k/16k/24k; FVD moves 327 → 289 → 297 → 319), so neither sign survives as evidence.
The one difference that is not noise is that the video dip does not reproduce. Shared-t
scores 17.60 dB / SSIM 0.611 at 12k but 18.19 dB / 0.628 at 14.3k — a 0.6 dB swing over
2,300 steps, where the baseline moves by 0.07 dB over 16,000. Read the 12k video row as one noisy point,
not as a cost of sharing t.
The diagnostic the flag was supposed to move did not move. The loss-vs-t
probe at the matched step is the same curve in both runs (audio, relative to a predict-nothing baseline:
0.12 / 0.11 / 0.12 / 0.14 at t = 0.02 / 0.05 / 0.1 / 0.2 for independent t, and
0.11 / 0.10 / 0.11 / 0.13 for shared; video is flat near 1.0 in both). Whatever the second timestep embedder
buys, it is not a different objective landscape — and it costs 3.67 M parameters, 0.31 % of the
trainable model.
Both runs are far above the earlier σ = 0.01 model (FVD 666, audio 19.81 dB), so the
change that mattered was the bridge noise, not how t is drawn.
Caveat: the ablation run is unfinished. It is at 14.3k of a planned 24k steps, and the baseline is not
monotone in training — its best audio point is 16k, not its last. The comparison has to be repeated at 24k
before the ablation cell can be reported as settled.
Clip by clip
Each card: the three reference streams, then the baseline run’s checkpoints, then the ablation’s.
The two tiles outlined in colour are the matched step-12k pair — the actual ablation. Videos carry their own
audio; the players underneath compare the waveforms directly. PSNR vs floor is that clip’s generated
PSNR minus its pass-through PSNR, so positive means the edit helped on that clip.
01
drum
multiple11 visible
EcMP56K2VvY
Reference
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted
Inputthe untouched clipConditionmask latent decoded back to pixels — what the model is told to highlightTargetground truth: object louder and visually lifted