Robustness report

Trace robustness benchmark: protocol

This is the written procedure behind the Trace Robustness Report. The harness lives in our private repository; we share it, with the test media, with any lab that wants to re-run the bench (info@creatorsenseai.com). Every attack below is also written out as a plain ffmpeg command, so it can be re-implemented without our code.

Version trace-bench/v1, 25 Oct 2026. This is the procedure behind the published results in /assets/bench/<date>/. It is written so a lab can re-run it with this repository, or re-implement the attacks without it. The attack definitions are also stored inside every results.json (attacks), so a result file always says what was run.

1. One command

cd wm-service
pip install -e '.[ml]'            # torch, videoseal, audioseal (CPU is fine)
python bench/run_bench.py --work /tmp/trace-bench --out bench/results/$(date +%F)

Options: --media DIR (your own clips; see section 3), --suites (voice,video_ab,collusion_sim,video_per_fan), --attacks (subset of the video attacks), --controls, --seed (default 20261025), --ab-seconds (default 64), --per-fan-seconds (18), --sim-segments (300,600,1200,2400), --sim-trials (10), --budget-min (40; items left when it runs out are listed as not run), --publish DIR (copy the result files there). Assets are cached in --work; delete it for a clean run.

Requirements: ffmpeg with libx264, libmp3lame, libopus and the flite filter (Ubuntu 24.04 ffmpeg package has all four). Python 3.11+.

2. What is measured

Three ways a creator's content leaves Trace, each read back by the production reader with its production accusation gate:

Every decode follows production order: a plain read, and if nobody is matched, a second read after registering the copy onto the original (undoes crop, mirror, bars). The second read counts as a second trial in the false-match bound.

Scoring

• identified / pass: the marked copy matched the recipient it was made for.
• inconclusive / fail: nobody matched. Not a false match.
• WRONG: a different recipient matched. Counted as a false match.
• Controls (unmarked source through the same attack): clean / pass if nobody is matched, FALSE_MATCH otherwise.
• Collusion cells (two recipients combine copies): pass unless someone outside the two is matched; caught says whether one of the two was named.
• bit accuracy: share of readable segments (A/B) or message bits (per-fan, averaged over windows; voice, the 16 ID bits) that equal what was embedded for the target, before any matching. For collusion cells it is measured against the first recipient's codeword.
• false-match bound: the reader's own union-bounded probability that a random codeword would match this well (A/B and simulations). Accusation needs ≤ 1e-6.

3. Media

Default wm-service/bench_media/ (provenance in PROVENANCE.json): three 6-second CreatorSense brand animations, abstract motion graphics, no people. The source for each video suite is built by:

1. normalising each clip with -vf scale=1280:720:force_original_aspect_ratio=decrease,pad=1280:720:(ow-iw)/2:(oh-ih)/2,fps=24,format=yuv420p and x264 CRF 16 (-preset veryfast), audio dropped;
2. joining them in file-name order;
3. topping up to the target length with ffmpeg-generated 12 s segments in rotation: testsrc2, mandelbrot, life (mold=10:ratio=0.1:death_color=#202020:life_color=#e0c080), all at 1280x720 24 fps, same normalisation;
4. one more pass through the same filter and CRF 16, cut to the target length.

The voice source is synthetic: ffmpeg's flite voice slt reading the first sentence of the US Declaration of Independence (public domain), 44.1 kHz mono, first 20 s, AAC 128 kbps.

Your own media: --media /path/to/clips. Any .mp4 .mov .mkv .webm .m4v files are used in name order. Results list them as clip_1, clip_2... with no hashes unless you pass --name-media. The harness never copies media into its output. Do not publish adult or third-party media; publish results only.

4. Marking

• Video: VideoSeal 1.0 public checkpoint (y_256b_img.pth, sha256 recorded in engine.videoseal_checkpoint_sha256), scaling_w 0.2, embedded on every frame. A/B segments are written with x264 CRF 18 -preset medium, per-fan files with CRF 6 (production settings).
• A/B release: variants A and B carry wmcore.splice.make_ab_messages(seed) (two 256-bit messages from a seeded torch generator); 1 s segments with forced keyframes; picture fingerprints of the source stored for alignment. Recipient codewords: ab_stream.codeword_for("bench-<seed>", "bench-video", "viewer-NNNN", segments) for viewer-0000...viewer-0999; the target is viewer-0417, the second colluder viewer-0099. A copy is the playlist of its A/B segments saved as one MP4 with -c copy (what a downloader does).
• Per-fan file: wmcore.video.embed_windowed, 3 windows, 16-bit ID; target fan 4321, second fan 999; roster = those two plus 998 IDs drawn with random.Random(seed).
• Voice: wmcore.voice.embed_voice, fan 4321; roster of 1,000 including it.

5. Attacks

{in} and {out} are the input and output files; multi-step attacks feed each step's output to the next. All commands are ffmpeg -y -loglevel error followed by the arguments shown.

Video

"..." is -c:v libx264 -preset veryfast -pix_fmt yuv420p -an.

screen_skew_moire is a simulation, not a phone filming a screen. Real phone-of-screen capture is a separate lab gate (PHONE_OF_SCREEN_LAB_GATE.md) and is not claimed by this benchmark.

Controls (unmarked source): none, crf28, screen_record by default.

Collusion on real media

Copy A = target, copy B = second recipient, {half} = half the duration.

Audio (voice notes)

Every audio attack is also run on the unmarked source as a control.

Collusion simulations

wmcore/collusion_sim.py. For each code length (segments), bit-error rate, number of colluders c = 1-5 and attack (interleave: each segment from a random colluder; splice: runs of 10 segments from each in turn; average: soft mean of the copies; majority; minority: where the copies differ keep the variant fewest have), and each scheme (uniform_v1, tardos_sym_v1), it runs --sim-trials independent releases with 1,000 recipients. The decoder reading is ±1 per segment (averaged for blends) plus Gaussian noise sized to the bit-error rate; bit = sign, confidence = magnitude. Bit-error rates: the rates measured by video_ab on crf23, crf35 and social_reupload (floored at 1%), plus a fixed 11% stress rate. Seeds: seed + cell index. Reported per cell: trials where at least one colluder and no innocent was named (caught), innocents named in total (false_accusations).

6. Outputs

results.json (schema: trace-bench/v1):

• engine: wmcore version, VideoSeal card and checkpoint hash, torch, videoseal, audioseal, numpy, OpenCV, ffmpeg versions.
• git: commit of the repository the run used, and whether wm-service had uncommitted changes.
• run: date, seed, command line, host, wall time, media list (names, hashes, durations, provenance), not_run (every skipped item and why), notes (anything done after the main run, such as a suite re-run).
• attacks: the definitions above, machine-readable.
• suites.<name>: cells, controls, collusion (dicts keyed by attack), each cell with outcome, pass, bit_accuracy, false_match_probability (A/B), seconds, and reader details; collusion_sim.cells is a list.
• summary: cells, passed, failed, false_matches, sim_false_accusations.

results.md: the same as tables. video_ab_stream.json, video_per_fan.json, voice.json: the 22 Oct per-file format, for pages that read it.

7. The 22 Oct 2026 run

The first published run (assets/bench/2026-10-22/) predates this harness. It used the same readers, gates, rosters (1,000), target IDs and the same attack commands for the attacks both runs share, with these differences:

• Video source: 64 s at 1280x720, 30 fps, made from the three bench_media clips plus the first 10 s of an internal test clip (not published), then mandelbrot, life and testsrc2 (12 s each), all with the normalisation above at fps=30.
• Extra video attacks: crf30 (CRF 30 instead of crf28), fps24 (-vf fps=24, CRF 23), speed1.05 (-vf setpts=PTS/1.05, CRF 23), trim7 (-ss 7, CRF 23), clip30_mid from 15 s in (-ss 15 -t 30), letterbox (-vf scale=1280:540,pad=1280:720:0:90), color (-vf eq=contrast=1.25:brightness=0.06:saturation=1.4), grayscale (-vf hue=s=0), overlay_box (-vf drawbox=x=iw*0.6:y=ih*0.05:w=iw*0.35:h=ih*0.2:color=white@0.7:t=fill), all -preset veryfast -an, CRF 23 unless stated.
• No unmarked video controls; no collusion simulations.
• Photos (TrustMark, 4 frames from the same clips, 21 attacks plus unmarked controls) and a recorded speech clip (not published) for voice notes.

8. Reproducibility notes

• Seeds fix every random choice (messages, codewords, rosters, simulations). Two things are not bit-exact across machines: x264 output depends on the libx264 build and thread count, and VideoSeal on CPU has float rounding differences between torch builds. Expect bit accuracies to match within a few tenths of a percent and verdicts to match, not identical files.
• The git commit recorded is the one checked out when the run finished. If the branch was rebased later, the same code is under a new hash; the engine block and the attack definitions in the file are authoritative.
• This is a robustness test of the marks under common handling, on short synthetic and animated footage, on CPU. It does not show how Trace performs on any specific creator's content, against an attacker who targets VideoSeal directly, or for real phone-of-screen recordings.