How we check and test AI video tools
The 19 measurements we run on AI video tools and the rules every test follows. No tests have been run yet.
Where listings come from
Each listing links to the vendor page or repository it came from. Licences, last commits and archive status come from the GitHub API.
The measurements
Each uses a fixed test clip so tools are compared on the same footage.
- Transcribe & subtitle
Transcript WER
Word error rate against human reference
- How
- WER = (substitutions + deletions + insertions) / reference words
- Test asset
- 5-min noisy 2-speaker English sample
- Unit
- Fraction; lower is better
- Record
- Record model version, language and device
- Transcribe & subtitle
Word-alignment drift
Absolute subtitle word timing error
- How
- Median absolute delta vs human timecodes
- Test asset
- 50 randomly sampled words with frame-accurate reference
- Unit
- Seconds; lower is better
- Record
- Report distribution and worst 10 errors
- Edit & repurpose
Usable clips / suggestions
Ratio of editor-approved clips to all agent suggestions
- How
- Approved clips divided by all suggested clips
- Test asset
- Same 20-min raw talking-head video
- Unit
- Percent; higher is better
- Record
- Approval rubric: hook clarity; context; sound; framing
- Edit & repurpose
Manual edit rescue time
Human minutes required after automated draft
- How
- Stopwatch time from AI draft to accepted export
- Test asset
- Single sample, one trained editor
- Unit
- Minutes / accepted short; lower is better
- Record
- Report creative-quality tradeoff
- Edit & repurpose
Narrative continuity
Does travel edit preserve chronology and meaning
- How
- Human rubric 1-5 with written notes
- Test asset
- 12 travel shots in fixed storyboard
- Unit
- Human rubric; do not pretend objective
- Record
- Review pacing and factual continuity
- Edit & repurpose
Aspect ratio/framing
Portrait crop keeps face/hands and on-screen text
- How
- Manual frame spot-check at 5 random intervals
- Test asset
- 9:16 export using real hand-held footage
- Unit
- Pass/fail + percentage of frames
- Record
- Check Shorts safe areas
- Generate video
AI video shot consistency
Do generated entities stay consistent across cuts?
- How
- Blind side-by-side review by two people
- Test asset
- Same 3 prompt shots, duration and aspect
- Unit
- Rubric 1-5 + notes
- Record
- Track model version and seed if available
- Agent orchestration
Render/export success
Output usable MP4 with correct dimensions and sound
- How
- Successful exports divided by fixed test attempts
- Test asset
- 3 runs per candidate on isolated environment
- Unit
- Percent; higher is better
- Record
- Measure codec, audio sync, subtitles and frame rate
- Cost
Cost per accepted minute
Real all-in third-party API + subscription usage
- How
- Total attributable cost / accepted finished video minutes
- Test asset
- Record API charges and reusable tool costs
- Unit
- AUD or USD per accepted minute
- Record
- No fake free labels; track GPU depreciation separately
- Speed
Time to publishable minute
True editor + machine time per finished minute
- How
- Elapsed wall time and human touch time
- Test asset
- Same raw source for each candidate
- Unit
- Minutes; lower is better
- Record
- Report human and machine separately
- Agent developer experience
Install success
First run produces valid output from clean machine
- How
- Successful install within 30min / attempt
- Test asset
- Fresh VM (macOS/Windows/Linux specified)
- Unit
- Pass/fail and minutes
- Record
- Record GPU and dependencies
- MCP security
Write permission boundary
Agent can only write explicitly permitted test assets
- How
- Attempt scoped access outside test folder/account
- Test asset
- Sandbox credentials and fixture data
- Unit
- Pass/fail; serious security flag
- Record
- Never execute untrusted code on production
- MCP security
Human approval before publish
External publish action requires explicit approval
- How
- Attempt upload/publish without final user consent
- Test asset
- Private test YouTube channel / Wistia account
- Unit
- Pass/fail
- Record
- No auto-publication, no live secrets in agent context
- Privacy
Local-only privacy
Tool processes source locally without unapproved outbound transfer
- How
- Monitor network under known test with internet denied
- Test asset
- Creator-owned clip containing dummy identifiers
- Unit
- Pass/fail and outbound destinations
- Record
- Never use live sensitive PII as test data
- Commercial use
Licensing audit
License permits actual intended deployment
- How
- Check repo file + model weights + software modules
- Test asset
- Current tagged release; rendering binary dependencies
- Unit
- Verified / unclear / restricted
- Record
- n8n, Remotion, CodeFormer, VEED binaries differ
- Translate & localise
Translation fidelity
Dubbed meaning survives idioms, dates and proper nouns
- How
- Native-speaker review across 20 phrases
- Test asset
- 2-min authorised English->Spanish sample
- Unit
- Error count / severity
- Record
- Also measure lip-sync and audible voice artifacts
- Design & thumbnails
Thumbnail readability
Legibility of one headline at mobile size
- How
- 5-person rapid view test at 160px wide
- Test asset
- 3 fixed creator brief thumbnails per service
- Unit
- Comprehension rate and edit minutes
- Record
- Do not invent CTR improvement
- Manage & distribute
API quota and error recovery
Workflow handles quota, retries and failed uploads
- How
- Induce expected 429 / invalid OAuth in sandbox
- Test asset
- Small private MP4 + blocked token
- Unit
- Pass/fail; recovery time
- Record
- Avoid repeated publish duplicates
- Research & optimise
Claims/source integrity
Video SEO advice cites actual source rather than fabricated demand
- How
- Review 10 topic/title suggestions
- Test asset
- Owner channel data + documented source list
- Unit
- Supported claims / total
- Record
- Never equate competitor view data with demand
What every test records
- The clip: owned by VloggingPro or used with written consent, its duration, and what it contains (speakers, noise, languages).
- The machine: model, chip, memory, operating system; GPU if used.
- The tool: exact version or model name, settings, date, and whether it ran locally or in the cloud.
- The run: number of runs, success or failure of each, wall time and hands-on time kept separate.
- The reviewer and their rubric, and the raw outputs (transcripts, SRT files, exports) where the footage rights allow publishing them.
- Cost: real charges from the invoice or usage page, divided by accepted finished minutes. No price is assumed.