How accurate is it?
Last updated 31 October 2026
Subgloss works with nothing set up, and an AI service you choose can make it more accurate. These are our own tests, to help you decide whether that's worth it for you. They are small, and they aren't finished: read them as a guide, not a ranking.
Still missing: how accurate word meanings are from the AI services you bring a key for. So far we have judged only Gemini flash-lite, on Japanese.
Word meanings
How often each word's meaning was right for its sentence. Gemini Nano is Chrome's built-in AI, which runs on your computer on those that can run it.
| Video language | Dictionary, as installed | Gemini Nano | Gemini flash-lite |
|---|---|---|---|
| Japanese | 81% | 88% | 98% |
| Spanish | 87% | 94% | Not tested |
| French | 87% | 95% | Not tested |
| German | 86% | 91% | Not tested |
Compare within a row, not between rows: Japanese was tested only on 172 words with several possible meanings, the hard cases, and the other languages on every content word (between 328 and 463 of them, from three to six videos). German's two figures come from different videos. Measured September and October 2026.
Translation
60 Japanese subtitle lines from three YouTube videos, translated into English. Four judges read all fourteen translations of each line, not told which was which: two Claude models, one OpenAI model and one Google model, in that order below. A translation was correct if a learner would get the meaning right from it.
| Translated by | Correct, of 60 | Wrong, of 60 | Time a line |
|---|---|---|---|
| OpenAI gpt-6.1-terra Your OpenAI key | 57, 55, 58 and 54 | 0 | 2.5 s |
| Google gemini-4-argon Your Google key, with billing | 59, 58, 55 and 48 | 0 | 3.1 s |
| Google gemini-3.8-flash Your Google key, with billing | 56, 56, 54 and 52 | 0, 0, 0 and 3 | 6.5 s |
| OpenAI gpt-6-luna Your OpenAI key | 54, 53, 53 and 50 | 0 | 2.0 s |
| Claude Opus 5.5 Your Anthropic key | 55, 56, 52 and 46 | 0, 0, 0 and 1 | 1.8 s |
| Claude Sonnet 5.5 Your Anthropic key | 51, 52, 50 and 44 | 0, 0, 0 and 2 | 1.4 s |
| OpenAI gpt-5.4-mini Your OpenAI key | 51, 49, 47 and 41 | 0, 0, 1 and 1 | 0.9 s |
| Google gemini-3.5-flash-lite Google's free tier | 52, 51, 47 and 37 | 0, 0, 1 and 1 | 0.7 s |
| Google gemini-3.1-flash-lite Google's free tier | 50, 49, 45 and 37 | 1, 0, 1 and 5 | 2.1 s |
| Claude Haiku 5.5 Your Anthropic key | 44, 44, 44 and 38 | 0, 0, 0 and 2 | 1.0 s |
| TranslateGemma 12B Open model, on a computer of your own | 35, 35, 35 and 23 | 0, 0, 1 and 7 | 8.6 s; froze a 16 GB Mac |
| TranslateGemma 4B Open model, on a computer of your own | 33, 31, 27 and 18 | 5, 5, 7 and 12 | 1.1 s |
| YouTube's auto-translate Free | 24, 24, 23 and 15 | 10, 10, 13 and 23 | Whole video at once |
| Chrome's translator Free, on your computer | 24, 26, 20 and 9 | 8, 6, 13 and 15 | Instant |
Most lines not counted correct or wrong were partly right. The top five are too close for this test to separate them. YouTube's figures are lower than they should be: 8 of its wrong lines were left empty by Subgloss when it fitted YouTube's translation onto the subtitle lines, not mistranslated by YouTube. We have improved that fitting since, and haven't measured YouTube again. Translations made in October 2026, judged together on 31 October.
What each one costs and how fast it answers in Subgloss
Translations and meanings for 316 subtitle lines in Japanese, Spanish, French and German, asked exactly as Subgloss asks. This measures cost and speed, not accuracy. Nearly every line got an answer: gemini-3.5-flash-lite on a paid key once gave an unusable answer for 8 lines.
| Model | Cost for 1,000 lines | Meanings for 8 lines |
|---|---|---|
| OpenAI gpt-6-luna Subgloss's default for OpenAI, asked for low effort | $0.29 | 30 s |
| Claude Haiku 5.5 Subgloss's default for Anthropic | $0.43 | 15 s |
| Google gemini-3.1-flash-lite Free on Google's free tier | $0.84 | 10 s |
| Google gemini-3.5-flash-lite Subgloss's default for Gemini; free on Google's free tier | $2.17 | 10 s |
| OpenAI gpt-5.4-mini | $2.85 | 16 s |
| OpenAI gpt-6.1-terra | $3.07 | 49 s |
| Google gemini-3.8-flash | $4.96 | 45 s |
| Google gemini-4-argon From one video only | $7.85 | 49 s |
| Claude Sonnet 5.5 | $8.80 | 20 s |
| Claude Opus 5.5 | $16.50 | 31 s |
| Claude Opus 5 Subgloss's default for Anthropic before version 0.5.0 | $23.70 | 45 s |
You pay your AI service directly, at its own prices; these use Anthropic's, OpenAI's and Google's list prices of October 2026. Subgloss asks only for the lines around where you are watching, and keeps the results, so watching the same part again costs nothing. Eight lines is roughly 20 to 30 seconds of speech, so the slower models can fall behind the video. Times are median, from Australia.
Google's free tier. It costs nothing, and gemini-3.5-flash-lite was as quick on it as on a paid key; we tried gemini-3.1-flash-lite only on the free tier. Google turned some requests away at busy times, so a pause is possible: 1 time in the 316 lines on gemini-3.5-flash-lite, 5 on gemini-3.1-flash-lite. We didn't reach their daily limits. gemini-3.8-flash allowed only 20 requests a day and answered slowly, so it isn't usable free. gemini-4-argon isn't offered free.
We couldn't test gpt-5-mini, Subgloss's default for OpenAI before version 0.5.0: OpenAI only allows it for organizations that have verified their identity, and ours hasn't. If Subgloss says it can't run that model, choose another one in Options.
How far to trust these numbers
- The judges were AI, not people. In the translation test, models from three companies judged, not told which translation came from where. They agreed on 75% to 96% of verdicts, and the Google judge was the strictest. Each could still favour its own company's style, and the word meanings had Claude judges only. We want native speakers to check them too.
- The translation test is small. 60 lines, three videos, Japanese into English only, and all clean subtitles written by the uploaders. Auto-generated captions, Netflix and other languages may come out differently.
- It isn't quite how Subgloss works. In the translation test every service got one line at a time and the same plain request. In Subgloss, an AI service gets 25 lines at a time with the lines around them, which may change its results. Claude was asked for low effort and the others used their default. Subgloss now also asks gpt-6-luna for low effort: in a further check by one Claude judge, its translations and meanings were as accurate that way, so its row in the table should still hold.
- Models change. These are the versions we tested on these dates, and a service can change its models at any time.
Spotted something wrong? Tell us.