这是我对Kaggle 基准测试挑战赛

目录

一只眼,零借口

我真的很想为这次挑战录一个视频,就像我每次挑战时都会做的那样。但昨天我在眼睑上做了一个小手术,现在其中一只眼睛肿着还包着纱布。这是目前的状态:

Me with a patched-up eye, pirate style

没错。我现在看起来像个海盗。就差一只鹦鹉和一个酷炫的绰号了(欢迎在评论区提建议)。

但别担心(如果你本来要担心的话):我完全没事,两只眼睛的视力依然像鹰一样锐利。这只是眼睑模块的一次小 bug 修复。眼睛本身仍在生产中,运行顺畅。

话虽如此,在一只眼睛包着纱布的情况下完成一个项目,比听起来要难得多。深度感知?没了。读代码?糟透了。我不相信诅咒,但每次我参加挑战,总会出点事。如果有人在那边诅咒我,拜托,解除了吧。让我喘口气!我只想好好干活啊!

顺便感谢我兄弟开车把我从医院送回家,因为一只眼包着纱布开车感觉真是个非常糟糕的主意。

但截止日期就是截止日期,专业人士无论如何都会交付。所以我给自己打了打气:"兄弟,Michael Bisping 基本上靠一只能用的眼睛就成了 UFC 冠军。你完成一个编程项目肯定没问题。别找借口,你又不差。"于是我做到了。

让我告诉你我构建了什么。


我基准测试了什么

为什么是格鲁吉亚语?

我对 OpenAI 和 Anthropic 的模型进行了基准测试,看它们理解和生成格鲁吉亚语的能力如何。整个过程变成了一场真正的冒险,但在我们进入正题之前,让我先说说为什么选了这个主题。

独一无二,坚如磐石

格鲁吉亚语是世界上最古老的语言之一。它有自己的字母表,不属于任何主要语系。它完全独一无二。在语音上,它有时听起来有点像意大利语,因为这两种语言都痴迷于元音,但相似之处仅此而已。而对元音的痴迷并不意味着辅音弱。恰恰相反:格鲁吉亚语有大多数人从未听过的辅音,它们堆叠成让所有人都犯难的辅音簇(试着念一下 გვფრცქვნი,连续八个辅音;它大致意思是“你剥我们的皮”,是的,我知道这听起来像胡言乱语,但严格来说这个词是存在的)。这是一门极其丰富的语言。

在语法上,它难得离谱。我对此习以为常,因为第一,我是格鲁吉亚人,第二,我是个语言迷(即使在我学其他语言的时候也是如此)。我很懂格鲁吉亚语语法,但就连大多数格鲁吉亚人也不了解格鲁吉亚语语法的每一个细节,而且我几乎不认识任何说话不出一个错的人。是的,它就是这么深奥复杂。

即使是我这样一个语法迷,也请了一位语言专家检查我的数据集,以确保我没弄错。当她只带回 3 到 4 处小修正时,我特别开心。

活脱脱出自一款奇幻游戏(“შენ გგონია მომიგე? არა, ცდები!”)

格鲁吉亚语有其独特的魅力。字母是圆形的,很漂亮,看起来像是某部奇幻电影或游戏里的精灵文字。说到游戏,它其实被用在了《巫师 3》的《石之心》扩展包里。还记得你击败Gaunter O'Dimm的那一刻吗?在他那段简短讲话的中间某处,他说:

“შენ გგონია მომიგე? არა, ცდები!”
(“你以为你击败了我?不,你错了!”)

Gaunter O'Dimm

然后他用其他几种语言继续说了下去。开发者想让他说某种古老、超凡脱俗的语言,所以他们没有用胡编乱造的话,而是挑了几种符合这种氛围的真实语言。格鲁吉亚语就是其中之一。

好了,吹嘘我的母语有多棒就到此为止。我们进入技术部分吧。

我构建了什么

这个格鲁吉亚语基准测试是一组包含五项任务的测试,位于Kaggle Benchmarks这些测试考察 AI 模型处理格鲁吉亚语的能力:不只是阅读,而是真正正确地生成它。每一个数据集都是我亲手从零开始编写的。没有机器翻译,没有生成文本。

这花了一些时间和精力……所谓“一些”,我指的是多到离谱的时间和精力。

Dashboard Header

计划 vs 现实

我最初的计划,可以说,很有野心。格鲁吉亚语的九个维度(数字、动词、翻译、用拉丁字母书写的聊天风格格鲁吉亚语、谚语、日期和时间、语法判断,甚至图像),以及全部 41 个模型在 Kaggle 的目录中。我估计完整跑一轮大约需要$345。

我完全能接受。重点是要构建一个全面的东西,而不是省钱。而且说实话:如果我赢了,奖金完全能弥补这些。

但是……

然后我弄清了 Kaggle 的模型额度实际上是怎么运作的:你每天能拿到$10,免费,就这些。你不能用自己的钱充值。我真的去找过“添加资金”按钮,它根本不存在。(有一个研究资助项目,但不适用于挑战赛截止日期。)

Kaggle Limitation

哦,我当时特别沮丧,也气自己没好好研究 Kaggle 的规则。我不得不大幅改变我的方法。

所以就是每天 $10。而这一个限制最终塑造了(或者更准确地说,重新塑造了)这个项目中的几乎每一个决定:

  • 41 个模型变成了 8 个。两个供应商,四个层级,真正的配对而不是一大群人。
  • 9 个维度变成了 5 个。视觉、谚语、日期和时间以及语法判断被砍掉了。聊天格鲁吉亚语任务(45 个条目)也构建好了,然后同样被砍掉。
  • 数据集被裁剪到那些真正能区分模型的条目(详见下文)。没有任何东西被丢弃:完整数据集存放在仓库的archive/文件夹里,供任何想在 Kaggle 之外运行它们的人使用。

另一方面,这个限制让基准测试变得更好。更少、更精准的条目,以及一场公平的较量,而不是拥挤的排行榜。

五个任务

以下是模型必须面对的内容。下面每个示例都是基准测试中的真实条目。

1. 数字:以二十为单位计数

格鲁吉亚语以二十为单位计数。99 是ოთხმოცდაცხრამეტი,字面意思是“四个二十加十九”。在此基础上还有序数词、分数、小数,以及一直到 10³⁹ 的数字,双向都要:数字转单词和单词转数字。

问题 预期
第 44,用单词表示 ორმოცდამეოთხე
თვრამეტი(18),其序数词的数字形式 მე-18
1000000,用单词表示 მილიონი

中间那个比看起来更狡猾。格鲁吉亚语的序数词数字形式,当单词以 მე- 开头时,前缀保留在前面(მე-18),当不以 მე- 开头时,则加后缀(21-ე)。记住这一点,它会在结果中再次出现。

158 道题,按精确匹配评分(用|分隔可接受的替代正确答案)。

2. 动词:谁对谁做了什么?

一个格鲁吉亚语动词可以承载整个句子:谁做、对谁做、以及何时做。მიყვარხარ意思是“我爱你”(顺便说一句,好甜),全在一个词里。这个任务有三类问题:

  • 谁对谁:读一个动词形式,选出谁作用于谁。
  • 形式:为给定的人称和时态构建正确的动词形式。
  • 前缀:选出在上下文中给出正确含义的前缀。
id,kind,input,choices,expected
verbs-000,who_to_whom,მიყვარხარ,A) ის → შენ|B) მე → ის|C) შენ → მე|D) მე → შენ,D

50 道题,选择题,答案为字母或精确匹配。

3. 翻译:格鲁吉亚语到英语

十二个短段落,满是习语和棘手的动词,需要翻译成英语。例如,ხელი სთხოვა字面意思是“请求(她的)手”,但实际意思是“求婚”。

由于不止一种好的翻译,一个评判模型会将含义(而非措辞)与我的参考译文进行比较,采用 5 级评分:

4 = same meaning, nothing missing, wrong or added
3 = a small slip (a minor detail, tense or nuance), the main meaning intact
2 = partly right: one important part is wrong or missing
1 = only fragments right, the overall meaning is lost
0 = wrong, unrelated, or not English

一个有趣的细节:格鲁吉亚语没有语法性别,所以评判模型被明确告知,当格鲁吉亚语文本没有揭示性别时,“he”、“she”或“they”都可以。评判模型始终是同一个模型,Gemini 3 Flash,它不是被测试的模型之一。

4. 完形填空:填补空缺

两篇短文,一篇关于计算机的历史,一篇关于罗马帝国,每篇有25 个空缺。模型会得到一份 30 个词的词典形式列表,其中包括 5 个不属于任何地方(比如关于罗马的文章里的“television”、“banana”或“airplane”)。它必须为每个空缺选出正确的词并且将其变成正确的语法形式。

示例:ძველ[1]... →დროში(“在旧时代”)

50 个空缺,逐一评分。

5. 纠错:修好它,别弄坏它

带有故意植入的语法、拼写和标点错误的段落。模型必须修正它们而不改动任何已经正确的内容。以下是来自一个真实条目的一句话:

❌ მთელი ოჯახიმაგიდაზე დაჯდადა ერთადვივახშმედ。
✅ მთელი ოჯახიმაგიდას შემოუსხდადა ერთადვივახშმეთ。

评分是我最引以为傲的部分。它统计模型编辑前后与正确文本在词级别的差异:

# errors_before: 输入与正确文本的差距
# errors_after:  模型答案与正确文本的差距score = max(0, (errors_before - errors_after) / errors_before)

修正 4 个错误中的 3 个,你得 0.75。但把一个正确的词“修正”成错误的词,那就是一个新错误,会拉低你的分数。用自己的风格重写整段也不划算。过度修正会受到惩罚,就像严格的老师会做的那样。

17 个段落。

全部手写

所有这些数据都是我一个字一个字想出来的:186 行数字、115 个动词条目、47 个翻译段落、51 个纠错段落、45 段聊天文本和两篇完形填空文章。然后一位语言专家审阅了全部内容,我采纳了她的修改。

但每天只有 10 美元(Kaggle 允许的额度),我负担不起在八个模型上跑完全部内容。所以我做了精简。对于数字和动词,我用了一个探测模型:我在完整数据集上跑了一个便宜的模型,保留它答错的每一个条目,并丢弃一部分它觉得太容易的条目。大数字保持原样。对于翻译,我保留了 12 个最具格鲁吉亚语特有陷阱的段落,纠错段落是我手工挑选的,完形填空文本则完整保留。

任务 编写量 保留量
数字 186 行 82 行(158 道题)
动词 115 50
翻译 47 12
完形填空 2 篇 2 篇(50 个空)
纠错 51 17

这意味着每个模型 287 个计分条目。而且没错,这意味着这个基准测试是故意设计得很难的。绝对分数会比在简单数据集上更低,而排名才是关键。

Written vs. Kept

它在 Kaggle 上如何运行

Kaggle Benchmarks 让你用纯 Python 配合kaggle-benchmarks库来编写任务。Kaggle 会在其目录中的任意模型上运行它,并存储每一次调用:提示词、答案、token 和成本。

我的基准测试中的每个任务都有相同的形态:一个小的条目任务它提出一个问题并对其打分,以及一个外层任务在整个数据集上运行它并报告一个分数。下面是数字任务,稍微简化了一下:

@kbench.task(name="glb-numbers-item", store_task=False)
def number_item(llm, qid, kind, form, text, expected) -> bool:
    got = normalize(ask(llm, PROMPTS[(kind, form)].format(text=text)))
    accepted = [normalize(e) for e in expected.split("|")]
    kbench.assertions.assert_in(got, accepted, expectation=f"{qid}:{text}->{expected}")
    return got in accepted


@kbench.task(name="glb-numbers")
def numbers(llm) -> float:
    runs = number_item.evaluate(
        llm=[llm], evaluation_data=QUESTIONS, n_jobs=2, on_failure="continue"
    )
    correct = sum(1 for r in runs.completed_runs if r.result is True)
    return correct / len(QUESTIONS)


numbers.run(kbench.llm)

数据存放在一个 Kaggle 数据集中,整个工作流通过 Kaggle CLI 从终端运行:

scripts/push_data.sh "what changed"                  # 将 data/ 作为 Kaggle 数据集上传
scripts/push_task.sh glb-verbs tasks/glb_verbs.py    # 推送一个任务
kaggle b t run glb-verbs -m gpt-5.4-2026-03-05 --wait
kaggle b t download glb-verbs -m gpt-5.4-2026-03-05  # 每个答案、token 和成本

Verbs

Task Run

障碍与变通方法

冒险真正从这里开始。以下是一切试图阻止我的东西,以及我是如何绕过它们的。

障碍 1:Kaggle 会预留最坏情况下的费用

在每一次调用之前,Kaggle 都会预留该次调用的最大可能成本:模型的最大输出长度乘以其价格。如果这无法塞进你剩余的配额,调用就会被拒绝,即使真实答案的成本可能只有几分之一美分:

max estimated cost of operation ($1.60195) exceeds your available quota
(based on max_output_tokens)

我最初的顶级模型运行正是这样死掉的:许多调用被拒绝,而配额里还剩下大量真金白银。

变通方法:一个慷慨但合理的输出上限,并且只并行两次调用:

# 测试中见到的最长答案是 26,600 个 token。OUTPUT_LIMIT = {"max_completion_tokens": 32000}
...
runs = number_item.evaluate(llm=[llm], evaluation_data=QUESTIONS, n_jobs=2, ...)

障碍 2:失败的调用看起来像错误答案

当一次调用失败时(配额耗尽、API 抽风),该项就直接算作错误。所以一次中途没钱的运行会心安理得地报告 0.3 的分数,并把它放到排行榜上,仿佛这个模型不擅长格鲁吉亚语。那不是分数,那是谎言。

变通方法:每个任务中都有一个错误防护。少量孤立失败(重试 3 次后)算作错误,但如果超过 5% 的调用失败,任务就会停止并且不报告任何结果:

errored = len(runs.errored_runs)
if errored > 0.05 * len(QUESTIONS):
    raise RuntimeError(f"{errored}of{len(QUESTIONS)}items failed with errors; no score reported.")

这立刻就见效了:当最坏情况的预留(障碍 1)让我最初的一流运行失败时,两者都干净地停止了,没有给出虚假的分数。

障碍 3:有些模型默认会思考

有些模型在回答前会“思考”:它们在隐藏文本中私下推理,而你要为每一个隐藏 token 付费。我的一次测试运行写了大约770,000 个隐藏 token来回答 251 个问题,其中大多数答案只有一个词。GPT-5.5 写下了大约 750 个隐藏 token,来回答一道只有一个字母答案的单选题。

更糟的是,这并不公平:我的一些模型默认会思考,而另一些不会。把一个思考模型和一个非思考模型作比较,说明的更多是设置问题,而不是模型本身。

变通方案:让所有人处于相同条件。不思考,每个模型都直接回答:

llm.prompt(prompt, schema=Answer, reasoning="none", extra_api_params=OUTPUT_LIMIT)

不思考的话,一次完整运行的成本大约降低 99%。而且每个模型都遵守相同的规则。

障碍 4:停不下来的思考模型

当我处理到最新的一对模型时,GPT-6 Astra 直接拒绝了:

reasoning_effort does not support 'none' with this model.
Supported values are: 'low', 'medium', 'high', and 'xhigh'.

变通方案:最新的一对模型有自己的规则。Claude Opus 5 和 GPT-6 Astra 都在最低级别(“low”)思考,因此它们彼此在平等条件下竞争。它们在仪表盘中被标记为 ⚡ /LOW,这样就不会有人不小心把它们与不思考的模型作比较:

LOW_THINKING = ("claude-opus-5", "gpt-6-astra")

def reasoning_for(llm) -> str:
    name = getattr(llm, "model", "") or ""
    return "low" if any(m in name for m in LOW_THINKING) else "none"

障碍 5:每次推送都会运行任务

每次你推送任务的新版本时,Kaggle 都会用它的默认模型运行一次,Gemini 3.7 Flash,而那次运行也会计入你的配额。每次重新推送完整基准测试,花费我 0.45 到 1.00 美元。到最后,这些自动运行花了我将近3 美元,超过最终对比运行成本的一半。教训:在本地测试,尽量少推送。

Obstacle 6: The Small Stuff

  • The dataset path moves.Kaggle mounts the dataset somewhere under/kaggle/input, but the exact path varies. So every task searches for its file:
  def data_file(name: str) -> Path:
      kaggle_input = Path("/kaggle/input")
      if kaggle_input.exists():
          return sorted(kaggle_input.rglob(name))[0]
      return Path(__file__).resolve().parent.parent / "data" / name
  • No automatic retries.Nested evaluations can't retry failed items, so every task retries API errors itself, 3 times with a 10-second pause.
  • Credentials expire.For local test runs, Kaggle's model proxy credentials expire every couple of hours.kaggle b auth -yrefreshes them.
  • Look-alike characters.A hyphen and an en dash look the same but aren't, so answers are normalized before comparison (Unicode normalization, dash-like characters, extra spaces).

Models Tested

I put eight models from Anthropic and OpenAI through the benchmark, paired level by level, under identical conditions:

Level Anthropic OpenAI
Light Claude Haiku 4.5 GPT-5.4 mini
Decent Claude Sonnet 4.6 GPT-5.4
Top Claude Opus 4.8 GPT-5.5
Newest⚡ Claude Opus 5 GPT-6 Astra

⚡ The newest pair is a special case: as you saw in Obstacle 4, GPT-6 Astra simply refuses to stop thinking, so both models of that pair think at "low".

Here are the rules every model played by:

Rule Why
No thinking Every model answers directly. Mixing thinking and non-thinking models wouldn't be fair.
One marked exception GPT-6 Astra can't switch thinking off, so the newest pair both think at "low". Marked ⚡ everywhere.
Comparable pairs Each level pairs an Anthropic and an OpenAI model of the same class.
Same prompts, same code English instructions, Georgian content, one task version for everyone.
A fixed judge Translations are judged by Gemini 3 Flash, which isn't one of the tested models.
Partial credit where it's fair Correction rewards every fixed error and penalizes every correct word broken.
Failures never look like scores Above 5% failed calls, a task reports no score at all.

Findings

Drumroll, please. 🥁

Model Numbers Verbs Translation Cloze Correction Average Cost
Claude Haiku 4.5 0.30 0.34 0.96 0.54 0.25 0.48 $0.10
GPT-5.4 mini 0.41 0.42 1.00 0.52 0.67 0.60 $0.05
Claude Sonnet 4.6 0.39 0.52 1.00 0.76 0.44 0.62 $0.29
GPT-5.4 0.46 0.54 1.00 0.88 0.66 0.71 $0.15
Claude Opus 4.8 0.54 0.68 1.00 0.84 0.68 0.75 $0.58
GPT-5.5 0.51 0.60 0.98 0.88 0.74 0.74 $0.30
Claude Opus 5 ⚡ 0.65 0.84 1.00 0.86 0.77 0.82 $1.22
GPT-6 Astra ⚡ 0.83 0.90 0.98 0.96 0.82 0.90 $1.73

Task Matrix

Level by Level

  • Light:GPT-5.4 mini beats Claude Haiku 4.5,0.60 vs 0.48, and costs half as much.
  • Decent:GPT-5.4 beats Claude Sonnet 4.6,0.71 vs 0.62, again at about half the price.
  • Top:Claude Opus 4.8 edges out GPT-5.5,0.75 vs 0.74. That's basically a tie.
  • Newest ⚡:GPT-6 Astra beats Claude Opus 5,0.90 vs 0.82, the best score of the whole benchmark. It's ahead on four of the five tasks, with the biggest jump on numbers (0.83 vs 0.65).

Final tally:OpenAI wins 3 of the 4 levels, and Anthropic's only win, at the top level, is basically a tie.

One disclaimer: this is my dataset, my rules. Test other areas of Georgian and the results might shift a bit. That said, I tried to keep the tasks as broad and general as I could.

Finding 1: Understanding Georgian Is Solved. Producing It Isn't.

Look at the translation column:0.96 to 1.00 for every model. Every one of them understands Georgian well enough to translate it into English almost perfectly.

Now look at the other four columns. The moment a model has toproducecorrect Georgian (build a number, form a verb, inflect a word, fix a sentence), the scores drop hard. Without thinking, even the best model (Claude Opus 4.8) gets almost half of the number questions wrong. Only GPT-6 Astra, with its small thinking budget, gets above 0.80.

Finding 2: Numbers Are the Final Boss

Numbers are the hardest task for 6 of the 8 models, and close to the bottom for the other two. And inside it, the gap is wild:

Question type Range across models
Words → digits, cardinal 0.74 to 0.96
Words → digits, ordinal 0.04 to 0.92
Huge numbers, 10¹⁵ to 10³⁹ 0.06 to 0.53

Reading a Georgian number into digits: fine. Writing its ordinal in digits: disaster. GPT-6 Astra is the big exception there with 0.92, while the next best model gets 0.62. Rememberმე-18from earlier? Most models wrote the suffix form instead, because that's the pattern they've seen more often. Withმე-40000(forty-thousandth), only 2 of the 8 models got it right.

Numbers Table

Finding 3: The Mistakes Are Very Georgian

My favorite part of the whole project is reading the wrong answers. They're not random. They're what you'd get if you tried to speak Georgian with grammar borrowed from somewhere else:

  • "I fell in love with you."In Georgian, falling in love is an inversion verb: the one who loves is marked like an indirect object, so it'sშემიყვარდი, literally "you became loved to me". Only GPT-6 Astra got it right. The others mostly built an ordinary transitive verb instead (შეგიყვარე).
  • "We sat."Georgian uses a different verb root for sitting when the subject is plural: "I sat" is ვიჯექი, but "we sat" isვისხედით. I planted ვიჯექით (a mix of the two) in a correction paragraph.Not a single model fixed it.
  • "Who wrote it?"Inდაგიწერია, the გ- marks "you" as the writer: "you have (apparently) written it". In the Georgian perfect, the markers flip. Only 2 of the 8 models got it right; the rest scattered across every other reading.
  • One million.It's simplyმილიონი. Most models write ერთი მილიონი ("one million") instead.

Examples Tab

Finding 4: Georgian Costs More Tokens for Claude

Same prompts, same Georgian text, very different token counts. Claude models used2 to 2.6 times more input tokensthan GPT models for exactly the same questions:

Task Claude, input tokens per call GPT, input tokens per call
Numbers 272 104
Verbs 310 130
Cloze 1,412 699
Correction 404 159

Part of that is each vendor's answer-format instructions, and part is how each tokenizer splits Georgian script. In practice it means a Georgian prompt costs a Claude user noticeably more than the price per token suggests.

Cost Charts

Finding 5: Cheap Can Be Smart

Cost per correct answer, on the tasks scored right or wrong:

  • GPT-5.4 mini:0.03 cents per correct answer. The bargain of the benchmark.
  • Claude Opus 5 ⚡:0.55 cents per correct answer, about 17 times the price of GPT-5.4 mini.
  • GPT-6 Astra ⚡:0.67 cents per correct answer. The best score, but about 21 times the price of GPT-5.4 mini per correct answer.

Side Quest: The Uninvited Guest

Remember Gemini 3.7 Flash, the model Kaggle runs automatically on every push? It's not part of my comparison, but it took the test every time I pushed, so I have its scores too. And it quietly scored around0.89 on average, above every model in my table except GPT-6 Astra (0.90).

The catch: even with thinking set to "none", it kept writing about480 output tokens per number question, while the models without thinking wrote about 16. So it seems to keep thinking a little no matter what you tell it. Which leads to a fun takeaway: for a language like Georgian,thinking seems to matter more than model size. The two best models in my table, GPT-6 Astra and Claude Opus 5, both think at "low" too.

What It All Cost

The whole project ran on Kaggle's free daily credits. Total spend across all the testing, probing and final runs:

Cost
Final comparison runs $4.63
Kaggle's automatic push runs $2.88
Earlier runs and cost probes $2.86
Local test runs $3.46
Total $13.84

Yes, the automatic push runs alone cost more than half as much as the actual benchmark. And spreading it over several days was the only way to stay under $10 a day.

Cost Breakdown

The Dashboard

Tables are nice, but I wanted something you can actually explore. So all the results live on an interactive dashboard:

  • Overview:the level-by-level showdown, the task matrix and the key findings
  • Cost & tokens:what each model paid per task, the token gap, cost per correct answer and the full project spend
  • Tasks:numbers by question type, verbs by type, correction split into fixed / missed / broken, and every cloze gap for every model
  • Examples:real questions with every model's actual answer and an explanation of the mistake
  • Method:where the data comes from, how each task is scored, the conditions and the limits

It's a single static HTML page with no framework, no build step and no chart library, generated by a Python script straight from Kaggle's run files. So the dashboard and this article can never disagree.

👉georgekobaidze.github.io/georgian-language-benchmarks

Limitations

Let's be honest about what this benchmark can and can't tell you:

  • Hard by design.The sets keep mostly items a model got wrong, so the absolute scores are lower than they'd be on an easy set. The ranking is the meaningful part.
  • Small sets.Some tasks have only 12 to 17 items, so a difference of a few points isn't significant.
  • One run per model.Re-runs of the same model differed by about ±0.03 per task.
  • The judge is a model.Translation scores depend on Gemini 3 Flash's judgment of meaning.
  • The token gap includes format overhead, not only the tokenizer.

What's Next

  • Bigger sets.The full, untrimmed datasets are already in the repo'sarchive/folder, ready for a run without the daily cap.
  • More vendors.Google, xAI and the open models, once the budget allows.
  • Thinking vs no thinking.The Gemini side quest begs for a proper experiment: the same model with and without thinking, on the same items.
  • More Georgian.Native speakers are very welcome to contribute new items, corrections and alternative answers.

My Benchmark

Wrapping Up

I started this with patches on one eye, and ended it with a benchmark, a dashboard and a lot of respect for how hard my own language is. The models understand Georgian almost perfectly. Producing it correctly is a whole different story, and that's exactly the gap this benchmark measures.

And where the hell is my PhD?I've seen people (multiple of them) getting PhDs for exponentially less, and I'm not even kidding. I'm dead serious.

If you speak Georgian, go check the Examples tab and tell me which mistake made you laugh the most. And if you have a good pirate nickname for me, you know where the comments are.

If you don't speak Georgian and are planning to learn it, think twice and ask yourself if you're fully committed, because it can take a good 3 to 4 years (or more) of absolutely intense learning, and even then you'll be barely average. Reach out and let me know if you need any tips.


Enjoyed this write-up? Let's stay connected!

I share more software engineering insights, projects, and experiments across these platforms: