在本教程中，我们使用 Laya，这是来自 Convai Innovations 的开源决策引擎，它成为了 2026 年 9 月获得最多 star 的机器学习仓库之一。Laya 是一个非自回归的 System 1 模型：它不是生成文本，而是一个拥有 4.21 亿参数的编码器读取一段文本和一组带类型的问题、标签之间的选择、量表上的分数或 yes/no，并在一次前向传播中为每个选项返回一个概率，输出 token 数为零。它的卖点是速度和经过校准的概率，是 TypeSafe 的 Jev 的开源答案。我们没有重复 README 中的示例，而是把这些承诺应用到带有已知答案的真实标注数据上，即 CLINC150 意图数据集的银行领域，并测量生产路由器实际得到的结果：相对于训练过的分类器的零样本准确率、选项的措辞和顺序有多大影响、所提供概率有多诚实、在验证数据上拟合温度能修复什么以及它会悄悄破坏什么、按错误预算拟合的弃权门控、超出范围的流量、一个温度无法修复的 yes/no 问题，以及来自 pydantic schema 的带类型输出。
复制代码已复制使用其他浏览器
import os
import sys
import time
import json
import warnings
import traceback
import subprocess
import urllib.request
RESULTS = {}
def banner(title):
print("\n" + "=" * 78)
print(title)
print("=" * 78)
def section(name):
def wrap(fn):
def run(*a, **kw):
banner(name)
try:
out = fn(*a, **kw)
RESULTS[name] = out if isinstance(out, str) else "ok"
return out
except Exception as e:
RESULTS[name] = f"SKIPPED / FAILED -> {type(e).__name__}: {e}"
print(f"\n[!] {name} did not complete: {type(e).__name__}: {e}")
traceback.print_exc(limit=3)
return None
return run
return wrap
banner("1. Install Laya and load the English checkpoint at its reviewed revision")
subprocess.run([sys.executable, "-m", "pip", "install", "-q", "laya==0.3.27"], check=True)
import numpy as np
import pandas as pd
import torch
import laya
from laya.calibrate import records_from_labeled
from laya.evals import selective_accuracy, aurc
from laya.common import temp_bucket
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
# laya.load() 默认遵循 Hub 的 main 分支，除非另有指定。该包附带的是
# 其作者为每个检查点审核过的 commit；固定它可以让本 notebook 的权重保持不变。
REVISION = laya.PINNED_REVISIONS["convaiinnovations/laya"]
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
agent = laya.load("convaiinnovations/laya", device=DEVICE, revision=REVISION)
# 在 CUDA 上 Laya 会自动转换为 fp16/bf16。关闭该功能会让所有设备都保持 fp32，因此 GPU 运行
# 复现出的下方 CPU 数值仅在浮点噪声范围内有差异。
agent.amp_enabled = False
SHIPPED = (list(agent.temperature), dict(agent.temperature_by_options))
n_params = sum(p.numel() for p in agent.model.parameters())
print(f" laya {laya.__version__} | torch {torch.__version__} | device {DEVICE}, fp32")
print(f" checkpoint convaiinnovations/laya @ {REVISION[:7]} | {n_params / 1e6:.0f}M parameters"
f" | max_len {agent.cfg['max_len']}, head_max_len {agent.cfg['head_max_len']}")
print("\n Temperatures shipped with the checkpoint (probabilities = softmax(logits / T)):")
for qt, name in enumerate(["choice", "score", "noul"]):
print(f" {name:7s} type-level T = {SHIPPED[0][qt]:.3f}")
for bucket, t in sorted(SHIPPED[1].items()):
print(f" {bucket:12s} T = {t:.3f}")
for w in caught:
if "temperature" in str(w.message):
print("\n Warning at load time:\n " + str(w.message).replace("; ", ";\n "))
print("\n T > 1 softens probabilities and T < 1 sharpens them. Remember the choice:11+ row: the")
print(" checkpoint ships 0.10 there, which the loader clamps to 0.5, so any choice question with")
print(" 11 or more options gets probabilities SHARPENED by 2x. Step 6 measures what that costs.")
我们安装已发布的包 laya 0.3.27，并加载英文 checkpoint。这里有两个选择保证了运行的可复现性。默认情况下 laya.load 会跟随 Hugging Face 主分支，因此我们固定到该库作者自己审阅过的 revision，它通过 laya.PINNED_REVISIONS 暴露出来。而在 CUDA 上 Laya 会自动转换为半精度，所以我们关掉它，让所有设备都保持 fp32，从而让 GPU 运行复现这里展示的 CPU 数值。打印 checkpoint 自带的 temperatures 在任何预测之前就揭示了第一个发现：针对有十一个或更多选项的选择题的条目是 0.10，超出了有效范围，因此加载器将其钳制为 0.5 并发出警告。低于一的 temperature 会锐化概率，因此对具有那么多选项的问题的每个答案都会看起来比原始模型确定两倍。
复制代码已复制使用其他浏览器
TICKET = "你好，我们三月份被重复扣费了。请今天退还重复的那笔费用，否则我们将取消我们的套餐。"
TRIAGE = {
"department": {"type": "choice", "instructions": "哪个部门应该处理这个？",
"criteria": {"billing": "发票、付款、退款",
"technical": "缺陷、故障、系统错误",
"other": "其他所有"}},
"urgency": {"type": "score", "instructions": "这件事有多紧急？",
"criteria": ["不紧急", "尽快", "阻塞"]},
"churn_risk": {"type": "noul", "instructions": "用户是否威胁要取消或离开？"},
}
@section("2. 一次前向传播，三个类型化问题，零输出 token")
def first_decision():
r = agent.predict(TICKET, TRIAGE)
a = r["answers"]
d, u, c = a["department"], a["urgency"], a["churn_risk"]
print(f" state: {TICKET!r}\n")
print(f" department (choice) -> {d['choice']!r} probabilities {d['probabilities']}")
print(f" urgency (score) -> {u['score']:.2f} on 0..2 level probabilities {u['probabilities']}")
print(f" churn_risk (noul) -> P(yes) = {c['noul']:.3f}")
print("\n Two confidence fields, two different quantities:")
for qid, ans in a.items():
print(f" {qid:11s} answer_confidence {ans['answer_confidence']:.3f} confidence {ans['confidence']:.3f}")
print(" answer_confidence is the probability of the reported answer, max(p): the number that")
print(" calibration, the abstention gate and every metric below use. confidence is 1 - normalised
print(" entropy, whose scale depends on the number of options. Do not threshold on it.")
print(f"\n usage: {r['usage']}")
print(" output_tokens is always 0: Laya scores the options it is given and never generates text.")
return f"{d['choice']} / urgency {u['score']:.2f} / P(churn) {c['noul']:.2f} in one pass"
first_decision()
对 predict 的一次调用即可在单次前向传播中回答关于一张支持工单的三个带类型问题：部门作为选择项，紧急程度作为 0 到 2 的分数，流失风险作为是/否。结果会为每个选项附带一个概率，以及两个容易混淆的置信度字段。answer_confidence 是所报告答案的概率，它正是本教程后续的 calibration、abstention gate 以及每一个指标所使用的量。confidence 是 1 减去归一化熵，其尺度取决于一个问题有多少个选项。usage 块显示输出 token 为零，因为 Laya 只对它被给定的选项打分，从不生成文本。
Copy CodeCopiedUse a different Browser
@section("3. What a pass costs: questions are rows, options are nearly free")
def cost_model():
def median_ms(q, n=7):
agent.predict(TICKET, q)
times = []
for _ in range(n):
t0 = time.perf_counter()
r = agent.predict(TICKET, q)
times.append(1000 * (time.perf_counter() - t0))
return float(np.median(times)), r["usage"]["input_tokens"]
print(f" {'one state, asking ...':34s} {'ms (median of 7)':>16s} {'input_tokens':>13s}")
rows = {}
for n in (1, 4, 16):
q = {f"q{i}": {"type": "noul", "instructions": f"Does the message mention topic number {i}?"} for i in range(n)}
rows[f"{n} yes/no"] = median_ms(q)
print(f" {f'{n:2d} yes/no questions':34s} {rows[f'{n} yes/no'][0]:16.1f} {rows[f'{n} yes/no'][1]:13d}")
for k in (3, 15, 40):
q = {"pick": {"type": "choice", "instructions": "Pick one", "criteria": [f"option {i}" for i in range(k)]}}
rows[f"{k} options"] = median_ms(q)
print(f" {f'1 choice question, {k:2d} options':34s} {rows[f'{k} options'][0]:16.1f} {rows[f'{k} options'][1]:13d}")
print("\n 每个问题都被编码为它自己的 (state, question) 行，所以 16 个是/否问题大约花费
print(" 大约 16 行。一个选择题的所有选项共享一行及其 head
print(" 预算，所以 40 个选项的花费远低于 40 个是/否问题。设计规则：问一个
print(" 带很多选项的选择题，而不是很多是/否问题。")
return (f"16 个是/否 {rows['16 yes/no'][0]:.0f} ms 对比一个 40 选项选择题
f"{rows['40 options'][0]:.0f} ms")
cost_model()
在基于 Laya 构建之前，我们先测量单条消息上一次前向传播的成本。每个问题都成为它自己的一行，与消息配对，所以十六个是/否问题大约要花一个问题的八倍时间。一个选择题的所有选项共享一行及其 head 预算，所以一个四十选项的选择题花费几乎只是一个三选项的两倍，而在我们的 CPU 上只有十六个是/否问题的四分之一。这给出了一条塑造后续一切的设计规则：问一个带很多选项的选择题，而不是很多是/否问题。
Copy CodeCopiedUse a different Browser
CARD = json.load(urllib.request.urlopen("https://huggingface.co/api/datasets/clinc/clinc_oos"))["cardData"]
PLUS = next(c for c in CARD["dataset_info"] if c["config_name"] == "plus")
NAMES = {int(k): v for k, v in PLUS["features"][1]["dtype"]["class_label"]["names"].items()}
def clinc(split):
df = pd.read_parquet(f"https://huggingface.co/api/datasets/clinc/clinc_oos/parquet/plus/{split}/0.parquet")
return df.assign(label=df.intent.map(NAMES))
# CLINC150 的银行领域（15 个意图），附带开发者会写的一行描述。
DESCRIBED = {
"balance": "查看账户里有多少钱",
"transactions": "查询账户上的近期交易",
"transfer": "在账户之间或向他人转账",
"freeze_account": "冻结或锁定账户",
"account_blocked": "被封锁或锁定、无法使用的账户",
"pay_bill": "支付账单",
"bill_balance": "账单上还欠多少钱",
"bill_due": "账单何时到期",
"interest_rate": "账户的利率",
"min_payment": "到期的最低还款额",
"order_checks": "订购新支票或支票本",
"pin_change": "更改 PIN 码"
"report_fraud": "举报欺诈或可疑活动"
"routing": "银行路由号码"
"spending_history": "某段时间或某个类别花了多少钱"
}
INTENTS = list(DESCRIBED)
ASK = "这是哪种银行业务请求？"
def route(states, criteria, **kw):
"""对每个 state 就 `criteria` 提出一个单选题；返回 choices、confidences、results。"""
res = agent.predict_batch(list(states), {"intent": {"type": "choice", "instructions": ASK, "criteria": criteria}},
batch_size=32, **kw)
return (np.array([r["answers"]["intent"]["choice"] for r in res]),
np.array([r["answers"]["intent"]["answer_confidence"] for r in res]), res)
@section("4. 真实标注数据：CLINC150 银行业务，零样本 vs. 训练过的分类器")
def zero_shot_vs_trained():
global train, val, test, BTRAIN, BVAL, BTEST
train, val, test = clinc("train"), clinc("validation"), clinc("test")
BTRAIN, BVAL, BTEST = (d[d.label.isin(INTENTS)].reset_index(drop=True) for d in (train, val, test))
print(f" CLINC150 'plus'：{len(train):,} / {len(val):,} / {len(test):,} 条训练 / 验证 / 测试查询，"
f" 150 个意图 + 范围外")
print(f" 银行领域：{len(INTENTS)} 个意图；{len(BTRAIN)} 条训练、{len(BVAL)} 条验证、{len(BTEST)} 条测试查询")
print(f" 例如 {BTEST.text[0]!r} -> {BTEST.label[0]}")
t0 = time.time()
pred, conf, _ = route(BTEST.text, DESCRIBED)
secs = time.time() - t0
acc = float((pred == BTEST.label).mean())
globals()["DESCRIBED_ACC"], globals()["DESCRIBED_PRED"] = acc, pred
print(f"\n Laya，零样本，15 个带描述的选项：准确率 {acc:.3f}"
f" （{len(BTEST)} 条查询在 {DEVICE} 上耗时 {secs:.0f}s）")
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
print("\n 每个意图用 k 条标注查询训练的 TF-IDF + 逻辑回归（k < 100 时抽 5 次）：")
curve = {}
for k in (1, 3, 10, 30, 100):
accs = []
for seed in range(5 if k < 100 else 1):
sub = BTRAIN.groupby("label", group_keys=False).sample(k, random_state=seed)
vec = TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True)
clf = LogisticRegression(max_iter=3000, C=10).fit(vec.fit_transform(sub.text), sub.label)
accs.append(float((clf.predict(vec.transform(BTEST.text)) == BTEST.label).mean()))
curve[k] = float(np.mean(accs))
print(f" k = {k:3d} ({k * len(INTENTS):5,d} labels) accuracy {curve[k]:.3f} (sd {np.std(accs):.3f})")
globals()["CURVE"] = curve
print("\n Zero-shot Laya, with nothing but the intent descriptions, lands between what a classic")
print(" classifier reaches with three and with ten labelled examples per intent. Step 5 shows the")
print(" descriptions are the weak part.")
return f"zero-shot {acc:.3f}; TF-IDF needs 10/intent for {curve[10]:.3f}"
zero_shot_vs_trained()
对于真实的标注数据，我们使用 CLINC150，这是一个公开的意图分类基准测试，涵盖十个领域的 150 个意图，外加一组超出范围的查询，直接从 Hugging Face Hub 以 parquet 文件读取。我们取其 banking 领域，包含十五个意图，每个意图有 100 条训练、20 条验证和 30 条测试查询，并让 Laya 以零样本方式路由这 450 条测试查询，为它提供每个意图的名称以及一行开发者会写的那种描述。它在没有任何标注示例的情况下达到 0.804 的准确率。作为参照，一个 TF-IDF 加逻辑回归分类器在每个意图三条标注查询时达到 0.651，十条时达到 0.848，三十条时达到 0.904。
Copy CodeCopiedUse a different Browser
@section("5. Criteria wording and option order change the answers")
def wording_and_order():
t0 = time.time()
pred_n, conf_n, res_n = route(BTEST.text, INTENTS)
secs = time.time() - t0
acc_n = float((pred_n == BTEST.label).mean())
pred_r, _, _ = route(BTEST.text, INTENTS[::-1])
acc_r = float((pred_r == BTEST.label).mean())
flips = float((pred_r != pred_n).mean())
print(f" {'criteria':44s} {'accuracy':>8s}")
print(f" {'15 names with one-line descriptions (step 4)':44s} {DESCRIBED_ACC:8.3f}")
tok = {name: agent.predict(BTEST.text[0], {"intent": {"type": "choice", "instructions": ASK, "criteria": c}})
["usage"]["input_tokens"] for name, c in (("described", DESCRIBED), ("names", INTENTS))}
print(f" {'15 bare intent names':44s} {acc_n:8.3f} ({secs:.0f}s; {tok['names']} input tokens per query"
f" vs {tok['described']})")
print(f" {'15 bare intent names, order reversed':44s} {acc_r:8.3f}")
print(f"\n Reversing the option order changes {flips:.1%} of individual answers, even where the")
print(" overall accuracy barely moves: the model has a position prior, so fix the order you deploy.")
changed = pd.DataFrame({"label": BTEST.label, "described": DESCRIBED_PRED, "names": pred_n})
gained = changed[(changed.names == changed.label) & (changed.described != changed.label)]
lost = changed[(changed.names != changed.label) & (changed.described == changed.label)]
print(f"\n Bare names fixed {len(gained)} answers the descriptions got wrong and broke {len(lost)};"
f" the most common fixes:")
for (lab, was), n in gained.groupby(["label", "described"]).size().sort_values(ascending=False).head(3).items():
print(f" {lab:16s} had been routed to {was:16s} x{n}")
print("\n More text is not more signal. The descriptions we wrote blurred intents the names keep")
print(" apart, and only labelled data could tell us. From here on we route on the bare names.")
globals().update(PRED=pred_n, CONF=conf_n)
return f"descriptions {DESCRIBED_ACC:.3f} -> names {acc_n:.3f}; reversed order flips {flips:.1%}"
wording_and_order()
Next we change only the wording of the options. Giving Laya the fifteen bare intent names, without our descriptions, lifts accuracy from 0.804 to 0.878 and halves the time, because the options take less than half the tokens. The descriptions blurred intents the names keep apart: account_blocked was routed to freeze_account ten times, and interest rate questions to balance. Reversing the order of the bare names changes 4.2 percent of individual answers even though overall accuracy barely moves, a sign of a position prior, so the option order you deploy should be the order you tested. Only labeled data could tell us either thing; from here on we route on the bare names.
Copy CodeCopiedUse a different Browser
def reliability(conf, correct, title):
bins = np.linspace(0, 1, 11)
idx = np.clip(np.digitize(conf, bins) - 1, 0, 9)
print(f" {title}\n {'confidence':>12s} {'answers':>8s} {'mean conf':>10s} {'accuracy':>9s}")
for i in range(10):
m = idx == i
if m.sum():
print(f" {bins[i]:5.1f}-{bins[i + 1]:.1f} {m.sum():8d} {conf[m].mean():10.3f} {correct[m].mean():9.3f}")
@section("6. 出厂时的概率有多诚实？")
def shipped_calibration():
ok = PRED == BTEST.label.values
ece = laya.ece_score(CONF, ok)
bucket = temp_bucket(laya.QTYPES["choice"], len(INTENTS))
print(f" 15 选项问题 -> 温度桶 {bucket!r}, T = {agent.temperature_by_options.get(bucket):.2f}\n")
reliability(CONF, ok, "450 条测试查询上的可靠性：")
print(f"\n 准确率 {ok.mean():.3f}, 平均 answer_confidence {CONF.mean():.3f}, ECE {ece:.3f} (15 个分箱)")
top = CONF >= 0.9
print(f" {top.mean():.0%} 的答案声称置信度 >= 0.9；其中 {ok[top].mean():.1%} 是正确的。")
print("\n 模型在这里过度自信，而截断正是原因之一：当 T = 0.5 时，每个")
print(" 15 选项答案在你看到之前就被锐化了。模型卡里的‘已校准’描述的是一种")
print(" 训练目标，而不是你的问题的属性。要在你自己的标签上测量它。")
globals()["ECE_SHIPPED"] = ece
return f"出厂时 ECE {ece:.3f}；准确率 {ok.mean():.3f}，平均置信度 {CONF.mean():.3f}"
shipped_calibration()
接着我们问这些概率有多诚实。一个十五选项问题落入该检查点的 choice:11+ 温度桶，也就是被截断到 0.5 的那个。450 条测试查询上的可靠性表显示了结果：92% 的答案声称置信度至少为 0.9，但其中 91.1% 是正确的，而 0.974 的平均置信度远高于 0.878 的准确率，期望校准误差为 0.102。Laya 的训练目标使用恰当评分规则，这就是模型卡所说的已校准。尽管如此，校准是问题在某个分布上的属性，你必须在你自己的标签上测量它。
Copy CodeCopiedUse a different Browser
@section("7. 在验证数据上拟合温度，同时不破坏其他问题")
def fit_temperature():
eye = np.eye(len(INTENTS))
pairs = [(s, {"intent": {"type": "choice", "instructions": ASK, "criteria": INTENTS}},
{"intent": eye[INTENTS.index(l)]}) for s, l in zip(BVAL.text, BVAL.label)]
t0 = time.time()
records = records_from_labeled(agent, pairs)
print(f" {len(records)} labelled validation records (raw logits + one-hot targets) in {time.time() - t0:.0f}s")
fit = agent.fit_temperatures(records)
after = (list(agent.temperature), dict(agent.temperature_by_options))
print(f" fitted on bucket counts {fit['n_by_bucket']}; choice temperature {fit['temperature'][0]:.3f}")
print("\n What agent.fit_temperatures() just installed, next to what shipped:")
print(f" {'':14s} {'shipped':>8s} {'after':>8s}")
for qt, name in enumerate(["choice", "score", "noul"]):
print(f" {name + ' (type)':14s} {SHIPPED[0][qt]:8.3f} {after[0][qt]:8.3f}")
for bucket in sorted(SHIPPED[1]):
now = after[1].get(bucket)
print(f" {bucket:14s} {SHIPPED[1][bucket]:8.3f} {('%.3f' % now) if now is not None else ' (gone)':>8s}")
print("\n One fit on choice questions replaced the whole map: every per-bucket entry is gone (a bucket")
print(" needs 2,000 records to keep its own temperature) and the score and noul temperatures were reset")
print(" to 1.0 because there were no records of those types. Every yes/no question now uses a")
print(" different temperature than it did a minute ago. Install only what you measured instead:")
agent.temperature, agent.temperature_by_options = list(SHIPPED[0]), dict(SHIPPED[1])
agent.temperature_by_options["choice:11+"] = fit["temperature"][0]
print(f" agent.temperature_by_options['choice:11+'] = {fit['temperature'][0]:.3f} (all else as shipped)")
pred, conf, _ = route(BTEST.text, INTENTS)
ok = pred == BTEST.label.values
ece = laya.ece_score(conf, ok)
print()
reliability(conf, ok, "Reliability on the same 450 test queries, after the fit:")
print(f"\n ECE {ECE_SHIPPED:.3f} -> {ece:.3f}; accuracy unchanged at {ok.mean():.3f} (temperature never changes the argmax)")
agent.save_calibration("laya_banking_calibration.json")
saved = json.load(open("laya_banking_calibration.json"))
print(f" saved with agent.save_calibration(): keys {sorted(saved)}; reload with laya.load(..., calibration=path)")
globals().update(RECORDS=records, CONF_FIT=conf, OK_FIT=ok)
return f"ECE {ECE_SHIPPED:.3f} -> {ece:.3f} from {len(records)} validation queries"
fit_temperature()
Laya 的校准模块将带标签的样本转换为原始 logits 和目标值的记录，并为其拟合一个温度。我们从验证集划分中构建 300 条记录，并调用 agent.fit_temperatures，它拟合出一个选择温度为 1.258 并将其安装，然后打印完整的温度表以及实际发布的值。这次拟合替换了整个映射：每个按选项数量划分的条目都消失了，因为一个桶需要 2,000 条记录才能保留自己的温度，而 score 和 yes/no 温度被重置为 1.0，因为没有这些类型的记录。一次针对选择题的拟合悄然改变了智能体中每个 yes/no 问题的校准方式。因此我们恢复发布的值，只安装我们测量过的那个桶。在测试集上，校准误差从 0.102 降至 0.059，准确率不变，因为温度从不改变哪个选项胜出，而 save_calibration 将结果写入一个 JSON 文件，laya.load 可以读回。
复制代码已复制使用其他浏览器
@section("8. 拟合到错误预算的弃权门控")
def abstention_gate():
t = agent.temperature_by_options["choice:11+"]
z = np.array([r[1] for r in RECORDS]) / t
p_val = np.exp(z - z.max(1, keepdims=True)); p_val /= p_val.sum(1, keepdims=True)
conf_val = p_val.max(1)
ok_val = np.array([np.argmax(r[1]) == np.argmax(r[2]) for r in RECORDS])
print(f" {'target':>6s} {'threshold':>10s} {'validation: kept / error':>25s} {'test: kept / error':>19s}")
gates = {}
for target in (0.02, 0.05, 0.10):
thr = laya.fit_abstention_thresholds(RECORDS, agent.temperature, agent.temperature_by_options,
target_error=target)
cut = thr["choice:11+"]
kv, kt = conf_val >= cut, CONF_FIT >= cut
gates[target] = thr
print(f" {target:6.0%} {cut:10.3f} {kv.mean():14.1%} / {1 - ok_val[kv].mean():5.1%}"
f" {kt.mean():11.1%} / {1 - OK_FIT[kt].mean():5.1%}")
_, _, res = route(BTEST.text, INTENTS, min_confidence=gates[0.05])
states = pd.Series([r["answers"]["intent"]["abstention"] for r in res]).value_counts().to_dict()
print(f"\n The 5% gate applied by Laya itself, predict_batch(min_confidence=...): {states}")
conf, ok = CONF_FIT.tolist(), OK_FIT.tolist()
print(f" 测试集上的选择性准确率：按置信度前 50% {selective_accuracy(conf, ok, 0.5):.3f}，"
f" 前 80% {selective_accuracy(conf, ok, 0.8):.3f}，全部 {np.mean(ok):.3f}；AURC {aurc(conf, ok):.3f}")
print(f"\n 在验证集上，每个切点都按构造满足其目标。在测试集上，实际误差更高，"
print(f" 因为测试查询更难：准确率在验证集上为 {ok_val.mean():.3f}，而在"
f" 测试集上为 {OK_FIT.mean():.3f}")
print(" 测试集上。拟合到某个误差预算的门控只对看起来像其拟合数据那样的流量有效。")
print(" 要在真实流量的样本上拟合它，随着流量漂移重新检查它，并留出")
print(" 余量。阈值按选项数量分桶，因为一个数字无法在")
print(" 二选一和十五选一的问题之间迁移。")
globals()["GATE"] = gates[0.05]
kept = CONF_FIT >= gates[0.05]["choice:11+"]
return f"5% 目标：在 {kept.mean():.1%} 覆盖率下，测试集误差为 {1 - OK_FIT[kept].mean():.1%}"
abstention_gate()
laya.fit_abstention_thresholds 接收相同的验证记录，并对每个选项数量分桶返回一个最宽松的置信度切点，使验证误差保持在目标之内。对于 5% 的目标，它选择 0.602，这使 95.7% 的验证查询保持在 4.5% 的误差，而 Laya 在将其作为 min_confidence 传给 predict_batch 时也会自行应用相同的切点，将 450 个测试答案中的 35 个标记为弃权。然而在测试集上，该门控使 92.2% 的查询保持在 9.2% 的误差，几乎是预算的两倍，而 2% 的目标实际达到 5.3%。数据解释了这一点：Laya 在 92.7% 的验证查询上是正确的，但只在 87.8% 的测试查询上正确，因此在一个样本上拟合的误差预算只对看起来像它的流量有效。排序本身是可靠的，最有信心的那一半测试答案有 97.8% 正确，但误差目标需要余量，并需要在真实流量上定期重新拟合。阈值保持按选项数量分桶，因为一个数字无法在二选一和十五选一的问题之间迁移。
复制代码已复制使用其他浏览器
@section("9. 范围外流量：门控 vs. 显式的 'other' 选项")
def out_of_scope():
global OOS, OTHER
OOS = test[test.label == "oos"].sample(150, random_state=0).reset_index(drop=True)
OTHER = test[~test.label.isin(INTENTS + ["oos"])].sample(150, random_state=0).reset_index(drop=True)
print(f" {len(OOS)} out-of-scope queries (e.g. {OOS.text[0]!r})")
print(f" {len(OTHER)} in-scope queries from other CLINC domains (e.g. {OTHER.text[0]!r})\n")
groups = {"banking": BTEST.text, "other domains": OTHER.text, "out of scope": OOS.text}
conf = {g: route(s, INTENTS)[1] for g, s in groups.items()}
thr = GATE["choice:11+"]
print(f" A) 15 intents + the 5% gate (threshold {thr:.3f}): share of each group the gate stops")
for g in groups:
print(f" {g:14s} mean confidence {conf[g].mean():.3f} abstained {(conf[g] < thr).mean():6.1%}")
with_other = INTENTS + ["not a banking request"]
print(f"\n B) 16 options: the 15 intents + 'not a banking request', no gate")
picks = {g: route(s, with_other)[0] for g, s in groups.items()}
for g in groups:
print(f" {g:14s} routed to 'not a banking request' {(picks[g] == 'not a banking request').mean():6.1%}")
acc_b = float((picks["banking"] == BTEST.label.values).mean())
print(f" banking accuracy with the extra option {acc_b:.3f} (15 intents alone: {OK_FIT.mean():.3f})")
print("\n Neither tool is free. The gate needs labelled data and gives up some in-scope coverage;")
print(" the extra option needs no data but changes the question every intent is scored against.")
stop = (conf["out of scope"] < thr).mean()
return f"gate stops {stop:.1%} of out-of-scope queries; 'other' option catches {(picks['out of scope'] == 'not a banking request').mean():.1%}"
out_of_scope()
生产流量包含路由器从未设计处理的请求，因此我们加入了 150 条超出范围的 CLINC 查询和 150 条来自其他 CLINC 领域的查询。校准后的置信度将它们清晰区分：银行查询平均为 0.912，其他查询约为 0.25，而 5% 的门槛拦截了 89.3% 的其他领域查询和 93.3% 的超出范围查询，同时对 7.8% 的银行查询选择弃答。另一种方案无需标注数据：第十六个选项「不是银行请求」能捕获 80.0% 和 90.0% 的此类查询，代价是将 2.0% 的银行查询路由出去，并使银行准确率从 0.878 降至 0.864，因为新选项改变了每个意图的评分基准。
复制代码已复制使用其他浏览器
IN_SCOPE = {"in_scope": {"type": "noul", "instructions": "这是关于用户银行账户、账单或支付的请求吗？"}}
@section("10. 一个有偏见的 yes/no 问题，以及为什么 temperature 无法解决它")
def biased_noul():
states = pd.concat([BTEST.text, OTHER.text, OOS.text], ignore_index=True)
y = np.r_[np.ones(len(BTEST)), np.zeros(len(OTHER) + len(OOS))].astype(bool)
res = agent.predict_batch(list(states), IN_SCOPE, batch_size=32)
p = np.array([r["answers"]["in_scope"]["noul"] for r in res])
from sklearn.metrics import roc_auc_score
auc = roc_auc_score(y, p)
print(f" '这是关于用户银行账户、账单或支付的请求吗？' 在 450 条银行")
print(f" 查询和 300 条非银行查询上：")
print(f" 平均 P(yes)：银行 {p[y].mean():.3f}，非银行 {p[~y].mean():.3f} AUROC {auc:.3f}")
print(f" 在 0.5 阈值处：召回率 {(p[y] >= 0.5).mean():.1%}，特异度 {(p[~y] < 0.5).mean():.1%}")
vstates = pd.concat([BVAL.text, val[~val.label.isin(INTENTS + ["oos"])].sample(150, random_state=0).text,
val[val.label == "oos"].text], ignore_index=True)
vy = np.r_[np.ones(len(BVAL)), np.zeros(len(vstates) - len(BVAL))]
recs = records_from_labeled(agent, [(s, IN_SCOPE, {"in_scope": np.array([1 - t, t])}) for s, t in zip(vstates, vy)])
t_fit = laya.fit_temperature_map(recs)["temperature"][2]
print(f"\n 在 {len(recs)} 条验证答案上拟合的 Temperature：T = {t_fit:.2f}（上限为 5.0）")
print(" 任何 T 都不会改变 0.5 这个切点：将两个 logit 除以 T 永远不会改变哪个更大。")
t_ship = agent.temperature_by_options.get("noul:2", agent.temperature[2])
logits = np.array([r[1] for r in recs])
p_val = 1 / (1 + np.exp(-(logits[:, 1] - logits[:, 0]) / t_ship))
cuts = np.linspace(0.01, 0.99, 99)
bal = [((p_val[vy == 1] >= c).mean() + (p_val[vy == 0] < c).mean()) / 2 for c in cuts]
cut = float(cuts[int(np.argmax(bal))])
print(f"\n 真正有效的是移动切点。在验证集上选定（最佳平衡准确率）：{cut:.2f}")
print(f" 在测试集上以 {cut:.2f} 为切点：召回率 {(p[y] >= cut).mean():.1%}，特异度 {(p[~y] < cut).mean():.1%}")
print("\n 这个问题的排序很好，但偏向于 'no'。温度缩放修复的是尺度，")
print(" 而不是偏移。把 P(yes) 当作一个分数，像对待其他分数一样在标注数据上选取它的切点。")
globals()["IN_SCOPE_CUT"] = cut
return f"0.5 处的召回率 {(p[y] >= 0.5).mean():.1%} -> 在经验证的切点 {cut:.2f} 处 {(p[y] >= cut).mean():.1%}"
biased_noul()
一个专门的 yes/no 问题看起来是自然的范围内检查，所以我们询问每条消息是否与用户的银行账户、账单或支付有关。它排序很好，AUROC 为 0.945，但它偏向于 no：银行相关查询的平均概率只有 0.361，所以在默认的 0.5 切点下它只能识别出其中的 28.9%。在 550 条验证答案上拟合温度会跑到上限 5.0，并且在该切点下没有任何改变，因为将两个 logit 除以任何温度都不会改变哪个更大。温度缩放修复的是尺度，而不是偏移。真正有效的是把概率当作一个分数，并在标注数据上选择它的切点：在验证集上选出的 0.09 切点在测试集上给出 92.0% 的召回率和 83.0% 的特异度。
复制代码已复制使用其他浏览器
from typing import Literal, Optional
from pydantic import BaseModel, Field
class BankingRequest(BaseModel):
intent: Optional[Literal[tuple(INTENTS)]] = Field(description=ASK)
in_scope: bool = Field(description=IN_SCOPE["in_scope"]["instructions"])
@section("11. 来自 pydantic schema 的类型化决策")
def schema_decisions():
planned = laya.structured.questions_from_pydantic(BankingRequest)
crit = planned["intent"]["criteria"]
print(f" questions_from_pydantic(BankingRequest): 'intent' -> {planned['intent']['type']} with "
f"{len(crit)} options, criteria values {set(crit.values())}; 'in_scope' -> {planned['in_scope']['type']}")
print(" A Literal becomes a choice over bare names (what step 5 found works best here); a bool becomes")
print(" a yes/no question that the projection cuts at 0.5.\n")
demo = pd.concat([BTEST.iloc[[0, 120, 300]], OOS.iloc[[0, 1]]], ignore_index=True)
out = laya.decide_batch(agent, list(demo.text), schema=BankingRequest, return_details=True,
min_confidence=GATE, batch_size=8)
for (text, label), d in zip(zip(demo.text, demo.label), out):
typed = BankingRequest(**d.values)
p_yes = d.probabilities["in_scope"]["true"]
print(f" {text[:46]!r:50s} truth {label}")
print(f" -> {typed!r} P(in_scope) {p_yes:.2f}, at our cut {p_yes >= IN_SCOPE_CUT}")
print("\n min_confidence=GATE turns a gated answer into None, so 'intent=None' means 'ask a human',")
print(" and the pydantic model still validates. in_scope is decided at a fixed 0.5 inside decide();")
print(" read d.probabilities['in_scope']['true'] and apply the cut from step 10 instead.")
nones = sum(BankingRequest(**d.values).intent is None for d in out)
return f"{len(out)} typed objects, {nones} intents gated to None"
schema_decisions()
最后，我们通过一个 pydantic schema 把它接入应用代码。laya.decide_batch 会把一个 Literal 字段变成一个针对裸名称的选择题，第 5 步表明这是此处更好的措辞；并把一个 bool 变成一个 yes/no 问题，然后把答案投影回一个经过验证的模型实例。把第 8 步的 gate 作为 min_confidence 传入，会把不确定的 intent 变成 None，而 Optional 字段接受它，因此 None 就成为一个明确的“询问人类”信号；两个超出范围的查询返回时 intent=None。然而，布尔值在 decide 内部以固定的 0.5 进行切分，所以一条欺诈报告在概率为 0.29 时被标记为 in_scope=False。从结果详情中读取概率并应用第 10 步的切分，就能得到正确答案。
Copy CodeCopiedUse a different Browser
banner("SUMMARY")
for name, res in RESULTS.items():
print(f" {name:<76s} {res}")
print("""
What to carry over
- Pin the checkpoint (laya.PINNED_REVISIONS) and know its shipped temperatures before trusting a
probability: 15-option questions were sharpened by a clamped 0.5.
- Measure criteria wording and option order on labeled data. Bare names beat our descriptions.
- Fit temperatures on held-out data, and install only the bucket you measured: fit_temperatures()
replaces the whole map, including question types you did not fit.
- Gate per option-count bucket with fit_abstention_thresholds, and leave margin below the target.
- A yes/no question can rank well and still be biased; choose its cut on labeled data.
Where to go next
- Fine-tuning: laya.train on the repository's main branch (not yet in the 0.3.27 wheel) and the
repo's Kaggle / Apple-silicon notebooks train with RLCD and refit temperatures.
- Other languages: laya.Router() detects the script and routes to laya-multilingual, which ships with
no fitted temperatures at all.
- Serving: pip install "laya[serve]" for an HTTP server; laya[mcp] for an MCP tool server.
- Docs: nandhakishorm.github.io/laya Code: github.com/NandhaKishorM/laya
""")
摘要会打印每个步骤的单行结果，以及值得沿用的习惯：固定 checkpoint 并阅读其随附的温度，在标注数据上测试标准措辞和选项顺序，在留出数据上拟合温度并只安装你测量过的那个分桶，按选项数量分桶设置带余量的阈值，并在标注数据上选择 yes/no 的切分点。最后以下一步方向收尾：微调，它位于仓库主分支的 laya.train 中，但尚未进入 0.3.27 wheel；laya.Router 背后的多语言 checkpoint；以及服务化。
In conclusion, Laya delivers most of what it promises: one forward pass answers several typed questions with no generated tokens, the bare-name router reached 0.878 on fifteen real banking intents with no training data, which a TF-IDF classifier needs between ten and thirty labeled examples per intent to match, and its calibrated confidence separated in-scope from out-of-scope traffic well enough to stop more than nine in ten out-of-scope queries. What the probabilities are worth, though, depends on work the library leaves to you, and several of its defaults point the wrong way. The shipped temperature for questions with eleven or more options sharpens rather than softens, the one-line calibration call wipes temperatures for question types it never saw, an error budget fitted on validation data held only for traffic that looked like it, and a yes/no question can rank well yet sit on the wrong side of 0.5, which no temperature can fix and which the schema projection then hard-codes. Each of these has a few-line fix shown above, and each would otherwise pass silently. The practical lesson is the same one this library’s own model card states and its defaults make easy to forget: a calibrated decision model is one you have calibrated, on your own labels, for your own questions.
Check out the FULL CODES here. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
[Sponsored] The web is the one API most agents are missing. Databases, calendars and repos have APIs. The open web mostly doesn’t. The TinyFish MCP server gives any MCP client four tools: TinySearch, TinyFetch (full pages as markdown, JavaScript included), TinyBrowser for logins and forms, and TinyAgent for multi-step jobs. Search and Fetch are free.
The post A Developer’s Guide to Laya: Zero-Shot Decisions and Calibration appeared first on MarkTechPost.