Hoi hoi! 👋
我是@nyaomaru,一名前端工程师,目前正在构建一个 AI 辅助的 CHANGELOG 生成器,并在尝试 Jev。😸
最近,我一直在思考一个问题。
我们如何提取这个
## 为什么
The previous implementation could leave stale release data after an update.
从一个 pull request 中提取出来并放入 CHANGELOG?
### 已修复
- Fix cache invalidation
Why: The previous implementation could leave stale release data after an update.
获取WHAT相对容易。
获取WHY则难得多。
而且我不希望 AI 只是编造一个看起来漂亮的理由。
让我们看看changelog-bot目前是如何处理这个问题的!👀
nyaomaru/changelog-bot
用 AI 自动创建 changelog。🤖 它提供 CLI 和 github actions。🚀 https://www.npmjs.com/package/@nyaomaru/changelog-bot
发布应该让人兴奋,而不是乏味。
@nyaomaru/changelog-bot🤖 在一次运行中把你的 Git 历史和发布说明变成一份精致的 changelog 条目(以及可选的 PR)。把它放进 CI、在本地运行,或者交给你的发布负责人——无论哪种方式,你都能带着一份清晰的 changelog 发布,零复制粘贴疲劳。
为什么选择 changelog-bot?
- 自动化叙事:结合提交历史、PR 标题和发布说明,生成适合人类阅读的 changelog 章节。
- LLM 超能力(可选):连接 OpenAI 或 Anthropic 以获得语气感知的摘要,或者完全跳过 API 密钥,依赖稳健的启发式回退方案。
- 可直接生成 PR 的输出:可以打开一个 pull request,其中已更新 changelog、比较链接和发布说明都已接好。
- 安全的默认设置:检测重复版本、保持比较链接最新,并且即使 AI 宕机也不会让发布失败。
- CI 原生:可作为 GitHub Action、可复用工作流或普通 CLI 使用——无需脆弱的脚本。
重要
这个项目目前处于早期阶段……
📝 WHAT 容易。WHY 难。
假设一个 PR 包含。
## 描述
Extract shared runtime configuration into ProviderBase.
Update OpenAI, Anthropic, and Gemini adapters to use the shared implementation.
This reduces duplicated setup logic and makes adding new providers easier.
前两句话主要描述WHAT和HOW。
但这一句不同
This reduces duplicated setup logic and makes adding new providers easier.
这更接近WHY。
我们可以简单地把整个 PR 交给 LLM 并问
Why was this change made?
但那样模型可能会总结、改写,或者推断出一个从未被明确写出的理由。
对于 CHANGELOG,我想要更严格的东西。
也许 WHY 提取不是生成问题。
也许它是一个选择问题。
✂️ 首先,找到可能的 WHY 候选
在调用模型之前,changelog-bot会预处理 PR 描述。
它寻找诸如以下信号
const STRONG_CANONICAL_SECTION_NAMES = new Set([
"why",
"reason",
"because",
"motivation",
"context",
"background",
"problem",
"rationale",
]);
const RATIONALE_MARKER_RE =
/\b(why|because|so that|in order to|reason|rationale|motivation|to avoid|to prevent|context|problem)\b|,\s*so\b/i;
它还会移除模板噪音,例如复选框、占位符和不相关字段。
然后它构建小的候选片段
const candidates = buildWhyCandidateSnippets(
extracted.sections,
body,
options.maxCharsPerPr,
!extracted.hasTargetSection,
);
所以,与其发送一大段 PR 正文,我们可能得到
[
"Extract shared runtime configuration into ProviderBase.",
"This reduces duplicated setup logic and makes adding new providers easier.",
];
现在问题小多了。
哪个候选才是真正的 WHY?
🧠 向 Jev 提两个问题
这就是我开始尝试Jev。
我不要求Jev写一个 WHY 句子。
相反,每个候选都会用两个问题来评估。
第一个
问题: "目标候选是否明确说明了为什么做出这个 changelog 变更?";
基本上
这真的是一个理由吗?
但仅凭这一点还不够。
想象一下
We changed the cache invalidation logic.
The README was updated because the previous explanation was confusing.
第二句话肯定包含一个理由。
但它解释的是 README 变更,而不是缓存失效变更。
所以我们还要问
问题: "目标候选是否陈述了一个适用于所识别 changelog 变更的理由?";
现在每个候选都有两个信号
{
probability: explicitWhyProbability,
relevanceProbability: changeRelevanceProbability,
}
所以真正的问题变成了
Is this a reason?
AND
Is it the reason for this change?
🎯 选择候选
changelog-bot比较候选的概率
const bestCandidate = candidateProbabilities.reduce((best, candidate) =>
combinedWhyCandidateProbability(candidate) >
combinedWhyCandidateProbability(best)
? candidate
: best,
);
然后两个最低阈值都必须通过
const accepted =
selectedCandidate !== undefined &&
bestCandidate.probability >= JEV_MIN_EXPLICIT_WHY_PROBABILITY &&
bestCandidate.relevanceProbability >= JEV_MIN_CHANGE_RELEVANCE_PROBABILITY;
如果候选没有通过
if(!accepted) continue;
不添加 WHY。
而且我认为这很重要。
有时正确答案是
There is no reliable WHY in this PR.
这比编造一个要好。
🔒 保留原始 PR 文本
这大概是我最喜欢的部分。
当候选被接受时
items.push({
prNumber: item.prNumber,
why: selectedCandidate,
confidence: mappedConfidence,
});
注意
why: selectedCandidate;
The final WHY is the original PR text.
Not newly generated prose.
So the pipeline becomes
PR description
↓
candidate extraction
↓
Jev evaluates candidates
↓
select candidate
↓
original source text
↓
CHANGELOG
Jevdecides.
The PR author still provides the sentence. 😸
🧱 Where Does WHY Fit in the v1 Pipeline?
While working on WHY extraction, I have also been redesigning thechangelog-botpipeline for v1.
The main idea isdeterministic-first.
A complete CHANGELOG should exist even when AI is disabled or fails.
So AI should enrich the result, not own the result.
The target v1 pipeline looks roughly like this
source changes
↓
ReleaseChange[]
↓
deterministic classification
↓
complete ReleaseDraft
↓
optional editorial enrichment
+
optional WHY enrichment
↓
ReleaseResult
↓
deterministic renderer
↓
CHANGELOG
WHY extraction is therefore an independent optional enrichment stage.
Conceptually
ReleaseDraft / ReleaseChange[]
↓
find eligible WHY targets
↓
collect PR rationale candidates
↓
LLM or Jev
↓
trust / confidence validation
↓
attach structured WHY notes
↓
ReleaseResult
One important change here is that the WHY engine doesnotneed to control the final Markdown.
Today, some of the pipeline still works by finding targets from generated Markdown and inserting WHY text afterward.
For v1, I want to remove that boundary.
Instead
{
prNumber,
why: selectedCandidate,
confidence,
}
becomes structured release data first.
Then one deterministic renderer decides how it appears
- Fix cache invalidation
- Why: The previous implementation could leave stale release data after an update.
So the responsibility becomes clearer
WHY engine
→ find the evidence
Renderer
→ format the evidence
This is also part of a larger rule I am using for the v1 migration
Deterministic code owns the CHANGELOG structure. AI only enriches it.
Phases 1–4 of this migration are already implemented, and the structured rendering and enrichment boundary is part of the next phase.
You can see the full v1 design here 👇
https://github.com/nyaomaru/changelog-bot/blob/main/v1-design.md
🧪 But What Threshold Should We Use?
Of course, once we have probabilities, another question appears.
0.50?
0.60?
0.80?
I didn't want to pick one only because it felt reasonable.
So I created an evaluation harness with an initial 14-case corpus.
For example
{
candidates: [
"Separate changelog execution into input-resolution and output-finalization phases.",
"This simplifies the main orchestration flow and runs independent GitHub lookups concurrently.",
],
expectedSelectedCandidateIndex: 1,
}
We also need negative examples
{
candidates: [
"Extract shared provider runtime configuration into ProviderBase.",
"Refactor OpenAI, Anthropic, and Gemini adapters to inherit it.",
],
expectedSelectedCandidateIndex: null,
}
There is plenty of implementation information.
But no strong explicit WHY.
Sonullis the correct result.
🤝 Then John Joined the Project
While I was working on this experiment,@johnnylemonny作为贡献者加入了项目。🎉
johnnylemonny (John) · GitHub
全栈开发者,打造快速、现代且精致的应用。我专注于简洁、性能以及对用户的真正价值。欢迎机会垂青。 - johnnylemonny
github.com
John 提议将小型基准测试扩展到大约 50 个案例,涵盖
Explicit Rationale
Implementation-Only
Template & Noise
Multilingual PRs
我喜欢这个想法,但我希望有一条规则保持清晰
基准真值应当独立于当前的 Jev 阈值。
因此基准测试首先定义候选是否真正有效。
然后我们再评估不同的阈值。
John 在 PR 中实现了扩展后的语料库#212。🙌
基准测试从 14 个案例增长到 50 个,包括棘手的负例。
他还添加了更精细的阈值扫描
[0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9];
现在我们可以比较精确率、召回率和 F1,而不是说
0.8 感觉挺安全!👍
让我们来测量它吧。😸
🧩 当前的流水线
如今,WHY 提取大致是这样的
Pull Request
↓
Normalize PR body
↓
Extract WHY candidates
↓
Remove noise
↓
Local trust check
↓
Jev:
explicit rationale?
relevant to this change?
↓
Apply threshold
↓
Candidate or none
↓
Original PR text
↓
CHANGELOG
对于小小的一行来说,步骤还挺多的Why:。😂
但正是这一点让这个问题变得有趣。
🎯 我学到了什么
当我开始做这个功能时,我以为主要问题是
How can AI write a good reason?
现在我觉得更有趣的问题是
How can we find evidence that the reason already exists?
所以我目前的方法很简单
Use deterministic code to find evidence.
Use Jev to evaluate the evidence.
Keep the original evidence as the output.
多亏了 John 的贡献,我们现在有了更好的方法来测试这个方法是否真的有效。
开源很有趣,当有人看着你的实验说:
如果我们好好测试一下会怎样?
然后和你一起把它构建出来。🙌
changelog-bot仍在演进,Jev 支持仍处于实验阶段。
我很好奇这个方法接下来会走向何方!
https://github.com/nyaomaru/changelog-bot
感谢阅读!
我们下篇文章见!👋😸

