Kimi K3 looks efficient on Moonshot's charts. Now I want a test it cannot grade itself

My first Kimi K3 post made a simple point: the benchmark is not the product.

That still stands.

Moonshot AI has now published more concrete evidence for Kimi K3, including score-vs-cost comparisons for coding and BrowseComp, plus internal knowledge-work results across online research, deck creation and finance tasks.

Those charts are more useful than a parameter headline. They show the trade-off Moonshot wants builders to notice: Kimi K3 appears competitive while using less cost per task than several larger proprietary alternatives.

That is worth testing.

It is not yet the test result I would use to choose a model for ChaseOS.

The vendor chart is the starting claim

Moonshot’s Kimi Code Bench V2 chart places Kimi K3 near the low-cost end of its comparison while reporting a score above several tested alternatives. Its BrowseComp chart makes a similar argument, with Kimi K3 positioned at a high reported score and a relatively low cost per task.

The internal knowledge-work chart reports Kimi K3 ahead on three selected categories:

Online Exp Bench;
DECK-Bench;
Finance-Bench.

These are Moonshot AI’s results from Moonshot AI’s evaluation setup. They are relevant first-party evidence, but they do not independently prove how Kimi K3 will behave inside my tools, task definitions, provider route or acceptance criteria.

The charts answer: “How did Kimi K3 perform in Moonshot’s harness?”

I need to answer: “Will Kimi K3 finish useful work inside mine?”

Cost per task is not cost per trusted result

A low provider bill can still produce an expensive workflow.

If a model chooses the wrong tool, repeats calls, loses state, misses a requirement or needs a human to repair the final artifact, the headline cost per task stops being the number that matters.

The useful unit is cost per trusted result.

That includes:

the model and provider cost;
retries;
tool calls;
validation;
recovery from failure;
human intervention;
the cost of incomplete or misleading work.

A model that looks cheaper on one benchmark may be more expensive in an operating system if it needs constant steering. A more expensive call may be better value if it completes the work cleanly and leaves a reliable proof package.

That is the comparison the next ChaseOS test should expose.

The independent workflow test

Kimi K3 and a model already used in the ChaseOS stack should receive the same bounded repository task.

Both models get:

the same repository snapshot;
the same prompt;
the same tool permissions;
the same context budget;
the same acceptance criteria;
the same validator;
the same evidence requirements.

The task should end in a real artifact, not a written claim that the work is complete.

The proof package should include the artifact, validation output, tool trace, retry count, unresolved failures and a clear statement of any human intervention.

No production credentials. No public posting rights. No uncontrolled account actions.

This is an operating test, not a risk demonstration.

The scorecard I would trust

I would compare:

Completed result - Did the final artifact meet every acceptance criterion?
Tool accuracy - Did the model choose and use the right tools?
Recovery - Did it notice failures and recover without making the situation worse?
Total cost - What did the complete successful run cost, including retries?
Latency - How long did it take to reach a validated result?
Human intervention - How much steering or repair was required?
Evidence quality - Could another operator verify what happened?

The winner is not the model with the most impressive paragraph.

The winner is the model that completes the work, proves the result and reaches that standard at a sensible total cost.

What would change my mind

If Kimi K3 reproduces the shape of Moonshot’s score-vs-cost story inside the ChaseOS harness, that is meaningful.

It would show that the model is not only strong inside the vendor’s selected evaluation, but also useful inside a provider-independent operating layer with controlled tools and explicit evidence requirements.

If it needs more retries, more repair or more human direction than the alternatives, the vendor chart remains interesting without becoming the buying decision.

That is not a criticism of the chart. It is the difference between first-party evidence and an independent operating result.

The builder takeaway

The earlier post asked whether Kimi K3 could navigate a repository, use tools, recover and finish useful work.

This follow-up turns that question into a test contract.

Moonshot’s charts have made Kimi K3 worth testing. ChaseOS should decide whether it is worth trusting.

Sources and claim boundary

All Kimi K3 benchmark, score, cost and knowledge-work figures discussed here are vendor-reported by Moonshot AI. ChaseInTech has not independently reproduced those results. This article proposes the next test and does not claim that the test has already run.

Follow the build

The full editorial analysis lives on ChaseInTech.com. ChaseOS.ai is the product destination for the governed agent operating system and harness described in the piece.

ChaseInTech.com