CounterProof

independent adversarial code review

A new vendor is not necessarily a new seat

So the unit we count is not the model, the vendor or the brand. It is the family: the lineage of weights and training a model descends from.

Why the CounterProof panel counts model families, not logos.

Our review process is designed so that more than one AI model reads each finding before a human signs it. A fair question follows: which models should such a panel include, and why not more? We recently asked ourselves a specific version of it: why does our panel have no Japanese model? The answer has nothing to do with Japan. It comes from what a second reviewer is for.

What a second reader is for

A reviewer can catch errors the author did not see. A second reviewer adds value as an independent check when it catches errors the first one missed. Two readers trained on overlapping material, by similar methods, can share blind spots, so their agreement should not automatically be treated as independent confirmation.

So the unit we count is not the model, the vendor or the brand. It is the family: the lineage of weights and training a model descends from. Seven models from one family should not be counted as seven independent readers.

Two questions a new name can raise

Two releases in 2026 illustrate the provenance questions we ask before adding a seat. They concern the documentation our counting rule needs. Neither is a judgement of the product’s quality or an evaluation of its review performance.

What training lineage is documented? Rakuten AI 3.0 is a large, openly licensed model developed under Japan’s GENIAC programme and optimised for Japanese. Its published configuration on Hugging Face declares the DeepSeek V3 architecture, with layer count, hidden size and expert layout matching DeepSeek V3’s published configuration, and the repository is tagged DeepSeek-V3. A matching architecture does not show where a model’s weights came from, and the sources reviewed here do not establish Rakuten AI 3.0’s weight lineage. Under our rules, we would need documented training provenance before counting any model as a new family; even a confirmed lineage would tell us to measure its errors, not to assume them.

Who answered? Sakana AI’s Fugu describes itself as a model trained to call various LLMs in an agent pool. According to Sakana, it solves some requests directly and, for harder ones, coordinates a team of expert models drawn from several providers. For the base Fugu model, customers can opt specific providers or models out of the pool; the pools of Fugu Ultra and Fugu Max are fixed. Sakana reports strong coding results. As a candidate seat it raises a different question. When Fugu delegates, an answer may combine contributions from several models, and our counting rule needs to know which families contributed. Sakana’s product FAQ says users cannot see which underlying models Fugu used for each query: that routing information is proprietary and “not exposed by design.” That is a reasonable product choice. It also means that, under our rules, we would not count Fugu as an additional identified family, whatever its quality as a tool.

What would earn a seat

We would consider a model, Japanese or otherwise, that meets three conditions:

  1. Documented training provenance. Evidence of a base lineage distinct from the families we already count. Architecture and configuration alone are not treated as proof of lineage either way.
  2. Competence on the work. It reviews security-relevant code and technical prose in English at a useful level. Language specialisation is valuable, but it is not what our reviews need.
  3. An attributable record. Each answer can be traced to an identified model and version, so its catches and its misses can be counted against it.

What we do not claim

Counting families is one screening rule in our process, not proof of independent errors. Separately trained families can still share blind spots. A family count should not be presented as a count of statistically independent reviewers, and we aim not to present ours that way. Our process is designed to compare which errors each seat catches and misses across review rounds, and to avoid treating agreement alone as independent confirmation.

We also prefer not to publish our panel’s composition, because a list of seats could help someone look for shared blind spots to aim at.

The question worth asking of any multi-model review, ours included, is not only “how many models?” but “how many different ways of being wrong?”

Sources (checked 16 Oct 2026):

  • Rakuten AI 3.0: huggingface.co/Rakuten/RakutenAI-3.0 (config.json: "model_type": "deepseek_v3", DeepseekV3ForCausalLM, hidden_size 7168, num_hidden_layers 61, 256 routed experts; tags in README.md); Rakuten announcement, 17 March 2026, global.rakuten.com/corp/news/press/2026/0317_01.html.
  • DeepSeek V3 configuration: huggingface.co/deepseek-ai/DeepSeek-V3 (config.json).
  • Sakana Fugu: sakana.ai/fugu/ (product page and FAQ, including Q9 on per-query model visibility and the pool controls per model tier); sakana.ai/fugu-release/; sakana.ai/fugu-beta/.
← All notes