What SuperCLUE’s Split Leaders Mean for Model Evaluation

SuperCLUE’s November 2026 results offer a useful warning against selecting a language model from one aggregate score. Across the published table of 19 models, the lead changes with the task. GPT-6.1-Sol has the highest displayed scores for agentic coding and mathematical and scientific reasoning. Gemini-4-Argon leads hallucination control and ultra-long context, and shares the instruction-following lead with GPT. Among entries classified as open by SuperCLUE, DeepSeek-V4.1-Flash-1017 leads coding, Darwin-180B-RSI-R3 leads reasoning, and Kimi-K3.1-Pro leads the other three categories as well as the open-model overall comparison.

Those results are more useful as a map for building a shortlist than as a verdict. They show where to begin testing, but not which model will be best in a particular application.

Read category scores as hypotheses

A benchmark category compresses many tasks and evaluation decisions into one number. That number can suggest a capability worth investigating, but it does not explain how the model will behave under a different prompt format, language, tool setup, or latency budget. Even similarly named categories can differ substantially between benchmarks.

Consider coding. The published table gives GPT 69.23 and both Gemini and DeepSeek 65.38. DeepSeek’s result makes it an obvious candidate for an open-model coding trial. It does not establish that all three systems will solve the same repository-level bugs, use tools equally well, or recover from a failed test in the same way. A team evaluating a coding agent should therefore preserve the benchmark’s signal while replacing the benchmark task with representative work: dependency upgrades, localized bug fixes, test repair, and multi-file changes under identical tool permissions.

The same discipline applies to reasoning. Darwin’s 73.68 is the strongest open-model score in that category and second among all listed entries, behind GPT at 76.32. That is a meaningful shortlist signal. It is not evidence that mathematics, scientific question answering, and quantitative analysis are interchangeable strengths. A follow-up set should separate them and score final answers independently from the validity of intermediate reasoning.

Avoid converting scores into unsupported rates

The hallucination-control column is especially easy to misread. Gemini leads at 83.19, while Kimi is the open-model leader at 67.78. These values should not be treated as percentages of factual answers or inverted into error rates. Without the full task definition and scoring protocol, such arithmetic creates a metric the benchmark did not report.

A practical factuality evaluation needs an explicit answer taxonomy. For evidence-based question answering, label whether an answer is supported, contradicted, insufficiently supported, or correctly abstained. Then measure usefulness as well as caution. A model that refuses every difficult request may avoid unsupported statements while still failing the product’s purpose.

Instruction following needs a similarly concrete rubric. Gemini and GPT share the published lead at 68.97, while Kimi leads the open entries at 44.83. For a structured automation workflow, teams can test valid JSON, required-field completion, adherence to length limits, and the absence of unsolicited prose. Factual correctness should remain a separate measure: perfect formatting does not rescue a wrong answer.

Test long context as behavior, not capacity

Gemini’s 95.63 leads the ultra-long-context category, followed by GPT at 88.14 and Kimi at 87.25. The narrow difference between GPT and Kimi is a reason to compare both on the intended documents, not a reason to declare them operationally equivalent.

Maximum accepted context and reliable use of context are different properties. A model may accept a large input while missing evidence in the middle, confusing similarly named entities, or failing to combine facts across files. A useful trial varies document length, evidence position, irrelevant material, and the number of sources that must be connected. It should also record time to first token, total latency, memory demand, and cost at realistic traffic.

For retrieval-augmented systems, compare at least two conditions: the same curated evidence supplied to every model, and each model paired with the production retrieval pipeline. The first isolates answer behavior; the second measures the system users will actually experience.

Build a decision table before running models

A compact evaluation plan prevents an overall leaderboard position from silently determining the outcome. Start with the workflow’s failure modes and assign weights before seeing local results. A coding assistant might emphasize test-passing patches and safe tool use. A document service might emphasize citation support, correct abstention, and evidence retrieval. An extraction pipeline might prioritize schema compliance and stable throughput.

Next, fix the comparison conditions: exact model revision, inference engine, numerical precision, prompt, tool access, token budget, retry policy, and hardware. Record quality, latency, and cost together. If the application serves languages beyond the benchmark’s primary focus, construct native-language tasks rather than assuming results transfer. License and usage conditions also require separate inspection; SuperCLUE’s open and closed labels are its own classification, not a complete deployment review.

Finally, examine category interactions. A model strong in long-context work may still struggle with exact output constraints. A coding leader may be less suitable for document grounding. Aggregate scores can hide these mismatches because their weighting may not reflect the product’s priorities.

A shortlist, not a finish line

The November table supports several targeted starting points: DeepSeek for open coding comparisons, Darwin for open mathematical and scientific reasoning, and Kimi for open-model trials involving factuality, constraints, or long documents. Gemini and GPT provide strong closed-model comparison points across the displayed categories.

The durable conclusion is methodological. Use leaderboards to decide which candidates deserve an evaluation slot, then let representative tasks, controlled conditions, and explicit failure costs determine deployment. A split leaderboard is not inconvenient ambiguity; it is evidence that model choice should follow the workload.