In a fresh evaluation from the nonprofit research group METR, OpenAI's GPT-5.6 model has demonstrated that its long-task performance score is highly sensitive to the scoring rubric used. The findings, reported by eWeek, highlight the importance of transparent and consistent evaluation criteria in AI benchmarking.

The METR Test and Its Implications

METR, the Model Evaluation and Threat Research group, designed a test to assess AI models on long-horizon tasks—those requiring sustained reasoning and execution over extended periods. When GPT-5.6 was put through the test, its score varied significantly depending on how the scoring rules were applied.

This variability suggests that the way we measure AI capability can dramatically influence the results. For businesses and researchers relying on AI benchmarks to make decisions, understanding the underlying scoring methodology is crucial.

Why Scoring Rules Matter

Scoring rules in AI evaluations can determine whether a model receives credit for partial progress, how errors are penalized, and whether the model is allowed to ask for help. In the METR test, tweaking these parameters led to noticeably different long-task scores for GPT-5.6.

This finding aligns with broader concerns in the AI community about benchmark reliability. As models become more advanced, the nuances of evaluation design can have outsized effects on perceived performance.

What This Means for AI Development

For developers and enterprises, these results underscore the need to look beyond headline scores. A model that excels under one scoring regime might underperform under another, making it essential to align evaluation metrics with real-world deployment scenarios.

METR's work is particularly relevant as organizations increasingly rely on AI for complex, multi-step tasks. The ability to accurately measure progress on such tasks is key to ensuring safety and reliability.

Industry Reactions and Next Steps

OpenAI has not yet issued a public response to the METR findings. However, the broader AI community is likely to engage in discussions about standardizing evaluation protocols.

As AI models continue to evolve, the demand for rigorous, transparent testing will grow. METR's test serves as a reminder that the journey toward reliable AI is as much about the measuring stick as it is about the model itself.

Key Takeaways

  • Scoring sensitivity: GPT-5.6's long-task score varies with METR's scoring rules, highlighting the impact of evaluation design.
  • Benchmark reliability: AI benchmarks must be interpreted with care, as different scoring methods can yield different conclusions.
  • Practical implications: Businesses should consider real-world task requirements when evaluating AI models.
  • Call for standards: The AI industry may benefit from more standardized evaluation practices.