GLM 5.2: A New Era in Security Research

AI & Machine LearningAI DiagnosticsOpen City Data

In a groundbreaking study, GLM 5.2 has been found to outperform Claude Code on IDOR detection, a significant breakthrough in security research.

TL;DR
• GLM 5.2 beats Claude Code on IDOR detection with a 39% F1 score.
• The harness plays a crucial role in the performance of the models.
• Open-weight models have crossed a threshold worth watching in security research.

Summary
Semgrep’s recent benchmark study compares open‑weight and frontier language models on IDOR detection, a common web‑app vulnerability. The study evaluates models under identical prompts, varying only the harness used, and measures precision, recall, F1, and cost per true positive.
• GLM‑5.2, an open‑weight Mixture‑of‑Experts model, achieved 39% F1 with a cost of $0.17 per vulnerability, outperforming Claude Code (32%) and other open‑weight models.
• Semgrep Multimodal pipelines, combining GPT‑5.5 or Opus 4.8 with endpoint discovery, topped the list with 61% and 53% F1, highlighting the importance of structured harnesses.
• The experiment shows that harness design can outweigh raw model capability, and that cost‑effective open‑weight models can match or exceed frontier agents on specific tasks.

The benchmark used a real IDOR dataset, consistent evaluation metrics, and a simple Pydantic AI harness for open‑weight models. GLM‑5.2’s large context window (1M tokens) and active parameter count (40B) enable efficient reasoning across multiple files, a key advantage for security analysis. Its open‑weight status allows on‑prem deployment, fine‑tuning, and inspection, addressing privacy and compliance concerns.

Semgrep’s findings suggest that organizations should not rely solely on expensive frontier models; instead, they should evaluate open‑weight alternatives and invest in robust harnesses that guide models to relevant code sections. This dual focus on model selection and harness architecture can reduce detection costs, improve precision, and accelerate secure development pipelines.

The implications of GLM 5.2 beating Claude Code on IDOR detection are significant, and it is essential to continue exploring the potential applications of open-weight models in security research.