We compile and analyze a catalog of 1,043 distinct vulnerabilities found in LLM-generated code across Python, JavaScript, Go, and C, drawn from 56,200 generations across eight models. We classify vulnerabilities along Common Weakness Enumeration (CWE) lines and find a heavy concentration in CWE-78 (OS command injection), CWE-89 (SQL injection), and CWE-22 (path traversal), together accounting for 47.
We audit 2,318 LLM-generated patches drawn from public agent benchmarks and find that 28.6% fail to reproduce when re-run on a fresh container, even when the originating evaluation reported success.
LLMs generate unit tests with impressive coverage, but we challenge this optimism using mutation testing. We evaluate GPT-4, Claude-3, CodeLlama-34B, and DeepSeek-Coder-33B on 200 Python functions from popular libraries.
Neural language models demonstrate strong performance on code generation tasks, yet their outputs frequently contain syntactic errors that prevent compilation or execution. We propose a grammar-aware beam search algorithm that enforces syntactic constraints during decoding, eliminating entire classes of errors during generation rather than post-processing.
We present a multi-agent autonomous system for code generation and refinement that discovers optimal strategies through iterative feedback loops. Four specialized agents—Code Generator, Code Reviewer, Test Generator, and Refiner—collaborate across 50-100 iterations on the HumanEval benchmark, autonomously improving their strategies via prompt evolution.