The 30-Year-Old Technique That Makes AI-Generated Code Safe to Ship

When the Ariane 5 rocket exploded sixty-seven seconds after launch in 1996, the cause was a software bug: a 64-bit floating-point number was converted to a 16-bit integer without the overflow check that would have prevented it. The software had been verified thoroughly on Ariane 4, where the conversion never overflowed. On Ariane 5, a different flight profile pushed the value out of range. The test suite passed. The rocket did not.

The aviation and defense industries spent the next two decades obsessing over that failure mode. The lesson they landed on was this: testing specific examples is not enough. You have to prove that a property holds for all possible inputs, including the ones you have not thought of yet.

That lesson is now urgently relevant to software teams shipping AI-generated code at scale. The failure pattern is identical: the code looks correct, it passes unit tests, and then it blows up on an input nobody anticipated.

Property-based testing (PBT) is the technique aerospace engineers have relied on for decades. It is now the most practical answer to one of the defining reliability problems of the AI era.

The LLM Code Quality Crisis Is Real and Getting Worse

The data from 2024 to 2026 research is stark. Multiple independent studies of models including GitHub Copilot, InCoder, and ChatGPT find that between 36% and 74% of AI-generated code contains security vulnerabilities, spanning 42 different CWE categories, including entries from the MITRE Top-25 most dangerous software weaknesses. A separate quality assessment found that even the best-performing models produce fully correct code (valid, secure, reliable, and maintainable) only 65% of the time.

The failure modes are not random. LLMs are predictably bad at specific things:

• Boundary conditions: off-by-one errors, empty input handling, negative numbers, and null values. The model has seen thousands of examples with typical inputs; the edges are underrepresented in training data.
• Invariant violations: properties that must hold across an entire function, like "the sum of elements before filtering must equal the sum of elements after." A unit test that checks one specific case will not notice a subtle accounting error on unusual data.
• Implicit assumptions: requirements the developer did not write down. The model does not know that the output must be idempotent, or that the function will be called concurrently, or that input values can be negative.

The code review problem compounds this. LLMs generate code that looks right: syntactically clean, idiomatically written, and documented. Junior and senior developers alike report missing bugs in AI-generated code that they would have caught in colleague-written code, because the surface quality suppresses the vigilance that would otherwise kick in. When development velocity demands daily merges of generated code, manual inspection cannot scale.

One finding makes this worse: research shows that code ChatGPT revises in response to security feedback often exhibits more vulnerabilities than the original generation. The model pattern-matches on the feedback rather than reasoning from first principles about correctness.

What Property-Based Testing Actually Does

Traditional unit testing is example-based. You pick a specific input, you compute the expected output by hand or from a specification, and you assert they match.

def test_sort():
    assert sort([3, 1, 2]) == [1, 2, 3]

This test passes. It also passes for a sorting function that only works when the input contains exactly three elements, all positive and unique. You would need to write dozens of examples to gain any confidence, and you would still miss the combination you did not think of.

Property-based testing inverts this. Instead of picking inputs, you define what it means for the function to be correct, and a framework generates thousands of random inputs to try to disprove your claim.

from hypothesis import given
from hypothesis import strategies as st

@given(st.lists(st.integers()))
def test_sort_properties(xs):
    result = sort(xs)
    # Property 1: the output has the same length as the input
    assert len(result) == len(xs), "Sort lost or duplicated elements"
    # Property 2: the output is in non-decreasing order
    assert all(result[i] <= result[i+1] for i in range(len(result) - 1)), "Output is not sorted"
    # Property 3: the output contains the same elements as the input
    assert sorted(result) == sorted(xs), "Sort changed the contents"

Hypothesis (Python) or QuickCheck (Haskell) will run this against hundreds of generated inputs: empty lists, single-element lists, lists with duplicates, lists with large negative numbers, lists that are already sorted, lists sorted in reverse. If any input violates any property, the framework reports it and then shrinks the failing case to the smallest input that still triggers the bug. A test that fails on [0, -1, 0, 0, 2147483647, -2147483648] is hard to debug; the same failure on [-1, 0] is not.

This shrinking behavior is particularly valuable for AI-generated code, because the bugs are often at the boundary between values the developer imagined during prompting and values that actually occur in production.

What Aerospace Taught Us About This

Formal verification has been mandatory in safety-critical software development for decades. The DO-178C standard for avionics software, the MISRA guidelines for automotive code, and the Common Criteria for cryptographic systems all require some form of mathematical proof that certain properties cannot be violated, regardless of inputs.

The techniques used in these domains include model checking (exhaustive exploration of all possible system states) and theorem proving (mathematical derivation of correctness). These methods are expensive and require specialized expertise. A full formal verification of a commercial flight control system takes years and costs millions of dollars.

But the field learned something important through that experience: you do not need to formally verify everything. The most cost-effective approach is to identify the 20% of code that carries 80% of the risk (authentication logic, data validation, cryptographic operations, financial calculations, API contracts) and apply rigorous methods there. Everything else gets conventional tests.

This selective approach is what makes property-based testing tractable for ordinary software teams. You are not proving the entire codebase; you are formalizing the invariants that matter most and automating the search for violations.

The other lesson from critical systems is that formal testing pays for itself immediately. The cost of specifying properties for an authentication module is a few engineer-hours. The cost of a single CVE in production is measured in incident response, customer notification, regulatory fines, and reputational damage. Security auditors in 2026 are beginning to ask whether AI-generated code has been property-tested, and insurance underwriters are not far behind.

Why PBT Catches What Unit Tests Miss in LLM Code

A 2025 paper presented at the AIware conference directly compared property-based tests and example-based tests applied to LLM-generated code. The finding is not subtle: PBT consistently catches edge cases that example-based testing misses, and the gaps align exactly with the failure modes LLMs are known to exhibit.

Consider an LLM-generated authentication token validator. A unit test might check:

def test_expired_token_rejected():
    token = create_token(expiry=yesterday)
    assert is_valid(token) == False

This passes. But a property-based version encodes the actual invariant:

from hypothesis import given
from hypothesis.strategies import integers

@given(integers(min_value=0, max_value=int(time.time()) - 1))
def test_any_expired_token_is_rejected(past_timestamp):
    token = create_token(expiry=past_timestamp)
    assert is_valid(token) == False, f"Token with expiry {past_timestamp} was incorrectly accepted"

This version runs against hundreds of past timestamps, including edge cases the developer did not think to write: a token that expired one second ago, a token that expired at Unix epoch zero, a token with a negative timestamp. LLM-generated validators commonly fail on exactly these cases because the prompt said "check if the token is expired" without specifying every degenerate input.

The same principle applies across categories of LLM-specific failure:

• Sorting and filtering functions: properties like "length is preserved" and "no element is lost" catch the silent failures LLMs produce on empty or single-element inputs.
• Data transformation pipelines: "round-trip" properties (encode then decode returns the original) catch encoding bugs that only appear with certain byte patterns.
• Mathematical operations: "commutativity" and "associativity" properties catch floating-point handling errors in generated financial calculations.

The Tooling Landscape Is Ready

Two persistent objections to property-based testing have been that it requires expertise to write good properties and that the tooling is not mature enough for production use. Neither holds up in 2026.

Hypothesis (Python) has over 500,000 users and is actively maintained. Its @given decorator and strategies module cover most data types and support custom generators. It integrates with pytest and Django test suites without configuration. QuickCheck (Haskell) pioneered the approach and has ports in virtually every major language, including Go, Rust, Java, and Elixir.

Teralizer (2025) takes the adoption barrier down further. It automatically converts existing conventional unit tests into property-based tests by inferring the generalizable invariant from the specific example. A team with an existing unit test suite can generate an initial property suite without writing properties from scratch.

PBT-Bench (2025) is an emerging benchmark that measures how well AI agents themselves can write property-based tests. The benchmark signals that the field is treating PBT generation as a first-class capability, not an afterthought. A code-generation model's score increasingly includes its ability to produce testable, formally specified outputs.

On the generation side, research into "agentic property-based testing" is closing the loop: an LLM generates code, a second agent generates properties that the code should satisfy, and the test framework runs automatically. This pipeline can be integrated into CI/CD so that every AI-generated function ships with its property suite.

SandboxEval addresses the related problem of safely executing untrusted generated code during testing. Running property tests against AI-generated code that you have not manually reviewed requires an isolated execution environment; SandboxEval provides that infrastructure.

How to Start Without a Greenfield Rewrite

The practical path to property-based testing for AI-generated code does not require changing your entire development process. It requires identifying your highest-risk modules and adding properties incrementally.

A working sequence for most teams:

1. Identify your critical path. Authentication, data validation, API contracts, cryptographic operations, and financial calculations are the starting list. These are the modules where an LLM edge-case bug causes a breach or data loss, not just a minor UX issue.
2. Write three to five properties per critical function. You do not need exhaustive coverage. You need the invariants that define correctness: "output length equals input length," "output satisfies the input constraint," "function is idempotent," "no data is lost in transformation."
3. Run Hypothesis or QuickCheck in CI. The default configuration generates 100 examples per property. Increase this for your most critical modules; leave it at the default for lower-risk code.
4. Use Teralizer or LLM assistance to draft initial properties. A model like Claude can translate a natural-language specification into a Hypothesis @given test. Review the output before committing, but the drafting cost drops to minutes per function.
5. Treat a property failure as a specification problem, not just a code problem. When a property fails in an LLM-generated function, the first question is whether the property correctly captured the requirement. Sometimes the bug is in the test. More often, the test exposed an assumption in the prompt that the LLM could not infer.

The Window Is Narrowing

The adoption timeline matters. In 2024, property-based testing for AI-generated code was primarily a research technique. By 2025, cloud providers and developer-tooling vendors began exploring PBT integration in AI-assisted code review workflows. In 2026, security auditors are beginning to require it for code with documented LLM provenance.

Organizations that adopt now get two to three years of compounding advantage: their engineers develop intuition for writing good properties, their property suites grow alongside their codebases, and their defect rates drop in a measurable and attributable way. Organizations that wait until it is mandatory will spend that time retrofitting properties onto code that was never designed to be formally tested, while racing to close vulnerabilities that have already shipped.

Aerospace engineering spent thirty years learning that formal methods applied selectively to high-risk code is the only way to build systems you can actually trust. AI-generated code has compressed that lesson into two years of production incidents.

The technique is proven. The tooling is ready. The cost of not adopting it is now higher than the cost of learning it.

Conclusion

Property-based testing is not a new idea. It has been running in safety-critical systems since before most working engineers entered the field. What is new is the scale at which developers are now shipping code they did not write and cannot fully read, produced by models that are confidently wrong in specific, predictable ways.

The same principle that prevented avionics bugs in the 1990s applies directly: do not test examples, prove properties. Define what correct means before the code runs, and let the framework spend computational resources finding the edge case you forgot to consider.

For teams integrating LLM code generation into their development workflows, property-based testing is not a nice-to-have. It is the verification layer that makes autonomous code generation safe enough for production systems.