Target: rust-lang/rust
Issue: rust-lang/rust#157047
Status: ready-for-review PR opened
PR: rust-lang/rust#157724
Branch: https://github.com/scarab-systems/rust/tree/scarab-systems/rust-157047-layout-wf-check
Patch commit: 66b7156c — check raw pointer pointee layout during codegen
This field test targeted a Rust compiler regression where infinitely recursive nested structs started compiling successfully in release mode.
The issue shape was narrow but serious. A type could expand into an infinitely nested structure, but because that type was only reached through a raw pointer, the optimized/codegen path could avoid forcing the same pointee layout validation that would normally reject the recursive layout. The result was that release mode could compile code that should have produced a layout overflow error.
That is a compiler boundary failure. Raw pointers should not imply dereferenceability, alignment, or safe access to the pointee. But preserving conservative raw pointer behavior should not let invalid infinitely recursive pointee layouts escape validation during optimized compilation.
Failure shape
The failing path involved raw pointer pointee metadata during optimized/codegen compilation.
A previous Rust change preserved conservative raw pointer metadata behavior. That part is important and should stay intact: asking about a raw pointer’s pointee should not automatically give the compiler stronger safety or alignment claims than the pointer actually carries.
The problem was that, in this regression shape, preserving that conservative metadata also allowed the compiler to avoid forcing the pointee layout query. For an infinitely recursive pointee type, that meant optimized builds could miss the layout overflow and continue compiling.
This was not a generic recursion-limit problem. It was a layout validation ownership problem.
The compiler needed to keep raw pointer metadata conservative while still forcing enough pointee layout validation to reject impossible recursive layouts in the codegen path.
Boundary
The boundary here is:
conservative raw pointer metadata
versus
pointee layout validation required by codegen
For ordinary raw pointer metadata, Rust should preserve the existing behavior. A raw pointer should not gain implied dereferenceability, alignment, or validity just because the compiler asks for pointee information during codegen.
But when the pointee type itself has an invalid infinitely recursive layout, the compiler still needs to force the pointee layout query. Otherwise, the optimized/codegen path can accidentally become less strict than the layout validation path.
The repair keeps the conservative raw pointer behavior intact and restores the missing validation step.
What changed
The patch updates Rust compiler layout handling in:
compiler/rustc_middle/src/ty/layout.rs
The change preserves conservative raw pointer PointeeInfo behavior for codegen metadata while also forcing the pointee layout query. In plain terms: the compiler still avoids making unsafe assumptions about raw pointers, but it no longer lets infinitely recursive pointee layouts slip through optimized builds.
A focused UI regression test was added in:
tests/ui/codegen/normalization-overflow/raw-ptr-recursive-layout-issue-157047.rs
with the matching .stderr output.
The regression covers the release/codegen path from the original issue.
Validation
The targeted regression UI test passed.
Validation passed:
./x.py test tests/ui/codegen/normalization-overflow/raw-ptr-recursive-layout-issue-157047.rs --stage 1 --force-rerun
./x.py test tidy
Additional local validation completed before PR:
codegen normalization-overflow directory test passed earlier
source diff whitespace check passed
private leakage scan found no SDS, Scarab, Codex, or local-path leakage
Field test result
This was a clean compiler-layout repair against a narrow raw pointer/codegen validation boundary.
The issue reduced to one rule:
Raw pointer metadata should stay conservative, but codegen should still force pointee layout validation when the pointee type itself may be invalid.
That distinction matters. The repair does not make raw pointers stronger than they are. It does not add new dereferenceability or alignment assumptions. It only restores deterministic layout validation for a recursive pointee shape that optimized builds were failing to reject.
This is the kind of bug that looks broad from the outside — “Rust release mode accepts impossible recursive structs” — but the actual repair surface is much smaller:
raw pointer pointee layout validation lost ownership in the optimized/codegen path.
The patch restores that ownership without weakening Rust’s conservative raw pointer metadata behavior.
Public claim
The correct claim for this field test is:
Scarab/SDS helped drive a bounded repair for rust-lang/rust#157047, where an infinitely recursive nested struct reachable through a raw pointer could compile successfully in release mode. The repair preserves conservative raw pointer metadata behavior while forcing pointee layout validation during codegen, and a focused UI regression test was added to cover the optimized/codegen path. The PR is open for Rust compiler review as rust-lang/rust#157724.
Disclosure: This field report was prepared with AI-assisted editing from my own field-test notes, patch summary, validation output, and repair record. The technical claims and final wording were reviewed before publication.
