Build directory (as a string) injected into _pydantic-core.*.so (which will cause non-reproducible builds)

[moto-timo]

While working on upgrading the Yocto Project recipe for pydantic-core to the latest version, I noticed that we are seeing build directory (TMPDIR) contamination in the _pydantic-core.*.so. This throws a QA warning for us, but would also lead to non-reproducible builds (because a different build directory would produce a different binary lib output).

On further inspection, it appears it might be a python backtrace related to the schema generation? (This happens on 2.19.0 and in this case 2.20.1 as well).

readelf -p 14 pydantic_core/_pydantic_core.cpython-312-x86_64-linux-gnu.so
pydantic-core-readelf-rodata.txt

Any ideas what could be causing this host information to be injected into the binary lib?

[davidhewitt]

From a quick read it looks like these might be panic messages which are created by the Rust standard library. See rust-lang/rust#40374

It looks like these are hard to prevent.

[moto-timo]

@davidhewitt I agree it looks like a panic message. It seems like it is related to schema validation?

Self SchemaError building self-schema:
  " validator:
  model-fieldsdataclasstuplefrozensetdefinition-refdefinitionsUnknown schema type: "2.20.1a-alphab-beta__version__profile=release pgo= -L /srv/build/ttorling/workspace-upgrades/build/tmp/work/core2-64-poky-linux/python3-pydantic-core/2.20.1/recipe-sysroot/usr/lib/rustlib/x86_64-poky-linux-gnu/lib --remap-path-prefix=/srv/build/ttorling/workspace-upgrades/build/tmp/work/core2-64-poky-linux/python3-pydantic-core/2.20.1=/usr/src/debug/python3-pydantic-core/2.20.1-Cprofile-use=args

What seems particularly strange is the 20.20.1a-alphab-beta substring, which so far I have only found in .github/check_version.py

Since it seems to be schema generation related, I can only guess the panic is from
generate_self_schema()?

[moto-timo]

In the GitHub 2.20.1 release whl, we see at least part of the same pattern

Self SchemaError building self-schema:
  Error building "" validator:
  modelmodel-fieldsdataclass-argsdataclasstuplefunction-afterfunction-beforefrozensetdefinition-refdefinitionsUnknown schema type: "2.20.1-alphaa-betabpydantic__version__profile=release pgo=args

And in the pypi wheel we see the same included host path pattern:

Self SchemaError building self-schema:
  " validator:
  model-fieldsdataclasstuplefrozensetdefinition-refdefinitionsUnknown schema type: "2.20.1a-alphab-beta__version__profile=release pgo=-Cprofile-use=/home/runner/work/pydantic-core/pydantic-core/merged.profdata-Cprofile-use=args

So it seems like we actually have a recurring general validation SchemaError?

[davidhewitt]

The -CProfile-use bit is the RUSTFLAGS baked in by

pydantic-core/src/lib.rs

Line 114 in 06a8348

option_env!("RUSTFLAGS").unwrap_or("").contains("-Cprofile-use="),

Perhaps if we wanted to write a way to evaluate that at build at const eval time then the flags could be omitted from the build.

The alpha / beta substring is probably coming from these pieces:

pydantic-core/src/lib.rs

Line 93 in 06a8348

version.replace("-alpha", "a").replace("-beta", "b")

Probably the self-schema text comes from

pydantic-core/src/validators/mod.rs

Line 427 in 06a8348

Err(err) => return py_schema_err!("Error building self-schema:\n {}", err),

In all cases these look like string constants which are reasonably necessarily part of the build rather than some "general validation SchemaError.

[fdebrabander]

The problem for the reproducible builds with Yocto project seems to come from the first snippet.

pydantic-core/src/lib.rs

Line 114 in 06a8348

option_env!("RUSTFLAGS").unwrap_or("").contains("-Cprofile-use="),

Yocto passes some flags with RUSTFLAGS, which includes paths that end up in the binary.

https://github.com/openembedded/openembedded-core/blob/bbcd56bace90f4a148960a7108dc8d0e6c364903/meta/classes-recipe/rust-common.bbclass#L15-L18

The '--remap-path-prefix' is to aid in creating reproducible builds, but that doesn't work in this case.