Skip to content

binwalk, diffoscope: update after Python 3.12#325623

Closed
philiptaron wants to merge 2 commits intoNixOS:masterfrom
philiptaron:fix-python-3.12/binwalk
Closed

binwalk, diffoscope: update after Python 3.12#325623
philiptaron wants to merge 2 commits intoNixOS:masterfrom
philiptaron:fix-python-3.12/binwalk

Conversation

@philiptaron
Copy link
Contributor

@philiptaron philiptaron commented Jul 8, 2024

Description of changes

related to #325657 (comment)
cc @Mag1cByt3s

Result of nixpkgs-review run on x86_64-linux 1

15 packages built:
  • binwalk (python312Packages.binwalk)
  • binwalk.dist (python312Packages.binwalk.dist)
  • diffoscope
  • diffoscope.dist
  • diffoscope.man
  • diffoscopeMinimal
  • diffoscopeMinimal.dist
  • diffoscopeMinimal.man
  • ghidra-extensions.ghidraninja-ghidra-scripts
  • python311Packages.binwalk
  • python311Packages.binwalk-full
  • python311Packages.binwalk-full.dist
  • python311Packages.binwalk.dist
  • python312Packages.binwalk-full
  • python312Packages.binwalk-full.dist

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@github-actions github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Jul 8, 2024
@philiptaron
Copy link
Contributor Author

@trofi fyi

@emilazy
Copy link
Member

emilazy commented Jul 8, 2024

Is that behavioural change in binwalk benign? I’m curious as to why a Python upgrade and an import internals change would lead to it finding more binaries.

@philiptaron
Copy link
Contributor Author

Is that behavioural change in binwalk benign? I’m curious as to why a Python upgrade and an import internals change would lead to it finding more binaries.

I was quite surprised to see it as well. You can see from the patch what the detected differences are.

I checked out upstream (https://salsa.debian.org/reproducible-builds/diffoscope) locally and ran master's copy of binwalk (for me, /nix/store/ckv7akdj4cjl8qkiij8vraiisw9n6mvj-python3.11-binwalk-2.3.4):

$ /nix/store/ckv7akdj4cjl8qkiij8vraiisw9n6mvj-python3.11-binwalk-2.3.4/bin/binwalk tests/data/test{1,2}.binwalk

Scan Time:     2024-07-08 13:07:08
Target File:   /home/philip/Code/salsa.debian.org/reproducible-builds/diffoscope/tests/data/test1.binwalk
MD5 Checksum:  03bb6be017bd9db1586ee5cc291dd992
Signatures:    411

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
112           0x70            ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
228           0xE4            ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x00000006"
356           0x164           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x000001BE"
924           0x39C           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"
1536          0x600           ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
1648          0x670           ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
1764          0x6E4           ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x0000000D"
1900          0x76C           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x0000029F"
2692          0xA84           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"


Scan Time:     2024-07-08 13:07:08
Target File:   /home/philip/Code/salsa.debian.org/reproducible-builds/diffoscope/tests/data/test2.binwalk
MD5 Checksum:  d47335a3e695dc2b3dd8bfc56ba476a5
Signatures:    411

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
112           0x70            ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
228           0xE4            ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x0000000D"
364           0x16C           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x0000029F"
1156          0x484           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"
1536          0x600           ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
1648          0x670           ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
1764          0x6E4           ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x00000006"
1892          0x764           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x000001BE"
2460          0x99C           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"

Then, with this PR (note the Python 3.12 in the derivation name):

$ /nix/store/2c7kkgy4hbvxjh1rj0cvcvc0r1yb136l-python3.12-binwalk-2.3.4/bin/binwalk tests/data/test{1,2}.binwalk

Scan Time:     2024-07-08 13:08:08
Target File:   /home/philip/Code/salsa.debian.org/reproducible-builds/diffoscope/tests/data/test1.binwalk
MD5 Checksum:  03bb6be017bd9db1586ee5cc291dd992
Signatures:    411

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
112           0x70            ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
228           0xE4            ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x00000006"
356           0x164           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x000001BE"
924           0x39C           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"
1536          0x600           ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
1648          0x670           ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
1764          0x6E4           ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x0000000D"
1900          0x76C           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x0000029F"
2692          0xA84           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"


Scan Time:     2024-07-08 13:08:08
Target File:   /home/philip/Code/salsa.debian.org/reproducible-builds/diffoscope/tests/data/test2.binwalk
MD5 Checksum:  d47335a3e695dc2b3dd8bfc56ba476a5
Signatures:    411

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
112           0x70            ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
228           0xE4            ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x0000000D"
364           0x16C           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x0000029F"
1156          0x484           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"
1536          0x600           ASCII cpio archive (SVR4 with no CRC), file name: ".", file name length: "0x00000002", file size: "0x00000000"
1648          0x670           ASCII cpio archive (SVR4 with no CRC), file name: "dir", file name length: "0x00000004", file size: "0x00000000"
1764          0x6E4           ASCII cpio archive (SVR4 with no CRC), file name: "dir/link", file name length: "0x00000009", file size: "0x00000006"
1892          0x764           ASCII cpio archive (SVR4 with no CRC), file name: "dir/text", file name length: "0x00000009", file size: "0x000001BE"
2460          0x99C           ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000"

What I don't understand is how there were ever two emitted items. Master appears to have 10 as well.

@ofborg ofborg bot added 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Jul 8, 2024
def test_listing(comparison):
differences = comparison.details
- assert comparison.comments == ["comprises of 2 embedded members"]
+ assert comparison.comments == ["comprises of 10 embedded members"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this by design i.e. a number that we can rely on to continue to be 10? or might this change in the future?

@dotlambda dotlambda marked this pull request as draft July 10, 2024 14:23
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/error-nose-1-3-7-not-supported-for-interpreter-python3-12/48703/22

@emilazy emilazy mentioned this pull request Jul 11, 2024
13 tasks
@superherointj
Copy link
Contributor

superherointj commented Jul 11, 2024

Please rebase. (And whatever new improvement/fix that isn't in 326295 can be integrated.)

@eclairevoyant
Copy link
Contributor

eclairevoyant commented Jul 11, 2024

Both binwalk and diffoscope appear to build on current master (6bf4ef3)

@superherointj
Copy link
Contributor

Both binwalk and diffoscope appear to build on current master (6bf4ef3)

Yes, because I merged #326295.

@emilazy
Copy link
Member

emilazy commented Jul 11, 2024

But there’s nothing to rebase now; #326295 replaces this.

@eclairevoyant

This comment was marked as duplicate.

@philiptaron
Copy link
Contributor Author

But there’s nothing to rebase now; #326295 replaces this.

Quite so.

@philiptaron philiptaron deleted the fix-python-3.12/binwalk branch July 13, 2024 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: python Python is a high-level, general-purpose programming language. 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants