Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare for v0.11 release and fix remaining issues #283

Merged
merged 21 commits into from
Dec 1, 2022
Merged

Prepare for v0.11 release and fix remaining issues #283

merged 21 commits into from
Dec 1, 2022

Conversation

phil-opp
Copy link
Member

@phil-opp phil-opp commented Nov 13, 2022

Prepares the new v0.11 merged in #232 for publishing.

TODO:

  • update changelog
  • create git tags after publishing

Fixes #291
Fixes #286

@phil-opp phil-opp added this to the v0.11 milestone Nov 13, 2022
@phil-opp
Copy link
Member Author

I just published a v0.11-beta pre-release from this branch.

Docs.rs has no network access, so we can't `cargo install` the binary dependencies. This is not required for doc generation, so we trick our way around that issue by providing a dummy build script just for docs.rs.
@asensio-project
Copy link
Contributor

In the README.md there is a broken link to the migration guides.
The actual link is: https://github.com/rust-osdev/bootloader/blob/main/doc/migration
But the correct one is: https://github.com/rust-osdev/bootloader/blob/main/docs/migration

@phil-opp
Copy link
Member Author

@asensio-project Thanks! Fixed in 4e2b8d7

@phil-opp
Copy link
Member Author

Looks like the size of the generated first stage code increased on the latest nightly: https://github.com/rust-osdev/bootloader/actions/runs/3511818449/jobs/5882914661 . So we need to reduce the first stage size even more before merging this.

As a workaround, we could ship a precompiled version built with a pinned nightly version, but we should try find a proper fix first.

@asensio-project
Copy link
Contributor

When I tried to compile with the new version of this crate I have the following error:

error: failed to select a version for the requirement `funty = "~1.2"`
candidate versions found which didn't match: 2.0.0, 1.1.0, 1.0.1, ...
location searched: crates.io index
required by package `bitvec v0.22.0`
    ... which satisfies dependency `bitvec = "^0.22"` of package `mbrman v0.4.2`
    ... which satisfies dependency `mbrman = "^0.4.2"` of package `bootloader v0.11.0-beta`
    ... which satisfies dependency `bootloader = "^0.11.0-beta"` of package `os v0.1.0 (/home/user/os)`

Looks like a dependency error.

@bjorn3
Copy link
Contributor

bjorn3 commented Nov 23, 2022

Funty 1.2 has been yanked. Cargo ignores yanked versions unless Cargo.lock already mentions them.

@asensio-project
Copy link
Contributor

I know. But how can be fixed?

@bjorn3
Copy link
Contributor

bjorn3 commented Nov 23, 2022

The dependency on mbrman will need to be updated to 0.5, which uses bitvec 1.0, which in turn uses the non-yanked funty 2.0.

@phil-opp
Copy link
Member Author

mbrman v0.5 still seems to use bitvec 0.22. There is an open PR to update to bitvec 1.0: rust-disk-partition-management/mbrman#25

@BigBadE
Copy link

BigBadE commented Nov 25, 2022

@asensio-project In the meantime, the following workaround can be used:

[patch.crates-io]
# temporary solution to [email protected] being yanked, we should propose bitvec upgrade to upstream filecoin crates
# tracking issue: https://github.com/bitvecto-rs/funty/issues/7
funty = { git = "https://github.com/bitvecto-rs/funty/", rev = "7ef0d890fbcd8b3def1635ac1a877fc298488446" }

It must be in the top-level Cargo.toml

Fixes the build on the latest Rust nightlies, which apparently generate slightly bigger code than older nightlies.
@asensio-project
Copy link
Contributor

What about adding an option for avoiding the bootloader to don't write into screen until loading the kernel?

This way, we don't have to clear screen.

@phil-opp phil-opp changed the title Release v0.11 Prepare for v0.11 release and fix remaining issues Nov 27, 2022
@phil-opp phil-opp temporarily deployed to crates_io_release November 27, 2022 18:05 Inactive
@phil-opp phil-opp marked this pull request as ready for review November 27, 2022 18:14
@phil-opp
Copy link
Member Author

Since I don't know when the mbrman issue will be fixed, I decided to publish a temporary mbrman---bitvec-1-0 crate based on rust-disk-partition-management/mbrman#25. I also reduced the code size of the boot sector a bit more in 4c5a4a0.

The new v0.11.0-beta.4 release should now work on the latest nightly without needing a patch.crates-io section.

@phil-opp
Copy link
Member Author

What about adding an option for avoiding the bootloader to don't write into screen until loading the kernel?

This way, we don't have to clear screen.

We could add a quiet option to the bootloader configuration. Would that work for you?
I don't want to delay the v0.11.0 release over this, so could you please open a separate issue?

@phil-opp
Copy link
Member Author

Looks like there is some kind of test failure on macOS, related to the PIE tests. We only see a timeout error on the CI, but I assume this is some kind of panic in the bootloader.

If someone has access to a macOS machine, it would be great if you could try these steps:

  • Clone this branch.
  • In tests/runner/src/lib.rs, remove the "-display", "none" argument from the QEMU_ARGS
  • Run cargo test --test pie. This will open some QEMU windows.
  • If any panics happen, please report them.

@micfong-z
Copy link

micfong-z commented Nov 27, 2022

The test seems successful on my macOS M1 device.

If I add "-display", "none" arguments back, the test only passed in shorter time.

Might be a problem solely with the speed of GitHub actions?

Log
info: syncing channel updates for 'nightly-aarch64-apple-darwin'
info: latest update on 2022-11-27, rust version 1.67.0-nightly (80a96467e 2022-11-26)
warning: Force-skipping unavailable component 'clippy-preview-aarch64-apple-darwin'
warning: Force-skipping unavailable component 'rustfmt-preview-aarch64-apple-darwin'
info: downloading component 'clippy'
info: downloading component 'llvm-tools'
info: downloading component 'rust-src'
info: downloading component 'rust-std' for 'x86_64-unknown-none'
info: downloading component 'rustfmt'
info: downloading component 'cargo'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: removing previous version of component 'llvm-tools'
info: removing previous version of component 'rust-src'
info: removing previous version of component 'cargo'
info: removing previous version of component 'clippy-preview'
info: removing previous version of component 'rust-docs'
info: removing previous version of component 'rust-std'
info: removing previous version of component 'rustc'
info: removing previous version of component 'rustfmt-preview'
info: installing component 'clippy'
info: installing component 'llvm-tools'
info: installing component 'rust-src'
info: installing component 'rust-std' for 'x86_64-unknown-none'
info: installing component 'rustfmt'
info: installing component 'cargo'
info: installing component 'rust-docs'
info: installing component 'rust-std'
info: installing component 'rustc'
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /Users/micfong/Documents/GitHub/bootloader/bios/boot_sector/Cargo.toml
workspace: /Users/micfong/Documents/GitHub/bootloader/Cargo.toml
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /Users/micfong/Documents/GitHub/bootloader/bios/stage-2/Cargo.toml
workspace: /Users/micfong/Documents/GitHub/bootloader/Cargo.toml
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /Users/micfong/Documents/GitHub/bootloader/bios/stage-3/Cargo.toml
workspace: /Users/micfong/Documents/GitHub/bootloader/Cargo.toml
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /Users/micfong/Documents/GitHub/bootloader/bios/stage-4/Cargo.toml
workspace: /Users/micfong/Documents/GitHub/bootloader/Cargo.toml
    Updating `ustc` index
  Downloaded anyhow v1.0.56 (registry `ustc`)
  Downloaded byteorder v1.4.3 (registry `ustc`)
  Downloaded chrono v0.4.19 (registry `ustc`)
  Downloaded fastrand v1.8.0 (registry `ustc`)
  Downloaded funty v2.0.0 (registry `ustc`)
  Downloaded fatfs v0.3.5 (registry `ustc`)
  Downloaded getrandom v0.2.6 (registry `ustc`)
  Downloaded gpt v3.0.0 (registry `ustc`)
  Downloaded num-integer v0.1.44 (registry `ustc`)
  Downloaded num-traits v0.2.14 (registry `ustc`)
  Downloaded crc v1.8.1 (registry `ustc`)
  Downloaded proc-macro2 v1.0.39 (registry `ustc`)
  Downloaded quote v1.0.18 (registry `ustc`)
  Downloaded radium v0.7.0 (registry `ustc`)
  Downloaded remove_dir_all v0.5.3 (registry `ustc`)
  Downloaded arrayvec v0.5.2 (registry `ustc`)
  Downloaded mbrman---bitvec-1-0 v0.0.1 (registry `ustc`)
  Downloaded rustversion v1.0.6 (registry `ustc`)
  Downloaded serde-big-array v0.4.1 (registry `ustc`)
  Downloaded serde v1.0.136 (registry `ustc`)
  Downloaded serde_derive v1.0.136 (registry `ustc`)
  Downloaded strip-ansi-escapes v0.1.1 (registry `ustc`)
  Downloaded libc v0.2.122 (registry `ustc`)
  Downloaded syn v1.0.95 (registry `ustc`)
  Downloaded tap v1.0.1 (registry `ustc`)
  Downloaded thiserror v1.0.30 (registry `ustc`)
  Downloaded tempfile v3.3.0 (registry `ustc`)
  Downloaded thiserror-impl v1.0.30 (registry `ustc`)
  Downloaded uart_16550 v0.2.17 (registry `ustc`)
  Downloaded time v0.1.43 (registry `ustc`)
  Downloaded unicode-ident v1.0.0 (registry `ustc`)
  Downloaded utf8parse v0.2.0 (registry `ustc`)
  Downloaded wyz v0.5.1 (registry `ustc`)
  Downloaded bitvec v1.0.1 (registry `ustc`)
  Downloaded uuid v0.8.2 (registry `ustc`)
  Downloaded build_const v0.2.2 (registry `ustc`)
  Downloaded vte_generate_state_changes v0.1.1 (registry `ustc`)
  Downloaded vte v0.10.1 (registry `ustc`)
  Downloaded x86_64 v0.14.9 (registry `ustc`)
  Downloaded bincode v1.3.3 (registry `ustc`)
  Downloaded ovmf-prebuilt v0.1.0-alpha.1 (registry `ustc`)
  Downloaded 41 crates (3.5 MB) in 5.09s (largest was `ovmf-prebuilt` at 1.4 MB)
   Compiling rustversion v1.0.6
   Compiling bit_field v0.10.1
   Compiling volatile v0.4.4
   Compiling bitflags v1.3.2
   Compiling bootloader_api v0.11.0-beta.4 (/Users/micfong/Documents/GitHub/bootloader/api)
   Compiling proc-macro2 v1.0.39
   Compiling unicode-ident v1.0.0
   Compiling libc v0.2.122
   Compiling syn v1.0.95
   Compiling autocfg v1.1.0
   Compiling serde_derive v1.0.136
   Compiling cfg-if v1.0.0
   Compiling serde v1.0.136
   Compiling num-traits v0.2.14
   Compiling num-integer v0.1.44
   Compiling radium v0.7.0
   Compiling build_const v0.2.2
   Compiling log v0.4.17
   Compiling crc v1.8.1
   Compiling quote v1.0.18
   Compiling tap v1.0.1
   Compiling wyz v0.5.1
   Compiling time v0.1.43
   Compiling getrandom v0.2.6
   Compiling x86_64 v0.14.9
   Compiling funty v2.0.0
   Compiling anyhow v1.0.56
   Compiling llvm-tools v0.1.1
   Compiling uuid v0.8.2
   Compiling chrono v0.4.19
   Compiling vte_generate_state_changes v0.1.1
   Compiling bootloader v0.11.0-beta.4 (/Users/micfong/Documents/GitHub/bootloader)
   Compiling utf8parse v0.2.0
   Compiling bitvec v1.0.1
   Compiling byteorder v1.4.3
   Compiling arrayvec v0.5.2
   Compiling remove_dir_all v0.5.3
   Compiling fastrand v1.8.0
   Compiling uart_16550 v0.2.17
   Compiling vte v0.10.1
   Compiling tempfile v3.3.0
   Compiling test_kernel_higher_half v0.1.0 (/Users/micfong/Documents/GitHub/bootloader/tests/test_kernels/higher_half)
   Compiling test_kernel_pie v0.1.0 (/Users/micfong/Documents/GitHub/bootloader/tests/test_kernels/pie)
   Compiling test_kernel_default_settings v0.1.0 (/Users/micfong/Documents/GitHub/bootloader/tests/test_kernels/default_settings)
   Compiling test_kernel_map_phys_mem v0.1.0 (/Users/micfong/Documents/GitHub/bootloader/tests/test_kernels/map_phys_mem)
   Compiling fatfs v0.3.5
   Compiling gpt v3.0.0
   Compiling ovmf-prebuilt v0.1.0-alpha.1
   Compiling strip-ansi-escapes v0.1.1
   Compiling thiserror-impl v1.0.30
   Compiling thiserror v1.0.30
   Compiling serde-big-array v0.4.1
   Compiling bincode v1.3.3
   Compiling mbrman---bitvec-1-0 v0.0.1
   Compiling bootloader_test_runner v0.1.0 (/Users/micfong/Documents/GitHub/bootloader/tests/runner)
    Finished test [unoptimized + debuginfo] target(s) in 2m 16s
     Running tests/pie.rs (target/debug/deps/pie-2b75b4a432cef9ed)

running 4 tests
BdsDxe: failed to load Boot0001 "UEFI QEMU DVD-ROM QM00003 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Secondary,Master,0x0): Not Found
BdsDxe: loading Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)
BdsDxe: starting Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)
UEFI bootloader started; trying to load kernel
BdsDxe: failed to load Boot0001 "UEFI QEMU DVD-ROM QM00003 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Secondary,Master,0x0): Not Found
BdsDxe: loading Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)
BdsDxe: starting Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)
UEFI bootloader started; trying to load kernel
BdsDxe: failed to load Boot0001 "UEFI QEMU DVD-ROM QM00003 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Secondary,Master,0x0): Not Found
BdsDxe: loading Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)
BdsDxe: starting Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)
UEFI bootloader started; trying to load kernel
BdsDxe: failed to load Boot0001 "UEFI QEMU DVD-ROM QM00003 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Secondary,Master,0x0): Not Found
BdsDxe: loading Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)
BdsDxe: starting Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)
UEFI bootloader started; trying to load kernel
BdsDxe: failed to load Boot0001 "UEFI QEMU DVD-ROM QM00003 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Secondary,Master,0x0): Not Found

>>Start PXE over IPv4.
  Station IP address is 192.168.17.15

  Server IP address is 192.168.17.2
  NBP filename is bootloader
  NBP filesize is 169984 Bytes
 Downloading NBP file...

  NBP file downloaded successfully.
BdsDxe: loading Boot0002 "UEFI PXEv4 (MAC:525400123456)" from PciRoot(0x0)/Pci(0x3,0x0)/MAC(525400123456,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)
BdsDxe: starting Boot0002 "UEFI PXEv4 (MAC:525400123456)" from PciRoot(0x0)/Pci(0x3,0x0)/MAC(525400123456,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)
UEFI bootloader started; trying to load kernel
test basic_boot ... ok
BdsDxe: failed to load Boot0001 "UEFI QEMU DVD-ROM QM00003 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Secondary,Master,0x0): Not Found

>>Start PXE over IPv4.
  Station IP address is 192.168.17.15

  Server IP address is 192.168.17.2
  NBP filename is bootloader
  NBP filesize is 169984 Bytes
 Downloading NBP file...

  NBP file downloaded successfully.
BdsDxe: loading Boot0002 "UEFI PXEv4 (MAC:525400123456)" from PciRoot(0x0)/Pci(0x3,0x0)/MAC(525400123456,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)
BdsDxe: starting Boot0002 "UEFI PXEv4 (MAC:525400123456)" from PciRoot(0x0)/Pci(0x3,0x0)/MAC(525400123456,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)
UEFI bootloader started; trying to load kernel
test global_variable ... ok
BdsDxe: failed to load Boot0001 "UEFI QEMU DVD-ROM QM00003 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Secondary,Master,0x0): Not Found

>>Start PXE over IPv4.
  Station IP address is 192.168.17.15

  Server IP address is 192.168.17.2
  NBP filename is bootloader
  NBP filesize is 169984 Bytes
 Downloading NBP file...

  NBP file downloaded successfully.
BdsDxe: loading Boot0002 "UEFI PXEv4 (MAC:525400123456)" from PciRoot(0x0)/Pci(0x3,0x0)/MAC(525400123456,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)
BdsDxe: starting Boot0002 "UEFI PXEv4 (MAC:525400123456)" from PciRoot(0x0)/Pci(0x3,0x0)/MAC(525400123456,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)
UEFI bootloader started; trying to load kernel
test should_panic ... ok
BdsDxe: failed to load Boot0001 "UEFI QEMU DVD-ROM QM00003 " from PciRoot(0x0)/Pci(0x1,0x1)/Ata(Secondary,Master,0x0): Not Found

>>Start PXE over IPv4.
  Station IP address is 192.168.17.15

  Server IP address is 192.168.17.2
  NBP filename is bootloader
  NBP filesize is 169984 Bytes
 Downloading NBP file...

  NBP file downloaded successfully.
BdsDxe: loading Boot0002 "UEFI PXEv4 (MAC:525400123456)" from PciRoot(0x0)/Pci(0x3,0x0)/MAC(525400123456,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)
BdsDxe: starting Boot0002 "UEFI PXEv4 (MAC:525400123456)" from PciRoot(0x0)/Pci(0x3,0x0)/MAC(525400123456,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)
UEFI bootloader started; trying to load kernel
test check_boot_info ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 14.24s

@phil-opp
Copy link
Member Author

@micfong-z Thanks for testing!

Might be a problem solely with the speed of GitHub actions?

I don't think that speed is the problem. The test runs for almost 20 minutes before it gets canceled by out timeout:

image

This typically happens on a panic. Unfortunately, we don't have serial output of the bootloader panic message, so we don't see it in the logs...

@micfong-z
Copy link

micfong-z commented Nov 29, 2022

fyi, I just ran a GitHub Actions on my fork, testing release branch on macOS and it successfully passes the test:
https://github.com/micfong-z/bootloader/actions/runs/3576183532

The test runs for almost 20 minutes before it gets canceled by out timeout

I would doubt this is a technical glitch from GitHub, but I'm not quite sure about how Actions works, and I just edited on the existing testing yml file. I might have missed some steps to replicate the problem.

@phil-opp
Copy link
Member Author

Interesting.. I just restarted the failed CI jobs manually, let's see if the problem is reproducible.

@micfong-z
Copy link

"All checks have passed" now :D

@cecton
Copy link

cecton commented Dec 1, 2022

I just updated mbrman to 0.5.1 which uses bitvec 1.0.1. Sorry for the long delay

@phil-opp
Copy link
Member Author

phil-opp commented Dec 1, 2022

I just updated mbrman to 0.5.1 which uses bitvec 1.0.1. Sorry for the long delay

Thanks a lot!

"All checks have passed" now :D

Well, I guess we can merge then :D. We have a scheduled CI build in place for the main branch, so we should notice if this error wasn't spurious and happens again at some point.

@phil-opp phil-opp merged commit 71f574d into main Dec 1, 2022
@phil-opp phil-opp deleted the release branch December 1, 2022 10:21
@phil-opp phil-opp restored the release branch December 1, 2022 10:53
@phil-opp phil-opp deleted the release branch December 1, 2022 10:54
@phil-opp phil-opp mentioned this pull request Dec 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
6 participants