Encoding 1-pixel L8 BMP requires infinite RAM on Windows 10 x64

[AlyoshaVasilieva]

Expected

Encoding a 1-pixel BMP should take very little RAM.

Actual behaviour

memory allocation of 26843545600 bytes failederror: process didn't exit successfully: `target\release\bug-repro.exe` (exit code: 0xc0000409, STATUS_STACK_BUFFER_OVERRUN)

Reproduction steps

Cargo.toml:

[package]
name = "bug-repro"
version = "0.1.0"
authors = ["Malloc Voidstar <[email protected]>"]
edition = "2018"

[dependencies.image]
version = "=0.23.7"
default-features = false
features = ["bmp"]

[profile.release]
lto = true
codegen-units = 1

main.rs:

use image::ColorType;

fn main() {
    let mut out = Vec::with_capacity(51200);
    let mut encoder = image::bmp::BMPEncoder::new(&mut out);
    encoder.encode(&[1], 1, 1, ColorType::L8).unwrap();
    println!("out len: {}", out.len());
}

Command: cargo run --release

Output on an ARM SBC (aarch64-unknown-linux-gnu): out len: 1082

On both my Windows 10 machines it fails due to memory allocation error (25GB on one, 100GB on the other).

I'm not entirely sure this is a bug in image rather than Rust itself; I first noticed this when a project suddenly began failing on 1.45 but worked fine on 1.44.1. While the above minimized code fails on both 1.45 and 1.44.1, it succeeds on other versions:

Rust version	Works?
1.45 (LLVM 10.0)	No
1.44.1	No
1.43.1	Yes (?!)
1.42.0	No
1.41.1	No
1.40.0	No
1.39.0	No
1.38.0 (LLVM 9.0)	No
1.37.0	Yes
1.36.0	Yes
1.35.0	Yes
1.34.0	Yes

[HeroicKatora]

Note: dec(26843545600) = hex(640000000)

[AlyoshaVasilieva]

Not a bug in image. New minimal form:

use std::io::Write;

use byteorder::WriteBytesExt;

fn main() {
    let mut out = Vec::with_capacity(51200);
    for val in 0u8..=255 {
        out.write_all(&[val, val, val, 0]).unwrap();
    }

    for _ in (0..1).rev() {
        for _ in 0..1 {
            out.write_u8(0).unwrap();
        }
    }
    println!("{} bytes written", out.len());
}

edit: I reported it at rust-lang/rust#74498 . Hopefully I didn't overlook anything. I guess this issue can be closed.