Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SrcLoc sorting is non-transitive and a false total order and equality (panics in 1.81) #1126

Open
Kriskras99 opened this issue Sep 15, 2024 · 13 comments · Fixed by #1128
Open
Labels
bug Something isn't working

Comments

@Kriskras99
Copy link
Contributor

Kriskras99 commented Sep 15, 2024

Seems one (or more) of the types used in the transpiler have a wrong Ord implementation.

Transpiling cr.c
thread 'main' panicked at core/src/slice/sort/shared/smallsort.rs:860:5:
user-provided comparison function does not correctly implement a total order
stack backtrace:
   0:     0x620e4a93b8da - std::backtrace_rs::backtrace::libunwind::trace::h4f532649ff050af0
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:     0x620e4a93b8da - std::backtrace_rs::backtrace::trace_unsynchronized::h2bd9c4760a46d04c
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x620e4a93b8da - std::sys::backtrace::_print_fmt::h06a2f1333d2c9eb7
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/sys/backtrace.rs:66:9
   3:     0x620e4a93b8da - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hc39aee40395be66f
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/sys/backtrace.rs:39:26
   4:     0x620e4a9628db - core::fmt::rt::Argument::fmt::h391dd44a8ecf182f
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/fmt/rt.rs:177:76
   5:     0x620e4a9628db - core::fmt::write::h718938895d6a6b80
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/fmt/mod.rs:1178:21
   6:     0x620e4a937d83 - std::io::Write::write_fmt::h97874dbf951485b0
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/io/mod.rs:1823:15
   7:     0x620e4a93b722 - std::sys::backtrace::BacktraceLock::print::h28b06058dc60edca
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/sys/backtrace.rs:42:9
   8:     0x620e4a93c9e7 - std::panicking::default_hook::{{closure}}::ha6790375217144d8
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/panicking.rs:268:22
   9:     0x620e4a93c816 - std::panicking::default_hook::h7a0db9554a291cc0
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/panicking.rs:295:9
  10:     0x620e4a93d027 - std::panicking::rust_panic_with_hook::h35232034e4df15db
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/panicking.rs:801:13
  11:     0x620e4a93ce93 - std::panicking::begin_panic_handler::{{closure}}::hf188e2435ae6fe44
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/panicking.rs:667:13
  12:     0x620e4a93bdb9 - std::sys::backtrace::__rust_end_short_backtrace::h63f9b1b738507ea6
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/sys/backtrace.rs:170:18
  13:     0x620e4a93cb54 - rust_begin_unwind
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/panicking.rs:665:5
  14:     0x620e4a960703 - core::panicking::panic_fmt::ha3d1a45ecd949c55
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/panicking.rs:74:14
  15:     0x620e4a964d9b - core::slice::sort::shared::smallsort::panic_on_ord_violation::hbf9c589b6c9b3724
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/slice/sort/shared/smallsort.rs:860:5
  16:     0x620e4a4c7f9f - core::slice::sort::shared::smallsort::bidirectional_merge::h6690bf85da2564d5
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/slice/sort/shared/smallsort.rs:838:13
  17:     0x620e4a4c7f9f - core::slice::sort::shared::smallsort::small_sort_network::h890c54e4f75c0902
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/slice/sort/shared/smallsort.rs:370:9
  18:     0x620e4a428107 - <T as core::slice::sort::shared::smallsort::UnstableSmallSortFreezeTypeImpl>::small_sort::hd5067ef6cf7d40f7
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/slice/sort/shared/smallsort.rs:164:13
  19:     0x620e4a428107 - <T as core::slice::sort::shared::smallsort::UnstableSmallSortTypeImpl>::small_sort::h2df5d3e38e278dc7
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/slice/sort/shared/smallsort.rs:101:9
  20:     0x620e4a428107 - core::slice::sort::unstable::quicksort::quicksort::hea4648e1605e90a5
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/slice/sort/unstable/quicksort.rs:24:13
  21:     0x620e4a428db8 - core::slice::sort::unstable::quicksort::quicksort::hea4648e1605e90a5
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/slice/sort/unstable/quicksort.rs:71:9
  22:     0x620e4a428db8 - core::slice::sort::unstable::quicksort::quicksort::hea4648e1605e90a5
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/slice/sort/unstable/quicksort.rs:71:9
  23:     0x620e4a512e46 - core::slice::sort::unstable::sort::h35235f49a1d553dd
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/slice/sort/unstable/mod.rs:43:5
  24:     0x620e4a512e46 - core::slice::<impl [T]>::sort_unstable_by::h504cc4e282cf16fc
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/slice/mod.rs:2981:9
  25:     0x620e4a512e46 - c2rust_transpile::c_ast::TypedAstContext::sort_top_decls::hc2a55804d3bb0a25
                               at /home/kriskras99/Source/c2rust/c2rust-transpile/src/c_ast/mod.rs:674:9
  26:     0x620e4a512e46 - c2rust_transpile::translator::translate::h599712b2056933b9
                               at /home/kriskras99/Source/c2rust/c2rust-transpile/src/translator/mod.rs:513:9
  27:     0x620e4a5882a5 - c2rust_transpile::transpile_single::he88bad69628bcf0b
                               at /home/kriskras99/Source/c2rust/c2rust-transpile/src/lib.rs:536:9
  28:     0x620e4a3c2fdb - c2rust_transpile::transpile::{{closure}}::h1c26c176d2edae16
                               at /home/kriskras99/Source/c2rust/c2rust-transpile/src/lib.rs:328:17
  29:     0x620e4a3c2fdb - core::iter::adapters::map::map_fold::{{closure}}::he4bccb3109727379
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/iter/adapters/map.rs:88:28
  30:     0x620e4a3c2fdb - <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::fold::hd20ded494caaffcb
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/slice/iter/macros.rs:232:27
  31:     0x620e4a3c2fdb - <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold::h45f09cacc691c60d
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/iter/adapters/map.rs:128:9
  32:     0x620e4a3c2fdb - core::iter::traits::iterator::Iterator::for_each::h2c40a859fa37c766
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/iter/traits/iterator.rs:813:9
  33:     0x620e4a3c2fdb - alloc::vec::Vec<T,A>::extend_trusted::hb59eb0450c1a5309
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/alloc/src/vec/mod.rs:3125:17
  34:     0x620e4a3c2fdb - <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend::hdfb895ec3556694f
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/alloc/src/vec/spec_extend.rs:26:9
  35:     0x620e4a3c2fdb - <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter::ha75a8e9b8b1dfed4
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/alloc/src/vec/spec_from_iter_nested.rs:60:9
  36:     0x620e4a3c2fdb - <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter::hd503974f9b2fb62a
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/alloc/src/vec/spec_from_iter.rs:33:9
  37:     0x620e4a584ece - <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter::hdcb5e575103de682
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/alloc/src/vec/mod.rs:2989:9
  38:     0x620e4a584ece - core::iter::traits::iterator::Iterator::collect::h748ec39779ff7b21
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/iter/traits/iterator.rs:2000:9
  39:     0x620e4a584ece - c2rust_transpile::transpile::hd0035b677d692a90
                               at /home/kriskras99/Source/c2rust/c2rust-transpile/src/lib.rs:337:14
  40:     0x620e4a35911d - c2rust_transpile::main::h74e6ff961002307d
                               at /home/kriskras99/Source/c2rust/c2rust/src/bin/c2rust-transpile.rs:259:5
  41:     0x620e4a361a33 - core::ops::function::FnOnce::call_once::h5ca91e7049f14308
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/ops/function.rs:250:5
  42:     0x620e4a361a33 - std::sys::backtrace::__rust_begin_short_backtrace::hd38ba6c044c85727
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/sys/backtrace.rs:154:18
  43:     0x620e4a361a29 - std::rt::lang_start::{{closure}}::h756998bcb6544777
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/rt.rs:164:18
  44:     0x620e4a933200 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h5412c97e5e68829b
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/core/src/ops/function.rs:284:13
  45:     0x620e4a933200 - std::panicking::try::do_call::h3d002e8ffa4f91b7
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/panicking.rs:557:40
  46:     0x620e4a933200 - std::panicking::try::h60529ec65ad686dc
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/panicking.rs:520:19
  47:     0x620e4a933200 - std::panic::catch_unwind::h28f4a63bd2319cba
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/panic.rs:345:14
  48:     0x620e4a933200 - std::rt::lang_start_internal::{{closure}}::h14b8d458d7b543a7
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/rt.rs:143:48
  49:     0x620e4a933200 - std::panicking::try::do_call::h9747ac76d396a084
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/panicking.rs:557:40
  50:     0x620e4a933200 - std::panicking::try::h0f984b4d1c23e9bd
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/panicking.rs:520:19
  51:     0x620e4a933200 - std::panic::catch_unwind::hcd3e02aab11d9c76
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/panic.rs:345:14
  52:     0x620e4a933200 - std::rt::lang_start_internal::h370157592acdbc7d
                               at /rustc/9b72238eb813e9d06e9e9d270168512fbffd7ee7/library/std/src/rt.rs:143:20
  53:     0x620e4a35dbac - main
  54:     0x787e58034e08 - <unknown>
  55:     0x787e58034ecc - __libc_start_main
  56:     0x620e4a354325 - _start
  57:                0x0 - <unknown>

To reproduce:

cargo +1.81.0 install --git https://github.com/immunant/c2rust.git c2rust
git clone https://github.com/chirlu/soxr
cd soxr
cmake -S . -B build -DCMAKE_BUILD_TYPE=None -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBUILD_EXAMPLES=OFF -DBUILD_SHARED_LIBS=ON -DWITH_AVFFT=ON -DWITH_LSR_BINDINGS=ON -DWITH_OPENMP=OFF -DWITH_PFFFT=ON
c2rust transpile -o rust build/compile_commands.json
@kkysen kkysen added the bug Something isn't working label Sep 15, 2024
@Kriskras99
Copy link
Contributor Author

I've added a better backtrace from a release build with debug symbols enabled.

Problem is c2rust_transpile::c_ast::TypedAstContext::sort_top_decls at c2rust/c2rust-transpile/src/c_ast/mod.rs:674:9

@kkysen
Copy link
Contributor

kkysen commented Sep 15, 2024

It seems like this is the incorrect Ord:

pub fn sort_top_decls(&mut self) {
// Group and sort declarations by file and by position
let mut decls_top = mem::take(&mut self.c_decls_top);
decls_top.sort_unstable_by(|a, b| {
let a = self.index(*a);
let b = self.index(*b);
use Ordering::*;
match (&a.loc, &b.loc) {
(None, None) => Equal,
(None, _) => Less,
(_, None) => Greater,
(Some(a), Some(b)) => self.compare_src_locs(&a.begin(), &b.begin()),
}
});
self.c_decls_top = decls_top;
}

Maybe this is also why the declarations were always backwards in rav1d?

@Kriskras99
Copy link
Contributor Author

I think it will end up being compare_src_locs, but I don't understand the code enough to know what's the problem

@kkysen
Copy link
Contributor

kkysen commented Sep 15, 2024

I think it will end up being compare_src_locs, but I don't understand the code enough to know what's the problem

Yeah, it seems like it's in there. @fw-immunant or @rinon, do you remember what was going on here and where a non-total ordering might be coming from?

pub fn compare_src_locs(&self, a: &SrcLoc, b: &SrcLoc) -> Ordering {
/// Compare `self` with `other`, without regard to file id
fn cmp_pos(a: &SrcLoc, b: &SrcLoc) -> Ordering {
(a.line, a.column).cmp(&(b.line, b.column))
}
let path_a = self.include_map[self.file_map[a.fileid as usize]].clone();
let path_b = self.include_map[self.file_map[b.fileid as usize]].clone();
for (include_a, include_b) in path_a.iter().zip(path_b.iter()) {
if include_a.fileid != include_b.fileid {
return cmp_pos(include_a, include_b);
}
}
use Ordering::*;
match path_a.len().cmp(&path_b.len()) {
Less => {
// compare the place b was included in a'a file with a
let b = path_b.get(path_a.len()).unwrap();
cmp_pos(a, b)
}
Equal => cmp_pos(a, b),
Greater => {
// compare the place a was included in b's file with b
let a = path_a.get(path_b.len()).unwrap();
cmp_pos(a, b)
}
}
}

@fw-immunant
Copy link
Contributor

I'm not familiar with this code but the last semantically relevant change here seems to be 5d3b0aa.

@Kriskras99
Copy link
Contributor Author

"Minimal" reproducer. Anybody know of something like creduce for JSON?
The sort_data.json will pass if you remove more than 5 consecutive spans, and weeding through 1200 spans by hand is a bit of a pain.

use std::{cmp::Ordering, fs::File};

use serde::Deserialize;


#[derive(Copy, Debug, Clone, PartialOrd, PartialEq, Ord, Eq, Deserialize)]
pub struct SrcLoc {
    pub fileid: u64,
    pub line: u64,
    pub column: u64,
}

#[derive(Copy, Debug, Clone, PartialOrd, PartialEq, Ord, Eq, Deserialize)]
pub struct SrcSpan {
    pub fileid: u64,
    pub begin_line: u64,
    pub begin_column: u64,
    pub end_line: u64,
    pub end_column: u64,
}

impl SrcSpan {
    pub fn begin(&self) -> SrcLoc {
        SrcLoc {
            fileid: self.fileid,
            line: self.begin_line,
            column: self.begin_column,
        }
    }
    pub fn end(&self) -> SrcLoc {
        SrcLoc {
            fileid: self.fileid,
            line: self.end_line,
            column: self.end_column,
        }
    }
}

pub type FileId = usize;
#[derive(Deserialize)]
struct Temp {
    spans: Vec<Option<SrcSpan>>,
    file_map: Vec<FileId>,
    include_map: Vec<Vec<SrcLoc>>,
}
use Ordering::*;


pub fn main() {
    let file = File::open("sort_data.json").unwrap();
    let mut temp: Temp = serde_json::from_reader(file).unwrap();

    println!("{}", temp.spans.len());

    sort_top_decls(&mut temp.spans[..], &temp.file_map, &temp.include_map);
}

pub fn sort_top_decls(spans: &mut [Option<SrcSpan>], file_map: &[FileId], include_map: &[Vec<SrcLoc>]) {
    // Group and sort declarations by file and by position
    spans.sort_unstable_by(|a, b| {
        match (a, b) {
            (None, None) => Equal,
            (None, _) => Less,
            (_, None) => Greater,
            (Some(a), Some(b)) => {
                compare_src_locs(file_map, include_map, &a.begin(), &b.begin())
            },
        }
    });
}

pub fn compare_src_locs(file_map: &[FileId], include_map: &[Vec<SrcLoc>], a: &SrcLoc, b: &SrcLoc) -> Ordering {
    /// Compare `self` with `other`, without regard to file id
    fn cmp_pos(a: &SrcLoc, b: &SrcLoc) -> Ordering {
        (a.line, a.column).cmp(&(b.line, b.column))
    }
    let path_a = include_map[file_map[a.fileid as usize]].clone();
    let path_b = include_map[file_map[b.fileid as usize]].clone();
    for (include_a, include_b) in path_a.iter().zip(path_b.iter()) {
        if include_a.fileid != include_b.fileid {
            return cmp_pos(include_a, include_b);
        }
    }
    match path_a.len().cmp(&path_b.len()) {
        Less => {
            // compare the place b was included in a's file with a
            let b = path_b.get(path_a.len()).unwrap();
            cmp_pos(a, b)
        }
        Equal => cmp_pos(a, b),
        Greater => {
            // compare the place a was included in b's file with b
            let a = path_a.get(path_b.len()).unwrap();
            cmp_pos(a, b)
        }
    }
}

sort_data.json

@kkysen
Copy link
Contributor

kkysen commented Sep 17, 2024

Thanks for that reproduction! That's very helpful. I don't know of any analogous tool like creduce but for JSON; that would be useful. That said, I was able to test that on the total order properties, and found this that violates == transitivity:

use serde::Deserialize;
use std::cmp::Ordering;
use std::fs::File;

#[derive(Copy, Debug, Clone, PartialOrd, PartialEq, Ord, Eq, Deserialize)]
pub struct SrcLoc {
    pub fileid: u64,
    pub line: u64,
    pub column: u64,
}

#[derive(Copy, Debug, Clone, PartialOrd, PartialEq, Ord, Eq, Deserialize)]
pub struct SrcSpan {
    pub fileid: u64,
    pub begin_line: u64,
    pub begin_column: u64,
    pub end_line: u64,
    pub end_column: u64,
}

impl SrcSpan {
    pub fn begin(&self) -> SrcLoc {
        SrcLoc {
            fileid: self.fileid,
            line: self.begin_line,
            column: self.begin_column,
        }
    }

    pub fn end(&self) -> SrcLoc {
        SrcLoc {
            fileid: self.fileid,
            line: self.end_line,
            column: self.end_column,
        }
    }
}

pub type FileId = usize;
#[derive(Deserialize)]
struct Temp {
    spans: Vec<Option<SrcSpan>>,
    file_map: Vec<FileId>,
    include_map: Vec<Vec<SrcLoc>>,
}

pub fn sort_top_decls(
    spans: &mut [Option<SrcSpan>],
    file_map: &[FileId],
    include_map: &[Vec<SrcLoc>],
) {
    use Ordering::*;
    // Group and sort declarations by file and by position
    spans.sort_unstable_by(|a, b| match (a, b) {
        (None, None) => Equal,
        (None, _) => Less,
        (_, None) => Greater,
        (Some(a), Some(b)) => compare_src_locs(file_map, include_map, &a.begin(), &b.begin()),
    });
}

pub fn compare_src_locs(
    file_map: &[FileId],
    include_map: &[Vec<SrcLoc>],
    a: &SrcLoc,
    b: &SrcLoc,
) -> Ordering {
    /// Compare `self` with `other`, without regard to file id
    fn cmp_pos(a: &SrcLoc, b: &SrcLoc) -> Ordering {
        (a.line, a.column).cmp(&(b.line, b.column))
    }
    let path_a = &include_map[file_map[a.fileid as usize]][..];
    let path_b = &include_map[file_map[b.fileid as usize]][..];
    for (include_a, include_b) in path_a.iter().zip(path_b.iter()) {
        if include_a.fileid != include_b.fileid {
            return cmp_pos(include_a, include_b);
        }
    }

    use Ordering::*;
    match path_a.len().cmp(&path_b.len()) {
        Less => {
            // compare the place b was included in a's file with a
            let b = &path_b[path_a.len()];
            cmp_pos(a, b)
        }
        Equal => cmp_pos(a, b),
        Greater => {
            // compare the place a was included in b's file with b
            let a = &path_a[path_b.len()];
            cmp_pos(a, b)
        }
    }
}

struct MappedSrcLoc<'a> {
    src_loc: SrcLoc,
    file_map: &'a [FileId],
    include_map: &'a [Vec<SrcLoc>],
}

impl PartialEq for MappedSrcLoc<'_> {
    fn eq(&self, other: &Self) -> bool {
        self.partial_cmp(other) == Some(Ordering::Equal)
    }
}

impl Eq for MappedSrcLoc<'_> {}

impl PartialOrd for MappedSrcLoc<'_> {
    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
        Some(compare_src_locs(
            self.file_map,
            self.include_map,
            &self.src_loc,
            &other.src_loc,
        ))
    }
}

impl Ord for MappedSrcLoc<'_> {
    fn cmp(&self, other: &Self) -> Ordering {
        self.partial_cmp(other).unwrap()
    }
}

pub fn main() {
    let file = File::open("sort_data.json").unwrap();
    let temp: Temp = serde_json::from_reader(file).unwrap();
    let Temp {
        spans,
        file_map,
        include_map,
    } = temp;
    let file_map = &file_map;
    let include_map = &include_map;

    // sort_top_decls(&mut spans[..], file_map, include_map);

    let spans = spans
        .into_iter()
        .filter_map(|span| span)
        .collect::<Vec<_>>();
    // spans.sort_unstable_by(|a, b| compare_src_locs(file_map, include_map, &a.begin(), &b.begin()));

    let locs = spans
        .into_iter()
        .map(|span| span.begin())
        .collect::<Vec<_>>();
    // locs.sort_unstable_by(|a, b| compare_src_locs(file_map, include_map, &a, &b));

    let mapped_locs = locs
        .into_iter()
        .map(|src_loc| MappedSrcLoc {
            src_loc,
            file_map,
            include_map,
        })
        .collect::<Vec<_>>();
    // mapped_locs.sort_unstable();

    let n = mapped_locs.len();
    for i in 0..n {
        let a = &mapped_locs[i];
        for j in 0..n {
            let b = &mapped_locs[j];
            for k in 0..n {
                let c = &mapped_locs[k];
                if a < b && b < c {
                    assert!(a < c);
                }
                if a > b && b > c {
                    assert!(a > c);
                }
                if a == b && b == c {
                    if a != c {
                        dbg!(a.src_loc);
                        dbg!(b.src_loc);
                        dbg!(c.src_loc);
                    }
                    assert!(a == c);
                }
            }
        }
        if i % 10 == 0 {
            println!("{i}");
        }
    }
}
> cargo run --release
...
[src/main.rs:177:25] a.src_loc = SrcLoc {
    fileid: 1,
    line: 31,
    column: 1,
}
[src/main.rs:178:25] b.src_loc = SrcLoc {
    fileid: 2,
    line: 214,
    column: 1,
}
[src/main.rs:179:25] c.src_loc = SrcLoc {
    fileid: 1,
    line: 32,
    column: 1,
}
...

@kkysen kkysen changed the title [Rust 1.81.0] user-provided comparison function does not correctly implement a total order SrcLoc sorting is non-transitive and a false total order and equality (panics in 1.81) Sep 17, 2024
@Kriskras99
Copy link
Contributor Author

If you also dbg! the include path the problem becomes clear:

[src/main.rs:177:25] a.src_loc = SrcLoc {
    fileid: 5,
    line: 5,
    column: 1,
}
[src/main.rs:178:25] &include_map[file_map[a.src_loc.fileid as usize]][..] = [
    SrcLoc {
        fileid: 5,
        line: 6,
        column: 10,
    },
]
[src/main.rs:179:25] b.src_loc = SrcLoc {
    fileid: 2,
    line: 4,
    column: 1,
}
[src/main.rs:180:25] &include_map[file_map[b.src_loc.fileid as usize]][..] = [
    SrcLoc {
        fileid: 2,
        line: 6,
        column: 10,
    },
]
[src/main.rs:181:25] c.src_loc = SrcLoc {
    fileid: 5,
    line: 3,
    column: 7,
}
[src/main.rs:182:25] &include_map[file_map[c.src_loc.fileid as usize]][..] = [
    SrcLoc {
        fileid: 5,
        line: 6,
        column: 10,
    },
]
thread 'main' panicked at src/main.rs:184:21:
assertion failed: a == c

a == b because a and b are from different files but are included at the same position. Same for b == c.
But a != c because a and c are from the same file so they are checked not for their include but their actual location.

If you change the start of compare_src_locs to:

    fn cmp_pos(a: &SrcLoc, b: &SrcLoc) -> Ordering {
        (a.line, a.column).cmp(&(b.line, b.column))
    }
    fn cmp_pos_include(a: &SrcLoc, b: &SrcLoc) -> Ordering {
        (a.fileid, a.line, a.column).cmp(&(b.fileid, b.line, b.column))
    }
    let path_a = &include_map[file_map[a.fileid as usize]][..];
    let path_b = &include_map[file_map[b.fileid as usize]][..];
    for (include_a, include_b) in path_a.iter().zip(path_b.iter()) {
        if include_a.fileid != include_b.fileid {
            return cmp_pos_include(include_a, include_b);
        }
    }

Then everything works. But that does go against the comment /// Compare `self` with `other`, without regard to file id, but I don't know why fileid would be excluded

If this is the correct way to fix this, I'll create a nicer patch and open a pull request

@kkysen
Copy link
Contributor

kkysen commented Sep 17, 2024

If you also dbg! the include path the problem becomes clear:
...
a == b because a and b are from different files but are included at the same position. Same for b == c. But a != c because a and c are from the same file so they are checked not for their include but their actual location.

Yeah, I realized that, too. I'm not sure why we're doing that.

If you change the start of compare_src_locs to:

    fn cmp_pos(a: &SrcLoc, b: &SrcLoc) -> Ordering {
        (a.line, a.column).cmp(&(b.line, b.column))
    }
    fn cmp_pos_include(a: &SrcLoc, b: &SrcLoc) -> Ordering {
        (a.fileid, a.line, a.column).cmp(&(b.fileid, b.line, b.column))
    }
    let path_a = &include_map[file_map[a.fileid as usize]][..];
    let path_b = &include_map[file_map[b.fileid as usize]][..];
    for (include_a, include_b) in path_a.iter().zip(path_b.iter()) {
        if include_a.fileid != include_b.fileid {
            return cmp_pos_include(include_a, include_b);
        }
    }

Note also that cmp_pos_includes can be just .cmp as it's the same as the #[derive(Ord)].

And I changed the .clone()s of Vec<SrcLoc>s for the path_* variables to just slicing, as we don't need the clones there. It only came up as a performance bottleneck when brute forcing the transitivity checks, but it's better to just not do a clone, so you could include that change as a separate commit as well.

Then everything works. But that does go against the comment /// Compare `self` with `other`, without regard to file id, but I don't know why fileid would be excluded

If this is the correct way to fix this, I'll create a nicer patch and open a pull request

I'm not sure either. Let me look into it.

@kkysen
Copy link
Contributor

kkysen commented Sep 17, 2024

I think it's for sure more correct than the current order, so you're welcome to open a PR for it.

Kriskras99 added a commit to Kriskras99/c2rust that referenced this issue Sep 18, 2024
This implementation is simplified compared to the previous one.
It is also almost twice as slow in the exhaustive test (15 vs 25 seconds) in
immunant#1126 (comment)
However, in real sort usage the impact should be significantly less.

Fixes immunant#1126
@Kriskras99
Copy link
Contributor Author

I've created a patch in #1128 where cmp_pos isn't used at all anymore, and I've added documentation to the function to try to explain what it's doing

Kriskras99 added a commit to Kriskras99/c2rust that referenced this issue Oct 22, 2024
This implementation is simplified compared to the previous one.
It is also almost twice as slow in the exhaustive test (15 vs 25 seconds) in
immunant#1126 (comment)
However, in real sort usage the impact should be significantly less.

Fixes immunant#1126
Kriskras99 added a commit to Kriskras99/c2rust that referenced this issue Oct 23, 2024
This implementation is simplified compared to the previous one.
It is also almost twice as slow in the exhaustive test (15 vs 25 seconds) in
immunant#1126 (comment)
However, in real sort usage the impact should be significantly less.

Fixes immunant#1126
kkysen added a commit that referenced this issue Oct 24, 2024
…otal order (#1128)

This implementation is simplified compared to the previous one. It is
also almost twice as slow in [the exhaustive
test](#1126 (comment))
(15 vs 25 seconds).
However, in real sort usage the impact should be significantly less.

* Fixes #1126
@xuxuck
Copy link

xuxuck commented Dec 7, 2024

I have observed this error again in the current git version (78c32c5). I also tried 0d2c6c2 e.g. #1128 which worked.
Steps to reproduce:

cargo install --git https://github.com/immunant/c2rust.git c2rust --locked --rev 78c32c5 --profile dev
git clone https://github.com/xiph/opus.git
cd opus
./autogen.sh
./configure
bear -- make
c2rust transpile compile_commands.json

@kkysen
Copy link
Contributor

kkysen commented Dec 8, 2024

@xuxuck, thanks for finding this! Ugh though, I thought it was finally fixed. I'll take another look soon.

@kkysen kkysen reopened this Dec 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants