Auto merge of #2059 - carbotaniuman:master, r=RalfJung

Initial work on Miri permissive-exposed-provenance

Miri portions of the changes for portions of a permissive ptr-to-int model for Miri. This is more restrictive than what we currently have so it will probably need a flag once I figure out how to hook that up.

> This implements a form of permissive exposed-address provenance, wherein the only way to expose the address is with a cast to usize (ideally expose_addr). This is more restrictive than C in that stuff like reading the representation bytes (via unions, type-punning, transmute) does not expose the address, only expose_addr. This is less restrictive than C in that a pointer casted from an integer has union provenance of all exposed pointers, not any udi stuff.

There's a few TODOs here, namely related to `fn memory_read` and friends. We pass it the maybe/unreified provenance before `ptr_get_alloc` reifies it into a concrete one, so it doesn't have the `AllocId` (or the SB tag, but that's getting ahead of ourselves). One way this could be fixed is changing `ptr_get_alloc` and (`ptr_try_get_alloc_id` on the rustc side) to return a pointer with the tag fixed up. We could also take in different arguments, but I'm not sure what works best.

The other TODOs here are how permissive this model could be. This currently does not enforce that a ptr-to-int cast happens before the corresponding int-to-ptr (colloquial meaning of happens before, not atomic meaning). Example:

```
let ptr = 0x2000 as *const i32;
let a: i32 = 5;
let a_ptr = &a as *const i32;

// value is 0x2000;
a_ptr as usize;

println!("{}", unsafe { *ptr }); // this is valid
```

We also allow the resulting pointer to dereference different non-contiguous allocations (the "not any udi stuff" mentioned above), which I'm not sure if is allowed by LLVM.

This is the Miri side of rust-lang/rust#95826.

README.md

@@ -318,6 +318,17 @@ to Miri failing to detect cases of undefined behavior in a program.
   application instead of raising an error within the context of Miri (and halting
   execution). Note that code might not expect these operations to ever panic, so
   this flag can lead to strange (mis)behavior.
+* `-Zmiri-permissive-provenance` is **experimental**. This will make Miri do a
+  best-effort attempt to implement the semantics of
+  [`expose_addr`](https://doc.rust-lang.org/nightly/std/primitive.pointer.html#method.expose_addr)
+  and
+  [`ptr::from_exposed_addr`](https://doc.rust-lang.org/nightly/std/ptr/fn.from_exposed_addr.html)
+  for pointer-to-int and int-to-pointer casts, respectively. This will
+  necessarily miss some bugs as those semantics are not efficiently
+  implementable in a sanitizer, but it will only miss bugs that concerns
+  memory/pointers which is subject to these operations. Also note that this flag
+  is currently incompatible with Stacked Borrows, so you will have to also pass
+  `-Zmiri-disable-stacked-borrows` to use this.
 * `-Zmiri-seed=<hex>` configures the seed of the RNG that Miri uses to resolve
   non-determinism.  This RNG is used to pick base addresses for allocations.
   When isolation is enabled (the default), this is also used to emulate system

src/bin/miri.rs

@@ -30,7 +30,7 @@ use rustc_middle::{
 };
 use rustc_session::{config::ErrorOutputType, search_paths::PathKind, CtfeBacktrace};
 
-use miri::BacktraceStyle;
+use miri::{BacktraceStyle, ProvenanceMode};
 
 struct MiriCompilerCalls {
     miri_config: miri::MiriConfig,
@@ -384,10 +384,14 @@ fn main() {
                     miri_config.tag_raw = true;
                 }
                 "-Zmiri-strict-provenance" => {
-                    miri_config.strict_provenance = true;
+                    miri_config.provenance_mode = ProvenanceMode::Strict;
                     miri_config.tag_raw = true;
                     miri_config.check_number_validity = true;
                 }
+                "-Zmiri-permissive-provenance" => {
+                    miri_config.provenance_mode = ProvenanceMode::Permissive;
+                    miri_config.tag_raw = true;
+                }
                 "-Zmiri-mute-stdout-stderr" => {
                     miri_config.mute_stdout_stderr = true;
                 }

src/eval.rs

@@ -113,9 +113,8 @@ pub struct MiriConfig {
     pub panic_on_unsupported: bool,
     /// Which style to use for printing backtraces.
     pub backtrace_style: BacktraceStyle,
-    /// Whether to enforce "strict provenance" rules. Enabling this means int2ptr casts return
-    /// pointers with an invalid provenance, i.e., not valid for any memory access.
-    pub strict_provenance: bool,
+    /// Which provenance to use for int2ptr casts
+    pub provenance_mode: ProvenanceMode,
     /// Whether to ignore any output by the program. This is helpful when debugging miri
     /// as its messages don't get intermingled with the program messages.
     pub mute_stdout_stderr: bool,
@@ -144,7 +143,7 @@ impl Default for MiriConfig {
             measureme_out: None,
             panic_on_unsupported: false,
             backtrace_style: BacktraceStyle::Short,
-            strict_provenance: false,
+            provenance_mode: ProvenanceMode::Legacy,
             mute_stdout_stderr: false,
         }
     }

src/helpers.rs

@@ -786,8 +786,8 @@ pub trait EvalContextExt<'mir, 'tcx: 'mir>: crate::MiriEvalContextExt<'mir, 'tcx
     fn mark_immutable(&mut self, mplace: &MemPlace<Tag>) {
         let this = self.eval_context_mut();
         // This got just allocated, so there definitely is a pointer here.
-        this.alloc_mark_immutable(mplace.ptr.into_pointer_or_addr().unwrap().provenance.alloc_id)
-            .unwrap();
+        let provenance = mplace.ptr.into_pointer_or_addr().unwrap().provenance;
+        this.alloc_mark_immutable(provenance.get_alloc_id().unwrap()).unwrap();
     }
 
     fn item_link_name(&self, def_id: DefId) -> Symbol {

src/intptrcast.rs

@@ -4,11 +4,25 @@ use std::collections::hash_map::Entry;
 use log::trace;
 use rand::Rng;
 
-use rustc_data_structures::fx::FxHashMap;
+use rustc_data_structures::fx::{FxHashMap, FxHashSet};
 use rustc_target::abi::{HasDataLayout, Size};
 
 use crate::*;
 
+#[derive(Copy, Clone, Debug, PartialEq, Eq)]
+pub enum ProvenanceMode {
+    /// Int2ptr casts return pointers with "wildcard" provenance
+    /// that basically matches that of all exposed pointers
+    /// (and SB tags, if enabled).
+    Permissive,
+    /// Int2ptr casts return pointers with an invalid provenance,
+    /// i.e., not valid for any memory access.
+    Strict,
+    /// Int2ptr casts determine the allocation they point to at cast time.
+    /// All allocations are considered exposed.
+    Legacy,
+}
+
 pub type GlobalState = RefCell<GlobalStateInner>;
 
 #[derive(Clone, Debug)]
@@ -21,35 +35,37 @@ pub struct GlobalStateInner {
     /// they do not have an `AllocExtra`.
     /// This is the inverse of `int_to_ptr_map`.
     base_addr: FxHashMap<AllocId, u64>,
+    /// Whether an allocation has been exposed or not. This cannot be put
+    /// into `AllocExtra` for the same reason as `base_addr`.
+    exposed: FxHashSet<AllocId>,
     /// This is used as a memory address when a new pointer is casted to an integer. It
     /// is always larger than any address that was previously made part of a block.
     next_base_addr: u64,
-    /// Whether to enforce "strict provenance" rules. Enabling this means int2ptr casts return
-    /// pointers with an invalid provenance, i.e., not valid for any memory access.
-    strict_provenance: bool,
+    /// The provenance to use for int2ptr casts
+    provenance_mode: ProvenanceMode,
 }
 
 impl GlobalStateInner {
     pub fn new(config: &MiriConfig) -> Self {
         GlobalStateInner {
             int_to_ptr_map: Vec::default(),
             base_addr: FxHashMap::default(),
+            exposed: FxHashSet::default(),
             next_base_addr: STACK_ADDR,
-            strict_provenance: config.strict_provenance,
+            provenance_mode: config.provenance_mode,
         }
     }
 }
 
 impl<'mir, 'tcx> GlobalStateInner {
-    pub fn ptr_from_addr(addr: u64, ecx: &MiriEvalContext<'mir, 'tcx>) -> Pointer<Option<Tag>> {
-        trace!("Casting 0x{:x} to a pointer", addr);
+    // Returns the exposed `AllocId` that corresponds to the specified addr,
+    // or `None` if the addr is out of bounds
+    fn alloc_id_from_addr(ecx: &MiriEvalContext<'mir, 'tcx>, addr: u64) -> Option<AllocId> {
         let global_state = ecx.machine.intptrcast.borrow();
-
-        if global_state.strict_provenance {
-            return Pointer::new(None, Size::from_bytes(addr));
-        }
+        assert!(global_state.provenance_mode != ProvenanceMode::Strict);
 
         let pos = global_state.int_to_ptr_map.binary_search_by_key(&addr, |(addr, _)| *addr);
+
         let alloc_id = match pos {
             Ok(pos) => Some(global_state.int_to_ptr_map[pos].1),
             Err(0) => None,
@@ -60,6 +76,7 @@ impl<'mir, 'tcx> GlobalStateInner {
                 // This never overflows because `addr >= glb`
                 let offset = addr - glb;
                 // If the offset exceeds the size of the allocation, don't use this `alloc_id`.
+
                 if offset
                     <= ecx
                         .get_alloc_size_and_align(alloc_id, AllocCheck::MaybeDead)
@@ -72,12 +89,65 @@ impl<'mir, 'tcx> GlobalStateInner {
                     None
                 }
             }
-        };
-        // Pointers created from integers are untagged.
-        Pointer::new(
-            alloc_id.map(|alloc_id| Tag { alloc_id, sb: SbTag::Untagged }),
-            Size::from_bytes(addr),
-        )
+        }?;
+
+        // In legacy mode, we consider all allocations exposed.
+        if global_state.provenance_mode == ProvenanceMode::Legacy
+            || global_state.exposed.contains(&alloc_id)
+        {
+            Some(alloc_id)
+        } else {
+            None
+        }
+    }
+
+    pub fn expose_addr(ecx: &MiriEvalContext<'mir, 'tcx>, alloc_id: AllocId) {
+        trace!("Exposing allocation id {:?}", alloc_id);
+
+        let mut global_state = ecx.machine.intptrcast.borrow_mut();
+        if global_state.provenance_mode == ProvenanceMode::Permissive {
+            global_state.exposed.insert(alloc_id);
+        }
+    }
+
+    pub fn ptr_from_addr_transmute(
+        ecx: &MiriEvalContext<'mir, 'tcx>,
+        addr: u64,
+    ) -> Pointer<Option<Tag>> {
+        trace!("Transmuting 0x{:x} to a pointer", addr);
+
+        let global_state = ecx.machine.intptrcast.borrow();
+
+        // In legacy mode, we have to support int2ptr transmutes,
+        // so just pretend they do the same thing as a cast.
+        if global_state.provenance_mode == ProvenanceMode::Legacy {
+            Self::ptr_from_addr_cast(ecx, addr)
+        } else {
+            Pointer::new(None, Size::from_bytes(addr))
+        }
+    }
+
+    pub fn ptr_from_addr_cast(
+        ecx: &MiriEvalContext<'mir, 'tcx>,
+        addr: u64,
+    ) -> Pointer<Option<Tag>> {
+        trace!("Casting 0x{:x} to a pointer", addr);
+
+        let global_state = ecx.machine.intptrcast.borrow();
+
+        if global_state.provenance_mode == ProvenanceMode::Strict {
+            Pointer::new(None, Size::from_bytes(addr))
+        } else if global_state.provenance_mode == ProvenanceMode::Legacy {
+            let alloc_id = Self::alloc_id_from_addr(ecx, addr);
+
+            Pointer::new(
+                alloc_id
+                    .map(|alloc_id| Tag::Concrete(ConcreteTag { alloc_id, sb: SbTag::Untagged })),
+                Size::from_bytes(addr),
+            )
+        } else {
+            Pointer::new(Some(Tag::Wildcard), Size::from_bytes(addr))
+        }
     }
 
     fn alloc_base_addr(ecx: &MiriEvalContext<'mir, 'tcx>, alloc_id: AllocId) -> u64 {
@@ -136,14 +206,27 @@ impl<'mir, 'tcx> GlobalStateInner {
         dl.overflowing_offset(base_addr, offset.bytes()).0
     }
 
-    pub fn abs_ptr_to_rel(ecx: &MiriEvalContext<'mir, 'tcx>, ptr: Pointer<Tag>) -> Size {
+    pub fn abs_ptr_to_rel(
+        ecx: &MiriEvalContext<'mir, 'tcx>,
+        ptr: Pointer<Tag>,
+    ) -> Option<(AllocId, Size)> {
         let (tag, addr) = ptr.into_parts(); // addr is absolute (Tag provenance)
-        let base_addr = GlobalStateInner::alloc_base_addr(ecx, tag.alloc_id);
+
+        let alloc_id = if let Tag::Concrete(concrete) = tag {
+            concrete.alloc_id
+        } else {
+            GlobalStateInner::alloc_id_from_addr(ecx, addr.bytes())?
+        };
+
+        let base_addr = GlobalStateInner::alloc_base_addr(ecx, alloc_id);
 
         // Wrapping "addr - base_addr"
         let dl = ecx.data_layout();
         let neg_base_addr = (base_addr as i64).wrapping_neg();
-        Size::from_bytes(dl.overflowing_signed_offset(addr.bytes(), neg_base_addr).0)
+        Some((
+            alloc_id,
+            Size::from_bytes(dl.overflowing_signed_offset(addr.bytes(), neg_base_addr).0),
+        ))
     }
 
     /// Shifts `addr` to make it aligned with `align` by rounding `addr` to the smallest multiple

src/lib.rs

@@ -78,9 +78,10 @@ pub use crate::eval::{
     create_ecx, eval_entry, AlignmentCheck, BacktraceStyle, IsolatedOp, MiriConfig, RejectOpWith,
 };
 pub use crate::helpers::{CurrentSpan, EvalContextExt as HelpersEvalContextExt};
+pub use crate::intptrcast::ProvenanceMode;
 pub use crate::machine::{
-    AllocExtra, Evaluator, FrameData, MiriEvalContext, MiriEvalContextExt, MiriMemoryKind, Tag,
-    NUM_CPUS, PAGE_SIZE, STACK_ADDR, STACK_SIZE,
+    AllocExtra, ConcreteTag, Evaluator, FrameData, MiriEvalContext, MiriEvalContextExt,
+    MiriMemoryKind, Tag, NUM_CPUS, PAGE_SIZE, STACK_ADDR, STACK_SIZE,
 };
 pub use crate::mono_hash_map::MonoHashMap;
 pub use crate::operator::EvalContextExt as OperatorEvalContextExt;