What is the provenance of an int-to-ptr result for not strictly inbounds live pointers?

[RalfJung]

When casting an integer back to a pointer as part of a roundtrip, we have to 'cook up' the provenance information that was lost on the ptr-to-int cast. In particular, we have to decide which 'allocation' the pointer is associated with -- or we have to decide to use a more general notion of provenance. (This is orthogonal to the extra provenance imbued by the aliasing model, e.g. Stacked Borrow's tags.) If the pointer is strictly inbounds of a live allocation, it seems pretty clear that that is the provenance we want to assign. But for other pointers that is much less clear.

C mostly avoids this problem by declaring such casts UB -- except for pointers at the top edge of an allocation ("one-past-the-end pointers"), where there is ambiguity whether they point to the allocation that ends at that address, or the one that starts there. The PNVI proposal introduces some fancy and complicated machinery (involving a global table of 'symbolic' provenances) to basically delay that choice as long as possible.

The twinsem paper uses a much simpler approach of recording which offsets need to be inbound, and then checking that constraint on the next load/store. This scales to languages that allow casting more pointers, but does not entirely match how the LLVM semantics are described in the LangRef. The analyses LLVM does are fine (to the extent that the paper checked them, anyway), but this allows some programs LLVM considers UB.

Both of these options would be rather invasive to implement in Miri. Currently, Miri uses a kind of heuristic, but that sometimes leads to problems (rust-lang/miri#1866).

I wish...

...we could just rule out int-to-ptr casts and make everyone use this function instead:

// Converts 'addr' to a pointer using the provenance of 'prov'.
fn int_to_ptr_with_provenance<T>(addr: usize, prov: *const T) -> *const T {
  let ptr = prov.cast::<u8>();
  ptr.wrapping_add(addr.wrapping_sub(ptr as usize)).cast()
}

Alas, that's just a dream.

[Lokathor]

The pointers for this topic are separate from constant pointer addresses, i take it?

[RalfJung]

If by 'constant pointer addresses' you mean hard-coded fixed addresses, then yes. This mostly concerns roundtrips, where one starts with a ptr, obtains an int, does something to it (or not), and casts back.

Hard-coded addresses, if they do not alias with any address 'managed' by the Rust Abstract machine, can just use a provenance that indicates exactly that -- a provenance that is valid only for such 'outside the machine' addresses. That is much less of a problem.

[RalfJung]

The from_exposed_ptr spec gives a precise answer for what the provenance of an int2ptr result is. So that is one possible answer to this question.

(the above link is broken, https://doc.rust-lang.org/1.63.0/std/ptr/fn.from_exposed_addr.html may be a good substitute as it is for the first stable that branched after this comment was written)

[theemathas]

The link to from_exposed_ptr is a dead link, and the with_exposed_provenance docs are currently unclear on the answer. So... what is the provenance of an out-of-bounds int2ptr cast? Can it get a provenance far away from the address?

[theemathas]

If the pointer is strictly inbounds of a live allocation, it seems pretty clear that that is the provenance we want to assign.

This is not obvious to me. An address given to with_exposed_provenance could accidentally land inside one allocation, and then later it is wrapping_offset into a different allocation and then dereferenced.

[RalfJung]

the with_exposed_provenance docs are currently unclear on the answer.

They are actually very clear on the answer -- it will pick whatever provenance makes your code work. So yes it can get the provenance of a far away address, if that is what it takes.

However, rust-lang/rust#130350 changes these docs to remove pretty much any guarantees.

An address given to with_exposed_provenance could accidentally land inside one allocation, and then later it is wrapping_offset into a different allocation and then dereferenced.

It is not at all clear to me that this is a usecase we want to support.

[RalfJung]

Anyway this issue predates the entire strict provenance / exposed provenance saga (though the API I "wish" for in the issue is indeed exactly with_addr :-). #392 better reflects the current open question.