New helper for sticking Serde-encodable data into arrow

[jleibs]

What

We're about to add a bunch more arrow components for managing blueprints.

Long term it would be nice for this to all be arrow schemas, but all of our state can already be represented with Serde, and we can move faster by just shimming that data through the store.

This gives us a nice escape hatch by adding an arrow_field helper that lets us add any serde-compatible data as a field of a component.

Exa:

use re_log_types::serde_field::SerdeField;
use arrow2_convert::{ArrowDeserialize, ArrowField, ArrowSerialize, field::ArrowField};
use arrow2::datatypes::{DataType, Field};

#[derive(Clone, Debug, PartialEq, serde::Deserialize, serde::Serialize)]
struct SomeStruct {
    foo: String,
    bar: u32,
}

#[derive(Clone, Debug, PartialEq, ArrowField, ArrowSerialize, ArrowDeserialize)]
struct SomeArrowStruct {
    #[arrow_field(type = "SerdeField<SomeStruct>")]
    field: SomeStruct,
}

Checklist

• I have read and agree to Contributor Guide and the Code of Conduct
• I've included a screenshot or gif (if applicable)

PR Build Summary: https://build.rerun.io/pr/2004

[emilk]

I wonder how we should handle the fact that we use MsgPack as the "codec". There are alternatives, like bincode, we could be using, and might want to switch too in the future.

We could, for instance, add an enum Codec { MsgPack = 1 } and put that as the first byte of the encoded array.

[jleibs]

I think the "right" thing to do in theory is to add this to the arrow meta-data for the field. We've been wanting to use that for some other stuff as well so this might be a good opportunity to investigate the plumbing.

[teh-cmc]

The extra struct layer will add a lot of unnecessary noise all over; and since people are most likely going to copy-paste this as a starting point, I'd rather show them how to avoid that straight from the start:

Suggested change

	/// #[derive(Clone, Debug, PartialEq, ArrowField, ArrowSerialize, ArrowDeserialize)]
	/// struct SomeArrowStruct {
	/// #[arrow_field(type = "SerdeField<SomeStruct>")]
	/// field: SomeStruct,
	/// }
	///
	/// assert_eq!(
	/// SomeArrowStruct::data_type(),
	/// DataType::Struct(vec![
	/// Field::new("field", DataType::Binary, false),
	/// ])
	/// );
	/// #[derive(Clone, Debug, PartialEq, ArrowField, ArrowSerialize, ArrowDeserialize)]
	/// #[arrow_field(transparent)]
	/// struct SomeStructArrow(#[arrow_field(type = "SerdeField<SomeStruct>")] SomeStruct);
	///
	/// assert_eq!(SomeStructArrow::data_type(), DataType::Binary);

[jleibs]

Not sure I fully agree. Demoing that this can be done on just a sub-field is also helpful. But onboard with making the docstring shorter.

[emilk]

We should add crate::profile_function!() probes to the serialization and deserialization