Arbitrary data in Memory using Any, fix #255 (#257)

* init work

* implement deferred deserialization

* many improvements
* rename `DataElement` -> `AnyMapElement`
* make `data` in `Memory` as public field of type with public interface
* make interface more rich
* transform most unwraps to proper error handling
* make `AnyMap` store by `TypeId`, so now individual type can be counted and reset

* improve storing TypeId between different rust versions

* rewrite system widgets to use AnyMap

* refactor everything
* replace `serde_json` -> `ron`
* move `any_map` to module
* separate `AnyMap` into `AnyMapId` and `serializable::AnyMapId` in order to not require `serde` traits in methods
* add `AnyMap` and `serializable::AnyMap` that stores elements just by type, without `Id`
* write documentation
* change tooltips and color picker to use `Memory::data_temp`

* fix bugs and docs

* Apply suggestions from code review

Co-authored-by: Emil Ernerfeldt <[email protected]>

* rename `AnyMap` → `TypeMap`

* rename `AnyMapId` → `AnyMap`, add generic <Key> to it

* rename files `id_map` → `any_map`

* move out usages from `serializable` mod

* rename `ToDeserialize` → `Serialized`

* fix bug with counting

* add tests, and...
* rename `reset` → `remove`
* add function `remove_by_type`
* format code

* improve code
* make identical interface for serialized and simple maps
* make serialized maps serialized fully, without features by moving this into `Memory` struct with `#[cfg(feature = "persistence")]` under fields
* move `serialized::TypeId` into `AnyMapElement` struct

* fix pipeline and add one more test

* update docs

Co-authored-by: Emil Ernerfeldt <[email protected]>

Author: optozorax

Cargo.lock

@@ -22,6 +22,7 @@ include = [
 [dependencies]
 epaint = { version = "0.11.0", path = "../epaint", default-features = false }
 serde = { version = "1", features = ["derive", "rc"], optional = true }
+ron = { version = "0.6.4", optional = true }
 
 [features]
 default = ["default_fonts", "single_threaded"]
@@ -30,8 +31,11 @@ default = ["default_fonts", "single_threaded"]
 # If you plan on specifying your own fonts you may disable this feature.
 default_fonts = ["epaint/default_fonts"]
 
-persistence = ["serde", "epaint/persistence"]
+persistence = ["serde", "epaint/persistence", "ron"]
 
 # Only needed if you plan to use the same egui::Context from multiple threads.
 single_threaded = ["epaint/single_threaded"]
 multi_threaded = ["epaint/multi_threaded"]
+
+[dev-dependencies]
+serde_json = "1"

egui/Cargo.toml

@@ -0,0 +1,205 @@
+use crate::any::element::{AnyMapElement, AnyMapTrait};
+use std::any::TypeId;
+use std::collections::HashMap;
+use std::hash::Hash;
+
+/// Stores any object by `Key`.
+#[derive(Clone, Debug)]
+pub struct AnyMap<Key: Hash + Eq>(HashMap<Key, AnyMapElement>);
+
+impl<Key: Hash + Eq> Default for AnyMap<Key> {
+    fn default() -> Self {
+        AnyMap(HashMap::new())
+    }
+}
+
+// ----------------------------------------------------------------------------
+
+impl<Key: Hash + Eq> AnyMap<Key> {
+    pub fn get<T: AnyMapTrait>(&mut self, key: &Key) -> Option<&T> {
+        self.get_mut(key).map(|x| &*x)
+    }
+
+    pub fn get_mut<T: AnyMapTrait>(&mut self, key: &Key) -> Option<&mut T> {
+        self.0.get_mut(key)?.get_mut()
+    }
+}
+
+impl<Key: Hash + Eq> AnyMap<Key> {
+    pub fn get_or_insert_with<T: AnyMapTrait>(
+        &mut self,
+        key: Key,
+        or_insert_with: impl FnOnce() -> T,
+    ) -> &T {
+        &*self.get_mut_or_insert_with(key, or_insert_with)
+    }
+
+    pub fn get_or_default<T: AnyMapTrait + Default>(&mut self, key: Key) -> &T {
+        self.get_or_insert_with(key, Default::default)
+    }
+
+    pub fn get_mut_or_insert_with<T: AnyMapTrait>(
+        &mut self,
+        key: Key,
+        or_insert_with: impl FnOnce() -> T,
+    ) -> &mut T {
+        use std::collections::hash_map::Entry;
+        match self.0.entry(key) {
+            Entry::Vacant(vacant) => vacant
+                .insert(AnyMapElement::new(or_insert_with()))
+                .get_mut()
+                .unwrap(), // this unwrap will never panic, because we insert correct type right now
+            Entry::Occupied(occupied) => occupied.into_mut().get_mut_or_set_with(or_insert_with),
+        }
+    }
+
+    pub fn get_mut_or_default<T: AnyMapTrait + Default>(&mut self, key: Key) -> &mut T {
+        self.get_mut_or_insert_with(key, Default::default)
+    }
+}
+
+impl<Key: Hash + Eq> AnyMap<Key> {
+    pub fn insert<T: AnyMapTrait>(&mut self, key: Key, element: T) {
+        self.0.insert(key, AnyMapElement::new(element));
+    }
+
+    pub fn remove(&mut self, key: &Key) {
+        self.0.remove(key);
+    }
+
+    pub fn remove_by_type<T: AnyMapTrait>(&mut self) {
+        let key = TypeId::of::<T>();
+        self.0.retain(|_, v| v.type_id() != key);
+    }
+
+    pub fn clear(&mut self) {
+        self.0.clear();
+    }
+}
+
+impl<Key: Hash + Eq> AnyMap<Key> {
+    /// You could use this function to find is there some leak or misusage.
+    pub fn count<T: AnyMapTrait>(&mut self) -> usize {
+        let key = TypeId::of::<T>();
+        self.0.iter().filter(|(_, v)| v.type_id() == key).count()
+    }
+
+    pub fn count_all(&mut self) -> usize {
+        self.0.len()
+    }
+}
+
+// ----------------------------------------------------------------------------
+
+#[cfg(test)]
+#[test]
+fn basic_usage() {
+    #[derive(Debug, Clone, Eq, PartialEq, Default)]
+    struct State {
+        a: i32,
+    }
+
+    let mut map: AnyMap<i32> = Default::default();
+
+    assert!(map.get::<State>(&0).is_none());
+    map.insert(0, State { a: 42 });
+
+    assert_eq!(*map.get::<State>(&0).unwrap(), State { a: 42 });
+    assert!(map.get::<State>(&1).is_none());
+    map.get_mut::<State>(&0).unwrap().a = 43;
+    assert_eq!(*map.get::<State>(&0).unwrap(), State { a: 43 });
+
+    map.remove(&0);
+    assert!(map.get::<State>(&0).is_none());
+
+    assert_eq!(
+        *map.get_or_insert_with(0, || State { a: 55 }),
+        State { a: 55 }
+    );
+    map.remove(&0);
+    assert_eq!(
+        *map.get_mut_or_insert_with(0, || State { a: 56 }),
+        State { a: 56 }
+    );
+    map.remove(&0);
+    assert_eq!(*map.get_or_default::<State>(0), State { a: 0 });
+    map.remove(&0);
+    assert_eq!(*map.get_mut_or_default::<State>(0), State { a: 0 });
+}
+
+#[cfg(test)]
+#[test]
+fn different_type_same_id() {
+    #[derive(Debug, Clone, Eq, PartialEq, Default)]
+    struct State {
+        a: i32,
+    }
+
+    let mut map: AnyMap<i32> = Default::default();
+
+    map.insert(0, State { a: 42 });
+
+    assert_eq!(*map.get::<State>(&0).unwrap(), State { a: 42 });
+    assert!(map.get::<i32>(&0).is_none());
+
+    map.insert(0, 255i32);
+
+    assert_eq!(*map.get::<i32>(&0).unwrap(), 255);
+    assert!(map.get::<State>(&0).is_none());
+}
+
+#[cfg(test)]
+#[test]
+fn cloning() {
+    #[derive(Debug, Clone, Eq, PartialEq, Default)]
+    struct State {
+        a: i32,
+    }
+
+    let mut map: AnyMap<i32> = Default::default();
+
+    map.insert(0, State::default());
+    map.insert(10, 10i32);
+    map.insert(11, 11i32);
+
+    let mut cloned_map = map.clone();
+
+    map.insert(12, 12i32);
+    map.insert(1, State { a: 10 });
+
+    assert_eq!(*cloned_map.get::<State>(&0).unwrap(), State { a: 0 });
+    assert!(cloned_map.get::<State>(&1).is_none());
+    assert_eq!(*cloned_map.get::<i32>(&10).unwrap(), 10i32);
+    assert_eq!(*cloned_map.get::<i32>(&11).unwrap(), 11i32);
+    assert!(cloned_map.get::<i32>(&12).is_none());
+}
+
+#[cfg(test)]
+#[test]
+fn counting() {
+    #[derive(Debug, Clone, Eq, PartialEq, Default)]
+    struct State {
+        a: i32,
+    }
+
+    let mut map: AnyMap<i32> = Default::default();
+
+    map.insert(0, State::default());
+    map.insert(1, State { a: 10 });
+    map.insert(10, 10i32);
+    map.insert(11, 11i32);
+    map.insert(12, 12i32);
+
+    assert_eq!(map.count::<State>(), 2);
+    assert_eq!(map.count::<i32>(), 3);
+
+    map.remove_by_type::<State>();
+
+    assert_eq!(map.count::<State>(), 0);
+    assert_eq!(map.count::<i32>(), 3);
+
+    map.clear();
+
+    assert_eq!(map.count::<State>(), 0);
+    assert_eq!(map.count::<i32>(), 0);
+}

egui/src/any/any_map.rs

@@ -0,0 +1,61 @@
+use std::any::{Any, TypeId};
+use std::fmt;
+
+/// Like [`std::any::Any`], but also implements `Clone`.
+pub(crate) struct AnyMapElement {
+    value: Box<dyn Any + 'static>,
+    clone_fn: fn(&Box<dyn Any + 'static>) -> Box<dyn Any + 'static>,
+}
+
+impl fmt::Debug for AnyMapElement {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        f.debug_struct("AnyMapElement")
+            .field("value_type_id", &self.type_id())
+            .finish()
+    }
+}
+
+impl Clone for AnyMapElement {
+    fn clone(&self) -> Self {
+        AnyMapElement {
+            value: (self.clone_fn)(&self.value),
+            clone_fn: self.clone_fn,
+        }
+    }
+}
+
+pub trait AnyMapTrait: 'static + Any + Clone {}
+
+impl<T: 'static + Any + Clone> AnyMapTrait for T {}
+
+impl AnyMapElement {
+    pub(crate) fn new<T: AnyMapTrait>(t: T) -> Self {
+        AnyMapElement {
+            value: Box::new(t),
+            clone_fn: |x| {
+                let x = x.downcast_ref::<T>().unwrap(); // This unwrap will never panic, because we always construct this type using this `new` function and because we return &mut reference only with type `T`, so type cannot change.
+                Box::new(x.clone())
+            },
+        }
+    }
+
+    pub(crate) fn type_id(&self) -> TypeId {
+        (*self.value).type_id()
+    }
+
+    pub(crate) fn get_mut<T: AnyMapTrait>(&mut self) -> Option<&mut T> {
+        self.value.downcast_mut()
+    }
+
+    pub(crate) fn get_mut_or_set_with<T: AnyMapTrait>(
+        &mut self,
+        set_with: impl FnOnce() -> T,
+    ) -> &mut T {
+        if !self.value.is::<T>() {
+            *self = Self::new(set_with());
+            // TODO: log this error, because it can occurs when user used same Id or same type for different widgets
+        }
+
+        self.value.downcast_mut().unwrap() // This unwrap will never panic because we already converted object to required type
+    }
+}

egui/src/any/element.rs

@@ -0,0 +1,61 @@
+//! Any-type storages for [`Memory`].
+//!
+//! This module contains structs to store arbitrary types using [`Any`] trait. Also, they can be cloned, and structs in [`serializable`] can be de/serialized.
+//!
+//! All this is just `HashMap<TypeId, Box<dyn Any + static>>` and `HashMap<Key, Box<dyn Any + static>>`, but with helper functions and hacks for cloning and de/serialization.
+//!
+//! # Trait requirements
+//!
+//! If you want to store your type here, it must implement `Clone` and `Any` and be `'static`, which means it must not contain references. If you want to store your data in serializable storage, it must implement `serde::Serialize` and `serde::Deserialize` under the `persistent` feature.
+//!
+//! # [`TypeMap`]
+//!
+//! It stores everything by just type. You should use this map for your widget when all instances of your widgets can have only one state. E.g. for popup windows, for color picker.
+//!
+//! To not have intersections, you should create newtype for anything you try to store here, like:
+//! ```rust
+//! struct MyEditBool(pub bool);
+//! ```
+//!
+//! # [`AnyMap<Key>`]
+//!
+//! In [`Memory`] `Key` = [`Id`].
+//!
+//! [`TypeMap`] and [`AnyMap<Key>`] has a quite similar interface, except for [`AnyMap`] you should pass `Key` to get and insert things.
+//!
+//! It stores everything by `Key`, this should be used when your widget can have different data for different instances of the widget.
+//!
+//! # `serializable`
+//!
+//! [`TypeMap`] and [`serializable::TypeMap`] has exactly the same interface, but [`serializable::TypeMap`] only requires serde traits for stored object under `persistent` feature. Same thing for [`AnyMap`] and [`serializable::AnyMap`].
+//!
+//! # What could break
+//!
+//! Things here could break only when you trying to load this from file.
+//!
+//! First, serialized `TypeId` in [`serializable::TypeMap`] could broke if you updated the version of the Rust compiler between runs.
+//!
+//! Second, count and reset all instances of a type in [`serializable::AnyMap`] could return an incorrect value for the same reason.
+//!
+//! Deserialization errors of loaded elements of these storages can be determined only when you call `get_...` functions, they not logged and not provided to a user, on this errors value is just replaced with `or_insert()`/default value.
+//!
+//! # When not to use this
+//!
+//! This is not for important widget data. Some errors are just ignored and the correct value of type is inserted when you call. This is done to more simple interface.
+//!
+//! You shouldn't use any map here when you need very reliable state storage with rich error-handling. For this purpose you should create your own `Memory` struct and pass it everywhere you need it. Then, you should de/serialize it by yourself, handling all serialization or other errors as you wish.
+//!
+//! [`Id`]: crate::Id
+//! [`Memory`]: crate::Memory
+//! [`Any`]: std::any::Any
+//! [`AnyMap<Key>`]: crate::any::AnyMap
+
+mod any_map;
+mod element;
+mod type_map;
+
+/// Same structs and traits, but also can be de/serialized under `persistence` feature.
+#[cfg(feature = "persistence")]
+pub mod serializable;
+
+pub use self::{any_map::AnyMap, element::AnyMapTrait, type_map::TypeMap};