Extending basic_registry

[iK4tsu]

For a specific use-case there's a need to restrict access to basic_registry. There are component relationships that need to be managed carefully, but since these can be generalized the extended registry handles that automatically. A couple of things to have in mind:

• I don't know what components exist
• The specific use-cases must work for any component
• The heavy work should be done internally
• The end-user mustn't do anything manually for it to work
• The registry must be copyable

To handle some of it entt::meta is great! My current approach works alright, however, I would like to improve it. Not only in terms of design but code complexity too.

For example, it makes perfect sense for my current wrapper to be a more specialized version of a basic_registry. I would like to extend it, and override some functionality from the base class to add specific behavior. The perfect example for this is assure. It would be perfect if it could be override and/or accessed from a derived class. For every new assured storage I want to set some meta_type information for the meta_ctx used by this instance. Right now I can do that by adding that extra functionality to all public methods available by the the specialized registry. That's not to bad but could be improved. What if I wanted all the functionality from the base class but with some specific functionality?

What about virtual methods? It wouldn't surprise me if someone wanted to have a container of custom extended basic_registry types, wanted to invoke the same method for all of them and expect it to dispatch correctly.

[skypjack]

The registry must be copyable

This is... huge and very expensive probably. What's the use case for that? I've never really had one that makes sense so far.

For every new assured storage I want to set some meta_type information for the meta_ctx used by this instance.

Why not do this in the storage constructor or when the bind function is invoked from the registry? Mixins fit both cases and are already a thing. You don't have to inherit and override all functions of a registry for that.

There are component relationships that need to be managed carefully

The same is true here. You can add a mixin and do the right thing in the pop function to avoid mistakes. The registry is provided to all storage objects upon construction, therefore you can easily access all other pools if you need to.

What about virtual methods?

Making the registry functions virtual would be a cost that everyone would pay for a few corner cases. Consider that this is the first time it has been requested in years. Literally.

[skypjack]

If I want to use the the current sigh_mixin there's the issue that it always calls get when emplace is used

Ah, that's funny. Long story short, it's meant to support callbacks that actively operate on the storage itself.
Originally, I remember that the problem was from a guy who sorted the storage in the callback. You can see how the returned element can be wrong if we don't get it again.
In any case, you can either disable signals as a whole if you don't use them or just make them part of your own mixin. The sigh_mixin is similar to all others and you don't have to add it necessarily if you don't need it. 🤷‍♂️
Consider that emplace returns a reference to the newly created element. Therefore, unless you have a good reason to re-get it (the sigh_mixin is a general purpose one but you don't have to copy-paste it as-is), you don't need to do so and you can return this reference directly.

[iK4tsu]

Ahh, get and get_as_tuple are not enough for what I was looking for. I phrased it wrong actually. I said sort on read, but technically it is more of a sort when iterating. The order only matters when iterating really. If I try to fetch some component based on some position, what I get is completely correct and fine. If I have some kind of ranged relationship component defining a [from .. to[ range with from and to being entities that own a position, and I try to get it, then that is completely fine as well, if I'm not missing something here. Iterating a storage that must be sorted is what should not happen. However I can't figure out if this is possible, once again, without redefining some of the registry's methods.

// lets call it Position
// sorted by less
struct Position final { size_t value; }

...

registry.emplace<Position>(entt, value);
// storage position is NOT sorted here
for (auto entt : registry.view<Position>())
{
  // storage position must be sorted at this point
}

Edit: Personally I wouldn't mind to have this documented and requiring an explicit sort, however, the requirements are that this has to be abstracted away. It makes sense since it is part of the core behavior, therefore it must be an internal implementation, but it makes finding a solution a bit more complex. 😅

Also, using a similar implementation to the above, it seems that bind isn't called for Position. It was called for int, char, std::string, others..., but not Position.

In relation to what I said here, I was doing owner->storage<Position>() which probably causes a circular dependency which fails the specialization causing the component to use the one given by the library, I think. Using the name overload owner->storage(type_id<Position>().hash()) fixes it.

Therefore, unless you have a good reason to re-get it (the sigh_mixin is a general purpose one but you don't have to copy-paste it as-is), you don't need to do so and you can return this reference directly.

Yeah, I would eventually do my own implementation of the entire mixin because I want the custom registry to be passed into those signals. I also noticed this PR does basically this.

[skypjack]

Sort only before iterating 🤔 I don't think this is possible actually. There is no way to tell if a storage lookup is for iteration purposes or if it happens for any other reasons. For example:

auto &&storage = registry.storage<T>();
storage.emplace(entity);

Versus:

for(auto entt: registry.view<T>()) { /* ... */ }

In theory, you could sort on storage::each but it's fragile imho and not that future proof. A specific and explicit method that does it seems like a better solution to me.

I also noticed this does basically this.

That PR (well, a refined version of it) is available on the wip branch, soon master.

[iK4tsu]

A specific and explicit method that does it seems like a better solution to me.

Yup I agree, but I cannot escape that requirement 😅

There is no way to tell if a storage lookup is for iteration purposes or if it happens for any other reasons.

An OKish solution would be to redefine view storage group to do a sort check. Not ideal but should works.

[skypjack]

To do that, you still need to inherit the registry and reimplement them on your own.
We couldn't make them virtual even if we wanted to (and I wouldn't want to 😅) since they are template functions.

[iK4tsu]

Sorry for having to come back to this discussion. I'm having difficulty in thinking on how to structure this. At the time I filled the team's requirements, I also advised against some of them because they could result in performance issues.

In sum we have a railway with properties. This takes an old implementation a the data source, so there are some old design choices we have to keep in mind. At its core the railway has positions. These positions are just a single value ordered low to high. Much like a double linked list of nodes. At each position we have properties, like gradients, signals, stations, etc. Some of these properties are given as intervals, for example, the railway has a gradient N from A to B. And these components have to be iterated by position order due to system requirements. The positions are not all known at once, they are given by the railway's properties, i.e. we get positions from gradient's intervals, and might get a completely new position from a signal or a station. Given that they iterate by position, this pool must be always sorted. They also iterate by interval changes, i.e all gradient positions don't need to be iterated, however, a gradient should be able to be fetched directly from any position. This also means that, each time a new position is added to the registry, it should also be assigned a gradient or any other interval component for the entity that owns it, as long as the position belongs to an interval.

At the time they wanted an implementation that was easy to use and maintained these relationships automatically, and time was not on my side 😅

So, to fulfill all requirements I developed a registry wrapper that restricts the usage to our use-case. This wrapper provides access to a const registry, or a mutable instance through a lambda. This mutable instance provides access to reduced number of functions and does a few things automatically. An entity is created given a position value. The entity is created and a position is emplaced. A map is also kept in the registry's context to provide the inverse relation, since there is a reuse of old code sometimes it's necessary to know an entity given a position. This map also ensures that positions are unique. This component is also not meant to be managed by the user, so a user trait was added.

// traits are attached to components that the wrapper then identifies
static constexpr bool read_only = true;

For the interval implementation another user trait was added.

static constexpr bool range = true;

Intervals, also called ranges in the implementation, behave the following way:

template<typename T> struct range_of { entity from; };
template<typename T> struct range_head_of { entity to; };
struct T { ... }; // any T component

images containing part of the documentation explaining how this works

A range component is added the same way as any other component is assigned to an entity. However, the component is only added to the entity that corresponds to the beginning of the interval. Then a range can be added through a method provided by the mutable instance, where it receives the entities that correspond to the beginning and end.

When this mutable instance goes out of scope, a synchronization sequence is invoked to sort all pools by position. Each time a new position is added, all ranged components are checked and if there is an existing interval containing the position, a new range_of is assigned to that entity pointing to the the correct head.

Note: the range implementation was eased with boost icl intervals that are stored in registry's context

So, as I expected, we're facing performance issues due to the amount of sorting done. Who would've thought... but oh well, I won't rant here 😅

I'm now exploring way of getting the design done the correct way, but I'm having difficulties on how to structure these interval relationships to allow the same usage. This is probably due to my inexperience with more complex requirements using an ECS. How could I design this in a better way?

[iK4tsu]

The amount of sorting is definitely hurting. Removing all of the majority would be ideal. I looked at pointer stability at the time, and even did a POC with it. However, it didn't go through because there is a need to check for other components while iterating. The hierarchy solution by storing entities also did not go through because it did not use the for iteration that comes with views. I still think it's a valid solution.

template<typename T>
struct list
{
    entt::entity parent{entt::null};
    entt::entity next{entt::null};
    entt::entity prev{entt::null};

    // needed [maybe] to know the interval size
    std::size_t size{};
}

struct gradient {};
struct block {};



using type_t = relationship<gradient>; 
registry.view< type_t > gradients;

// iteration starts at the first position from low to high
// or at other defined by the current context
entt::entity pos = ...;

for (auto it = view.find(pos); it != view.end(); it = view.find(view[*it].next))
{
    gradient g = registry.get<gradient>(view[*it].parent);

     if (condition)
     {
         do the same for blocks starting at *it until the end of that interval
     }
}

In other words, loops get a bit more "manual" with these types of component's relationships. Given that, in our case, we noticed we would have more components that have relation like this than "single entity" ones, the final decision was against this.

TL:DR: I know about pointer stability, the POCs I presented at the time were rejected in favor of the current design. If there aren't any other options, even if less optimal, I may be able to persuade them now.

[skypjack]

Sorry for the late reply, I had a pretty busy week.

However, it didn't go through because there is a need to check for other components while iterating.

Not sure why it matters here. You can enable pointer stability for a single storage, then find other components using the entity currently in use.
Remember that there exists a to_entity utility that returns you the entity of a given component. Therefore, as you walk through your (pointer stable) hierarchy, you have all you need to get other components already. No need to make them stable nor anything else.

[iK4tsu]

Remember that there exists a to_entity utility that returns you the entity of a given component.

Ahhhh I didn't know about this one!

Is there a "nicer" to avoid duplicating component data? Let's take the gradient as an example. Multiple entities "share" the same value. The way I avoided component duplication was to add this data only to the entity where a change happens, what I called the head entity. Then, all others referenced that entity. Is this ok? Should it be stored elsewhere, like a resource?

template<typename Type>
struct relationship
{
    relationship<Type>* parent{nullptr};  // steps higher into the current interval
    relationship<Type>* child{nullptr};   // steps deeper into the current interval
    relationship<Type>* next{nullptr};    // jumps to the next interval
    relationship<Type>* prev{nullptr};    // jumps to the prev interval
    Type* instance{nullptr};              // points to the interval value
};

And type is stored elsewhere, or I basically mimic the old design but with pointer stability, where I divide this relationship in multiple components.

struct gradient
{
   double value;             // current interval value
   gradient* next{nullptr};  // jumps to the next interval
   gradient* prev{nullptr};  // jumps to the prev interval
};

template<typename Type>
struct interval
{
    Type* instance{nulloptr};         // points to the interval value
    interval<Type>* parent{nullptr};  // steps higher into the current interval
    interval<Type>* child{nullptr};   // steps deeper into the current interval
}

Probably there is no right answer for these things, since there are multiple ways to work with ECS 😅

[skypjack]

Yeah, it also depends on the access patterns. If the gradient is cold data (and probably also in other cases), a flyweight-like approach would work well too.
Access patterns (and ofc measurements) are what rule in these cases. There is no perfect solution or right way to arrange your data in general.

[iK4tsu]

Yeah, I figured, a bit of trial and error and see what helps the most.

After some other testing it seems that sorting all pools isn't what's affecting this the most. Sorting the main pool, the position component, and emplacing new positions is what affects is the most. All pools are sorted as positions so its actually relatively fast. Pools are sorted by value low to high. I used to use the insertion sort implementation, but found the radix sort more consistent and faster.

As said, components are registered in phases. Register all gradients, then all stations, etc. So, new positions might be inserted in each phase, and positions are always sorted.

a flyweight-like approach would work well too.

This similar to what I have with positions and the main component from ranges. Positions are unique, and also must serve as a bridge between the old and the new system. A simple map from position values to entities is enough. This map is also used to ensure 1 entity per position value.

Sadly to communicate between both systems to_entity cannot be used since the information is value based. Also there is a need to know values from "virtual" positions, a position hint. Order comes in handy here also because it is possible to find the closest value using binary search.

But I digress...

Each time a new position is added all ranged pools must be updated

range A --->  [ ] [ ] [ ] [ ]
   new position here ^

In other words if a there is a range [1 .. 10] for component A, and a new position with 8 is added, then this same entity must also own the range component of component A and point at its head entity with position 1

Since the implementation is generic there isn't a way to know which pools need this special handling. Hence the range traits. This is then done by also relying on the reflection system. Each range type has a method to add an entity to "its" range pool. All type infos of range components are iterated to resolve and invoke the reflected function for each new emplaced position. This is also quite heavy.

Btw, love the reflection implementation you have. Pretty easy to use after understanding how it works. Also, thank you for the advise so far :)