Archetypes

[genaray]

Soooo lets talk about archetypes.

If i am correct, archetypes offer a higher iteration speed. This is because entities with the same components are stored in one big array. Which means that the iteration over them can be loaded more efficient into the cpu. So iteration is a lot more cache friendly, which results in a performance boost.

So we should probably look into this. I assume that this wouldnt even require a major rework. Only the component storage would need some optimization.

I recently looked into those articles.
Theory behind archetypes
Unitys ECS
C# Archetype ECS

So a very quick and dirty first draft of a archetype based component storage would look like this...

    // Just stores an array of components, any smarter solution ? 
    public unsafe class ComponentPool {

        public void* components;  // We tightly pack all archetype components behind each other e.g. Player, Transform, Player, Transform
        public int length;

        // Unsafe operations to add, set or get certain components from the array
    }
    
    // An archetype, basically all entities with the same components. 
    public class Archetype {

        // An id which represents a bunch of types, somehow generated from a Type[] array basically
        public int id;
        public Type[] types;

        // The entities within that archetype
        public Entity[] entities;
        public ComponentPool components; 
        public int size;
    }

A few problems remain...

• Adding/Removing components is getting costly ( Can be solved by just disabling then )
• Iteration over a subset is a bit more difficult i guess ( Not sure if its slower or faster, but more complicated )

You know the source code of default.ECS way better than i do, would you mean that such an archetype based architecture would improve the speed and how hard would it be to change the underlaying component storage ?

[Doraku]

I assume that this wouldnt even require a major rework

It would actually be a MASSIVE rework :D So currently DefaultEcs relies on a sparse set implementation to store components, each type in their own array. To speed up entity enumeration they need to be cached based on the component composition you are querying (those are the entity containers set, map, ...) or else you would need to check every entities each time you want to enumerate a query. While this add some complexity it also allow to build more feature on top of it like component sharing (I know Unity allow that too but with my implementation it is completely free and require no extra code), predicate on component in the query, reactive container, ...
The problem with sparse set is that as you add/remove component on your entities, they will be completely out of order in the component arrays/entity containers (which you wouldn't have with archetype). To remedy this problem I introduced the World.Optimize method to sort the inner storages so that you always move forward in memory when iterating the entities/components.

Now from what I understand archetype can be done in two ways. The first one is Array of Structs like your draft { A, B, C}[]. The big problem with going this path in c# is that you will fight against the GC memory management. We don't know how many component types we will have on a single entity so we can't really create a finite number of Archetype<C1, C2, ...>. We may try to use unsafe pointer to handle memory ourselves but then we would need to tell the GC to leave our reference components alone even if there is no visible reference (because it is hidden in a IntPtr or something else), this is a big can of worms. Maybe it is possible to use a different strategy through reflection to create those archetypes at runtime but we may lose the compatibility with AoT only platforms (like iOS).

The other way is Struct of Arrays { A[], B[], C[] }. I believe this is what SimpleECS and Unity do. This should be simpler as the archetype would be composed of a pool for each component types (so kinda like what we currently have in DefaultEcs but with one more level of indirection Entity > World > Component Pool VS Entity > World > Archetype > Component Pool). Those pool would not need to be sparse set (Entity > Index > Component VS Entity Index > Component) so the number of indirection would stay the same but we would win a lot when enumerating and no more need for containers to cache query, the query would just need to cache the archetypes instead of the entities. But as said we would lose the predicate rules on components and you would need to check during enumeration on each entities instead but maybe it's not so bad and the speed up gained from enumerating components would end up positive.

The more I type about it the more I am dangerously close of thinking that it may be worth the trouble haha. Maybe I will try to do a POC on a branch and see how bad the api loss would be with this change and if it can be mitigated.

[genaray]

I assume that this wouldnt even require a major rework

It would actually be a MASSIVE rework :D So currently DefaultEcs relies on a sparse set implementation to store components, each type in their own array. To speed up entity enumeration they need to be cached based on the component composition you are querying (those are the entity containers set, map, ...) or else you would need to check every entities each time you want to enumerate a query. While this add some complexity it also allow to build more feature on top of it like component sharing (I know Unity allow that too but with my implementation it is completely free and require no extra code), predicate on component in the query, reactive container, ... The problem with sparse set is that as you add/remove component on your entities, they will be completely out of order in the component arrays/entity containers (which you wouldn't have with archetype). To remedy this problem I introduced the World.Optimize method to sort the inner storages so that you always move forward in memory when iterating the entities/components.

Now from what I understand archetype can be done in two ways. The first one is Array of Structs like your draft { A, B, C}[]. The big problem with going this path in c# is that you will fight against the GC memory management. We don't know how many component types we will have on a single entity so we can't really create a finite number of Archetype<C1, C2, ...>. We may try to use unsafe pointer to handle memory ourselves but then we would need to tell the GC to leave our reference components alone even if there is no visible reference (because it is hidden in a IntPtr or something else), this is a big can of worms. Maybe it is possible to use a different strategy through reflection to create those archetypes at runtime but we may lose the compatibility with AoT only platforms (like iOS).

The other way is Struct of Arrays { A[], B[], C[] }. I believe this is what SimpleECS and Unity do. This should be simpler as the archetype would be composed of a pool for each component types (so kinda like what we currently have in DefaultEcs but with one more level of indirection Entity > World > Component Pool VS Entity > World > Archetype > Component Pool). Those pool would not need to be sparse set (Entity > Index > Component VS Entity Index > Component) so the number of indirection would stay the same but we would win a lot when enumerating and no more need for containers to cache query, the query would just need to cache the archetypes instead of the entities. But as said we would lose the predicate rules on components and you would need to check during enumeration on each entities instead but maybe it's not so bad and the speed up gained from enumerating components would end up positive.

The more I type about it the more I am dangerously close of thinking that it may be worth the trouble haha. Maybe I will try to do a POC on a branch and see how bad the api loss would be with this change and if it can be mitigated.

Thanks for your fast reply and the insight ! :D

Looks like thats right, there two different ways as you mentioned. I researched a bit, using an AOS ( array of structs ) (Player, Transform, Rotation)[] should offer the best iteration performance because its the most cache friendly way out there. But well, as you said it requires some dark unsafe pointer magic.

I actually thought that using the SOA ( structs of arrays ) approach is a bit more difficult... atleast i dont know how we could realize this properly... generics cant be used here i guess. So doesnt this also involve pointer magic ? Or code generation atleast ^^ But the performance difference should not be that huge, because both are pretty cache efficient... so even if the array of structs approach should be faster, it should only be a bit faster ( Would actually require some benchmarking ).

About unity... no one really can tell what exactly they are using under the hood. I asked some "experts" in the different discords and even they arent sure about what unity exactly uses :D

I also think its worth the trouble ^^ Furthermore it would really speed up this ecs even further which is always a good thing. So thanks a lot for your time and effort !

-- Update

According to this Unity Talk here, unity uses structure of arrays ^^

I think the easiest way to represent SOA for the archetypes is using pointers. This might be unsafe, but its easier than code generation. After some brainstorming, it might could look like this ( just found out that theres a system.array type which could help us )

    // An archetype, basically all entities with the same components. 
    public unsafen class Archetype {

        // An id which represents a bunch of types, somehow generated from a Type[] array basically
        public int id;
        public Type[] types;

        // The entities within that archetype
        public Entity[] entities;
        public System.Array[] componentArrays; // A array which stores a bunch of different arrays e.g. ((Vector3[])componentArrays[0])[0]
        public int size;
    }

[Doraku]

I actually thought that using the SOA ( structs of arrays ) approach is a bit more difficult... atleast i dont know how we could realize this properly... generics cant be used here i guess. So doesnt this also involve pointer magic ? Or code generation atleast ^^ But the performance difference should not be that huge, because both are pretty cache efficient... so even if the array of structs approach should be faster, it should only be a bit faster ( Would actually require some benchmarking ).

Oh generic can be completely abused! That's actually what DefaultEcs is already doing to remove any cast/reflection when setting/getting components. It uses static generic type to hold worlds data, basically each world has its own internal id that is used to access a specific instance of component pool in a static generic array. This remove the need for casting and is much faster than a dictionary access that would be traditionally used, we use the type system as our dictionary. There may be some memory overhead but it is far outweighed by the benefits.

So if we give our archetype their own id, we can now give each of them their own component pool with the bonus that when we need to enumerate their entities, every component access will be direct

What we currently have

public ref T Get<T>() => ref ComponentManager<T>.Pools[WorldId].Get(EntityId);

What it could look like

public ref T Get<T>() => ref ComponentManager<T>.Pools[World.EntityInfos[EntityId].ArchetypeId].Get(EntityId);

An out of context Get<T> would be slower, but when querying we would get the internal pools directly and enumerate them in order, a direct index access on the array without any indirection.

Now all that is good by I just remembered the feature that would be hard to port with archetypes: enable/disable entities and component individually. It's probably possible to replicate by duplicating archetype with the same composition but pools that are not part of the queryable data. I definitely need to look into it.

[genaray]

I actually thought that using the SOA ( structs of arrays ) approach is a bit more difficult... atleast i dont know how we could realize this properly... generics cant be used here i guess. So doesnt this also involve pointer magic ? Or code generation atleast ^^ But the performance difference should not be that huge, because both are pretty cache efficient... so even if the array of structs approach should be faster, it should only be a bit faster ( Would actually require some benchmarking ).

Oh generic can be completely abused! That's actually what DefaultEcs is already doing to remove any cast/reflection when setting/getting components. It uses static generic type to hold worlds data, basically each world has its own internal id that is used to access a specific instance of component pool in a static generic array. This remove the need for casting and is much faster than a dictionary access that would be traditionally used, we use the type system as our dictionary. There may be some memory overhead but it is far outweighed by the benefits.

So if we give our archetype their own id, we can now give each of them their own component pool with the bonus that when we need to enumerate their entities, every component access will be direct

What we currently have

public ref T Get<T>() => ref ComponentManager<T>.Pools[WorldId].Get(EntityId);

What it could look like

public ref T Get<T>() => ref ComponentManager<T>.Pools[World.EntityInfos[EntityId].ArchetypeId].Get(EntityId);

An out of context Get<T> would be slower, but when querying we would get the internal pools directly and enumerate them in order, a direct index access on the array without any indirection.

Now all that is good by I just remembered the feature that would be hard to port with archetypes: enable/disable entities and component individually. It's probably possible to replicate by duplicating archetype with the same composition but pools that are not part of the queryable data. I definitely need to look into it.

Thats actually cool :D I didnt knew this.
Yeah the get operation will take a bit more time, but i actually think thats acceptable. Like 95% of the time we just iterate over entities. I only have 2-3 places in my code where i really make use of the get operation directly... and those are no hotspots ^^ probably the entity can store the archetype id in its struct. This would increase ram useage a bit, but it would require one less lookup, which makes it faster... not sure if its worth it.

If you want to disable a whole instance, you can swap it to the end of the active instances and decrement your active count, and then only iterate over the first activeCount entries of each array. That keeps the iteration dense. But requires some more effort to move it to the end.

For disabling/enabling components... flags are mostly used. Either directly inside the component ( Wrapper for example ) or by using another lookup.

Another option is to "lend" the entity to another archetype that has only the active components on it. The original archetype's copy stays around - swapped to the inactive pile as described above - so you don't lose the data stored on the inactive components. When the component is re-enabled, copy the entity's other data back from the archetype you lent it to, and re-activate it in this archetype. We probably mean the same here with those pools :D

If our solutions arent enough, i bet there still many others out there ^^

[Doraku]

probably the entity can store the archetype id in its struct

we can't do that because if the entity change of archetype and you stored it on your own, you won't get the updated archetype id because it is a struct :(

If you want to disable a whole instance, you can swap it to the end of the active instances and decrement your active count, and then only iterate over the first activeCount entries of each array. That keeps the iteration dense. But requires some more effort to move it to the end.

If all the entity data need to be copied anyway, maybe it is still simpler to think about it as a new archetype. The only big optimization is when by chance you disable the last entity of the archetype: nothing to copy then but it probably won't happen a lot.

[genaray]

Well thats correct :/ i havent thought about that. So we need a lookup at that palce... but i guess its actually just an array operation, right ? If thats the case it would still be very fast.

And yes thats true, the copy operation ( and algo, because you need additional work to swap it ) are probably to much and there cleaner ways of doing that :D Im excited how your test turns out and how it will look like.

[nrader95]

A few problems remain...
Adding/Removing components is getting costly ( Can be solved by just disabling then )

Although i like the idea of raw increase in speed on iteration through entities when you only getting components, please keep in mind that is not always possible or convenient. Things like Set/Remove or Enable/Disable is already have their share of limitations (threading, for example) and it would be nice to not increase that list.

[genaray]

A few problems remain...
Adding/Removing components is getting costly ( Can be solved by just disabling then )

Although i like the idea of raw increase in speed on iteration through entities when you only getting components, please keep in mind that is not always possible or convenient. Things like Set/Remove or Enable/Disable is already have their share of limitations (threading, for example) and it would be nice to not increase that list.

There should be no further limitations, add/set operations should behave pretty normal and shouldnt decrease in performance that much ( It needs to copy the entity over into a new archetype but thats all, everything has pros and cons... and disabling/enabling will probably being used more often ). The only part ( right now ) which needs some rework is the disabling/enabling of components, which we will find a good solution for.

The new architecture also should have no constraints in terms of threading. Atleast i cant imagine one. I assume the entity recorder will also work fine ( just needs to change some code under the hood ).

[Doraku]

Things like Set/Remove or Enable/Disable is already have their share of limitations (threading, for example) and it would be nice to not increase that list

Those limitations will stay the same but the new access will also comes with its own caveats. When accessing components you will probably get something like a Span<T> and to access the entity component you would need to use the entity "index" in its own ReadOnlySpan<Entity>. In memory those would point to the archetype storage so if you do an operation that changes the entity composition (new set, remove, enable, disable) the actual data of the entity would move to a new archetype and the Span<T> you previously had would be completely out of sync with the reality. I will see if I can put some safeguard to throw explicit exception when things go wrong so it doesn't fail silently with users having no idea of the problems to come.
For those scenario the EntityCommandRecorder would still need to be used as before.

Anyway I started experimenting a little. For now I should be able to keep the existing entity api intact (except for SetSameAs but it will probably come back later), my main issue is creating the archetype transitions at runtime (what is the new archetype when you add/remove a specific component starting from the current archetype) and making it fast! Once this is put in place, I should be able to create a simple benchmark to compare the gains before going further.

[genaray]

Things like Set/Remove or Enable/Disable is already have their share of limitations (threading, for example) and it would be nice to not increase that list

Those limitations will stay the same but the new access will also comes with its own caveats. When accessing components you will probably get something like a Span<T> and to access the entity component you would need to use the entity "index" in its own ReadOnlySpan<Entity>. In memory those would point to the archetype storage so if you do an operation that changes the entity composition (new set, remove, enable, disable) the actual data of the entity would move to a new archetype and the Span<T> you previously had would be completely out of sync with the reality. I will see if I can put some safeguard to throw explicit exception when things go wrong so it doesn't fail silently with users having no idea of the problems to come. For those scenario the EntityCommandRecorder would still need to be used as before.

Anyway I started experimenting a little. For now I should be able to keep the existing entity api intact (except for SetSameAs but it will probably come back later), my main issue is creating the archetype transitions at runtime (what is the new archetype when you add/remove a specific component starting from the current archetype) and making it fast! Once this is put in place, I should be able to create a simple benchmark to compare the gains before going further.

That actually sounds great ! :D I bet you will find a good solution for making those runtime transistions fast. Thanks a lot for all your effort and time ! ^^ Excited to see the outcome.

[Doraku]

Ok so I have something working. There is still a lot of work to do but here are some rough numbers:

SingleComponentEntityEnumeration
before: 189.6 μs
after: 87.00 μs

DoubleComponentEntityEnumeration
before: 415.9 μs
after: 244.4 μs

TripleComponentEntityEnumeration
before: 391.9 μs
after: 273.5 μs

things to note:

• those benchmarks are terrible, like the DoubleComponent being slower than TripleComponent >_>?
• all of those results are in mono thread (still need some work for multithreading with archetypes)
• I didn't reboot my computer and had a lot of stuff opened (did I say those bencharks were terrible?)
• there will probably be a small overhead for archetypes when you have multiple ones corresponding to your query
• the performance of the old way (entity.Get<T>()) obviously suffered a lot, the api is still here for convenience but it is ten times slower than the archetypes access (maybe there is some stuff that can be done to make it not so bad)
• more than 20% broken tests :p

api that may be lost:

• get/set component maximum capacity (not sure if it is worth it, keeping count across all archetypes would be a pain)
• the query with predicate (or at least they won't be taken into account for the archetypes, maybe for the entity containers?)
• not lost but reactive entity containers will take a hit because they probably won't be usable through archetypes and rely on the old access (entity.Get<T>())

Obviously this is just the beginning and for now I still consider it a POC, I need to work on the query for archetypes and see where it goes.

[genaray]

Ok so I have something working. There is still a lot of work to do but here are some rough numbers:

SingleComponentEntityEnumeration
before: 189.6 μs
after: 87.00 μs

DoubleComponentEntityEnumeration
before: 415.9 μs
after: 244.4 μs

TripleComponentEntityEnumeration
before: 391.9 μs
after: 273.5 μs

things to note:

• those benchmarks are terrible, like the DoubleComponent being slower than TripleComponent >_>?
• all of those results are in mono thread (still need some work for multithreading with archetypes)
• I didn't reboot my computer and had a lot of stuff opened (did I say those bencharks were terrible?)
• there will probably be a small overhead for archetypes when you have multiple ones corresponding to your query
• the performance of the old way (entity.Get<T>()) obviously suffered a lot, the api is still here for convenience but it is ten times slower than the archetypes access (maybe there is some stuff that can be done to make it not so bad)
• more than 20% broken tests :p

api that may be lost:

• get/set component maximum capacity (not sure if it is worth it, keeping count across all archetypes would be a pain)
• the query with predicate (or at least they won't be taken into account for the archetypes, maybe for the entity containers?)
• not lost but reactive entity containers will take a hit because they probably won't be usable through archetypes and rely on the old access (entity.Get<T>())

Obviously this is just the beginning and for now I still consider it a POC, I need to work on the query for archetypes and see where it goes.

Those results are actually looking pretty good :D How many entities were processed ?
But yeah, i think the benchmark probably needs another run, without any other stuff going on in the background ^^

I actually spend some time debugging the unity ecs source code... Their entityManager.GetComponentData<T>(in entity) is quite similar to our approach entity.Get<T>().... Its actually even more complicated.

Their archetype is split into chunks and each Component-Type has its own id. At some point they actually use an array to map the entity.Index to its chunk it lays in... the complete flow looks like this.

var componentTypeId = Type.GetType<T>();
var chunk = world.archetype[entity.Index].chunk[entity.Index];
// Loop over ALL components inside the chunk till we found a component with the same type id as above

(Yeah... they really loop over all components inside the chunk)

public ref T Get<T>() => ref ComponentManager<T>.Pools[World.EntityInfos[EntityId].ArchetypeId].Get(EntityId);
So i actually think the way you mentioned above is quite fine ^^ normally such operations shouldnt be used that often at all. But i bet theres always room for improvement.

Again, thanks for your time & effort ! :D

[nrader95]

the performance of the old way (entity.Get()) obviously suffered a lot

Will new way of access components be as convenient as Get, then? For example, i often request components from other entities, or request components that are not part of system entity filter(after Has call, ofc).

[Doraku]

How many entities were processed ?

100000, benchmarks adapted from here

At some point they actually use an array to map the entity.Index to its chunk it lays in...

I didn't split it into chunk because one array per component per archetype is already complicated enough but I also have a entity id to archetype index mapping to access an entity data randomly through the Get/Set. The big performance loss of those method is because I handle enable/disable state for each component type :/ this cause an indirection on the actual component pool I need to access (is it the archetype pool with the mapping of the archetype or the shared pool for disabled component with its own mapping). The goal of this feature was to speed up composition change because no data needed to be moved but because of archetype this is no longer true so it doesn't really give any speed up compared to just removing the component from the entity and adding it back later. The only benefit is keeping the value of the component for you to later add it back :/ I am not sure if I just drop it (but keep the enable/disable at the entity level) or find an other way to do it.
On the plus side #69 is in at least!

Will new way of access components be as convenient as Get, then?

The goal is to try to make as little as possible breaking changes, old codes should works as before with maybe a slight performance loss until you switch to the new stuff with all the benefit. Futur will tell if this holds true ^^"
I am not sure what the final new api will be but probably something like this (pseudo code):

foreach (Archetype archetype in Query)
{
    ReadOnlySpan<Entity> entities = archetype.GetEntities();
    Span<Component1> c1s = archetype.Get<Component1>();
    Span<Component2> c2s = archetype.Get<Component2>();

    for (int i = 0; i < archetype.Count; ++i)
    {
        Update(entities[i], ref c1s[i], ref c2s[i]);
    }
}

Will there be extension methods to wrap all this stuff for you or exclusively code generation through the DefaultEcs.Analyzer package I still don't know, I really don't want to maintain stuff like this or this x)

[genaray]

he big performance loss of those method is because I handle enable/disable state for each component type :/ this cause an indirection on the actual component pool I need to access (is it the archetype pool with the mapping of the archetype or the shared pool for disabled component with its own mapping). The goal of this feature was to speed up composition change because no data needed to be moved but because of archetype this is no longer true so it doesn't really give any speed up compared to just removing the component from the entity and adding it back later.

Wouldnt that disable/enable mechanic still be way faster than triggering an remove/add operation which would copy the entity into a different archetype ? ^^ Atleast thats what i always hear... furthermore the state gets lost during that operation.

My 2 cents... I could imagine four different ways to implement disabling/enabling in an archetype way.

• Archetype{ EntityInfo[] info } which stores a Bitvector to tell our archetype which components are enabled/disabled
• Interface IToggleable which can be implemented by components and queries check that one during iteration
• Wrapping components to add disabled/enabled state : WrappedComponent<T>{ T component, bool enabled} and the archetype stores all components as those Archetype{ WrappedComponent<Transform>[], WrappedComponent<Rotation>[]... }
• Optional component wrapping, on top of the default archetype... Toggleable<T>{ T component, bool enabled } and the archetype uses a mixed approach new Archetype(typeof(Transform), typeof(Rotation), typeof(Toggleable<WasHit>)) so the archetype looks like this Archetype{ Tranform[], Rotation[], Toggleable<WasHit>[]}. This way we only pay the price when we really want it. This would require us to check disabled/enabled components by ourselfs when we iterate over queries but it would still be blazing fast.

The last way was often recommended by people all over the different discords im in ^^
I actually think code generation is the way to go... those wrapper stuff looks... annoying.

[Doraku]

The way I currently handle it is like this:
An entity with components A, B ,C will go in archetype ABC, each component being stored in the archetype respective component pool, ArchetypeABC<A> ArchetypeABC<B> ArchetypeABC<C> all sharing the same maping EntityId > index.
An entity with components A, B, disabled C will go in archetype AB, its components will be stored in ArchetypeAB<A> ArchetypeAB<B> Generic<C>, the generic pool being a shared pool across all archetypes to store disabled components, with its own mapping EntityId > index (this is basically the current storage that DefaultEcs use for all component).
So disabling/enabling a component will actually move the entity data between archetypes storage. It cost more but then when iterating you have a single path.

• Archetype{ EntityInfo[] info } which stores a Bitvector to tell our archetype which components are enabled/disabled

when enumerating you would have to check each entity individually for what is activated or not, you would lose the benefit of no branch path when iterating on a archetype entities. The Without queries would also become a lot more complicated because even if you ask Without<C>() you would have to include archetype ABC to check if C is disabled for a particular entity.
A naive solution could be to sort entities in the archetype by combination of component enable/disable (so ABC first, ABCdisabled then, ABdisabledC, disabledABC, AdisabledBdisabledC, ...) but we can imagine how quickly problematic it would become, sometimes simple is the best.

• Interface IToggleable which can be implemented by components and queries check that one during iteration

not a fan of this as you can't enable/disable any component type that could come from an external library, you would have to create a wrapper (your last point) for them, and the previous problem still remains :/

Some frameworks like bevy allow you to chose between table storage (archetype) or sparse set for each component, maybe this is something that could be used to reallow shared component with minimal effort.

[nrader95]

Interface IToggleable which can be implemented by components and queries check that one during iteration

This would be causing boxing if your component is a struct and you work with it after casting your type to interface.

[genaray]

An entity with components A, B ,C will go in archetype ABC, each component being stored in the archetype respective component pool, ArchetypeABC<A> ArchetypeABC<B> ArchetypeABC<C> all sharing the same maping EntityId > index.
An entity with components A, B, disabled C will go in archetype AB, its components will be stored in ArchetypeAB<A> ArchetypeAB<B> Generic<C>, the generic pool being a shared pool across all archetypes to store disabled components, with its own mapping EntityId > index (this is basically the current storage that DefaultEcs use for all component).
So disabling/enabling a component will actually move the entity data between archetypes storage. It cost more but then when iterating you have a single path.

Oh guess i forget a way ^^ This also looks fine.
Will you stick with that way ? If not i would recommend not implementing enabling/disabling inside the world/archetype. The wrapper approach Toggleable<T>{ T cmp, bool enabled }/query.With<Transform>.With<Toggleable<Hit>.. is probably the most performant way. So the user only pays the price if he really needs enabled/disabled components. That would actually speed up get/set, iteration and prevents entities from copy over to another archetype.

[Doraku]

If you do not use the built-in enable/disable feature and do your own logic for it through Toggleable<T> you can do it :)
I am also experimenting with allowing different pool implementation for component storage.

• Archetype would be the default, component being stored separately for each archetype and need to be copied when it changes
• Single, component are store in a basic sparse set and shared for all archetypes, no need to be copied when the archetype changes
• Shared, the current DefaultEcs storage, a basic sparse set that allow entities to share the same component value (SetSameAs<T>), shared for all archetypes
• Multi, I won't do it now but it would allow you to add multiple components to the same entity, again shared for all archetypes

So in theory if you would set all your component types to the Shared mode (maybe a parameter of the World constructor?), you would end up with the current DefaultEcs storage (minus a small performance hit because of an interface call) and there would be no copy when enabling/disabling components. You would get the best performance if all the components of your query are in Archetype mode.

Maybe later I will try to make the storage mode change as needed so for example by default you have Archetype but the first time you do a SetSameAs<T> it will create the new pool and copy over all the data from all archetypes for you so you don't have to think about setting the correct mode in the first place. Not all transitions would be possible obviously

• Archetype > Single
• Archetype > Shared
• Archetype > Multi
• Single > Shared
• Single > Multi

Anything else would throw.

[Doraku]

It is starting to look great, when using Single or Shared mode for component, the overhead is ~x2 compared to current implementation, all api should stay intact. In archetype mode the overhead is ~x4 when using the old Get<T> api for convenience but the new api gives x3 more performance compared to current implementation!

A lot of work still remains, the next big question is what to do with MultiMap systems, is it possible to do something so they can benefit from archetypes? A new api should be also created for entities to batch modifications together so the archetype is only changed once even if we do multiple Set/Remove to limit copy.

[genaray]

It is starting to look great, when using Single or Shared mode for component, the overhead is ~x2 compared to current implementation, all api should stay intact. In archetype mode the overhead is ~x4 when using the old Get<T> api for convenience but the new api gives x3 more performance compared to current implementation!

A lot of work still remains, the next big question is what to do with MultiMap systems, is it possible to do something so they can benefit from archetypes? A new api should be also created for entities to batch modifications together so the archetype is only changed once even if we do multiple Set/Remove to limit copy.

That actually sounds insane ! ^^ Keep up the great work and its great to hear such good news.

[crener]

I hate to jump into this discussion especially since I've just skimmed what has been covered already.

Unity have a patent about architypes and their implementation in a single array, other people with ECS implementations have been discussing this and how it affects them. Bevy for example have a sparse array implementation that wouldn't be affected (as it doesn't use archetypes)...

public void* components; from the very first post make it a little close but I guess the patent determines that the memory layout has to be entity ID first then the component data. I don't know if the implementation would be in violation of this (as I don't know where to find it) and would possibly exclude DefaultEcs from commercial projects sold/used in the US (maybe even within Unity? I'm not a lawyer).

I just wanted to make sure people are aware before someone gets stung really badly by this even though I would love to see speed improvements.

[Doraku]

Thanks for the links, I have already seen it around the same time as the reddit thread started. Keep in mind that the change to Archetypes for DefaultEcs is still far away but the implementation I went with is probably closer to Bevy one: components are stored in their own sparse array but each "archetype" has its own sets of component so an entity components all share the same index for access, probably not as fast as Unity solution but from my first benchmark it still gives a noticeable boost! Archetype or not at least it should be safe from this horrible patent...
If you are curious it is available on the archetype branch, anyway thanks for your concern :)