Are the benchmarks really unbiased?

[lennybakkalian]

When comparing prisma-server-node.ts and drizzle-server-node.ts, it seems that Drizzle is heavily preconfigured to perform better in benchmarks (e.g., connection pooling, logging disabled, prepared statements enabled).

I don't have deep knowledge of either framework, but this setup feels quite biased, as one ORM client is fine-tuned while the other appears to be used with its default configuration. Was there a genuine effort to ensure a fair comparison here?

[AndriiSherman]

• Prisma performs connection pooling automatically, and there is no way to configure it (https://www.prisma.io/docs/orm/prisma-client/setup-and-configuration/databases-connections/connection-pool#manual-database-connection-handling). In contrast, Drizzle allows you to specify connection pooling settings.

• The logger is disabled by default in both Prisma and Drizzle. However, including logger: false in the Drizzle configuration file is ambiguous and has no impact on benchmarks.

• Prisma uses prepared statements by default for all queries, which can lead to issues such as those documented here: Configure prisma not to use prepare statements prisma/prisma#5510 and here: Prisma Query Engine Creating large number of prepared statements. prisma/prisma#20913. I'm sure there are more such issues, if I will search more. Drizzle, on the other hand, does not use prepared statements by default, but you can enable them if needed.

Having APIs in Drizzle that are not available in Prisma does not mean we should avoid using them. The purpose of benchmarks is to test the fully optimized usage of both libraries and showcase how much more you can achieve with them. Failing to demonstrate this means we would only be testing a fraction of the available features and reducing the scope to raw SQL or database performance tests, where we should be evaluating the features and capabilities of ORMs themselves.