[Feature] Integration with DbGate and WhoDB applications

[jakoss]

Related to an existing integration?

No

Existing integration

No response

Overview

DbGate and WhoDB are small and fast web-based database management tools (think of PgAdmin, PgWeb, or Mongo Express). The advantage over mentioned applications is that both DbGate and WhoDB can support multiple databases.

My scenario

I'm currently working on a huge microservice system. We have some microservices based on Mongo, other on Postgres, but (since it's partly white-label tool) we want to support more SQL variants like SqlServer and Oracle soon. Also we use redis for caching and few message brokers. For all our local development needs we are using Aspire and it's being super helpful.

After we hit a 20 microservice mark we spot one common issue people had - computers started running out of memory. The amount of containers was getting too big, CPUs were getting swetty, annoying stuff. Aspire is great, so we could run some containers optionally limiting the issue.

But there is one issue that is still there and we did not had a great solution for that - until now. The problem - each database needs to have it's own management ui. Need to spin postgres? Add PgWeb to that (PgAdmin is so slow we gave up on it very fast). Want to add Mongo? Sure, add express. Redis? Redis insights come in. So we have this issue where number of containers is growing with no gain. Each of those tools have the same reason to exists - manage data.

Hence the idea to use DbGate or WhoDB. Effectively we are able to replace all those additional containers with a single one. We can manage the data in a single place. We are using it a single day so far and already love it.

Why both?

DbGate and WhoDB have the same purpose, but different set of features:

DbGate seems to be more mature and have a lot more database supported

WhoDB looks and feels much better, and have some interesting AI features (even supporting local LLMs)

Supporting both is not a big overhead, since the implementation will be very similar.

Moving forward I will describe API from a perspective of DbGate, but the WhoDB API will be pretty much identical to use.

Usage example

Nuget

Most of the implementation will live in the CommunityToolkit.Aspire.Hosting.DbGate.Core, but user won't be using that. Instead, there will be bunch more nugets for each database user wants to use, for example:

• CommunityToolkit.Aspire.Hosting.DbGate.Postgres
• CommunityToolkit.Aspire.Hosting.DbGate.MySql
• CommunityToolkit.Aspire.Hosting.DbGate.MongoDB
• CommunityToolkit.Aspire.Hosting.DbGate.Redis
  etc

Those packages are necessary, because if we just integrate with everything in a single nuget - user will bring whole bunch of dependencies with DbGate, which, we probably can all agree, is a bad thing :)

API

API will be very simple. Sample usage for 3 databases can look like this:

var postgresServer = builder.AddPostgres("postgres");
var database1 = postgresServer.AddDatabase("database1");
var mongoDbServer = builder.AddMongoDB("mongodb");
var database2 = mongoDbServer.AddDatabase("database2");
var redis = builder.AddRedis("redis");

builder.AddDbGate("dbgate")
    .WithDbReference(database1)
    .WaitFor(database1)
    .WithDbReference(redis)
    .WaitFor(redis)
    .WithDbReference(database2)
    .WaitFor(database2);

I think pretty much everything is self-explanatory besides the WithDbReference. Why not keep with WithReference?

Let's say user wants to add MySql to that. He adds server, then database and then adds WithReference(mysqlDatabase) to the AddDbDate. The compiler won't shout, but.. nothing will happen. Because user forgot to add the CommunityToolkit.Aspire.Hosting.DbGate.MySql NuGet. I think introducing new function will help avoiding mistakes like this.

Under the hood all references will be translated to bunch of environment variables that then will be used by DbGate to populate connections.

What do you all think?

Breaking change?

No

Alternatives

An alternative is to keep using dedicated database management tools for each database

Additional context

No response

Help us help you

Yes, I'd like to be assigned to work on this item

[aaronpowell]

Firstly, rather than a DbGate.Core package, it's probably better to follow the model that Aspire has for shared functionality, and that we've adopted for the upcoming Dapr Azure extensions, have them as files that live in the repo and are added as required (similar to how we use some of the utils classes too). This means you reduce the confusion of "do I need this package", and its less packages that we ship.

Regarding the API, given that the DbGate resource isn't something you would directly consume into other resources, why not flip it to be:

var postgresServer = builder.AddPostgres("postgres").WithDbGate();
var database1 = postgresServer.AddDatabase("database1").WithDbGate();
var mongoDbServer = builder.AddMongoDB("mongodb").WithDbGate();
var database2 = mongoDbServer.AddDatabase("database2").WithDbGate();
var redis = builder.AddRedis("redis").WithDbGate();

This would follow the same API that is in place for using other UI tools, like PgAdmin, PgWeb, PhpMyAdmin, etc. The method would work by looking at the resources that have been created and if it finds a DbGateResource, it'll to it, otherwise it'll create a new one.

The other advantage of this is that we can consider if it'd be right to create "extension" packages for these Aspire resources, like we have for Node.js and Python, in which we could roll in other future extensibility to them.

[jakoss]

Firstly, rather than a DbGate.Core package, it's probably better to follow the model that Aspire has for shared functionality, and that we've adopted for the upcoming Dapr Azure extensions, have them as files that live in the repo and are added as required (similar to how we use some of the utils classes too). This means you reduce the confusion of "do I need this package", and its less packages that we ship.

You mean this, right? I never used such thing, won't there be an assembly conflict when i add 2 nugets (like DbGate.Postgres + DbGate.Mongo)?

Regarding the API, given that the DbGate resource isn't something you would directly consume into other resources, why not flip it to be:

var postgresServer = builder.AddPostgres("postgres").WithDbGate();
var database1 = postgresServer.AddDatabase("database1").WithDbGate();
var mongoDbServer = builder.AddMongoDB("mongodb").WithDbGate();
var database2 = mongoDbServer.AddDatabase("database2").WithDbGate();
var redis = builder.AddRedis("redis").WithDbGate();
This would follow the same API that is in place for using other UI tools, like PgAdmin, PgWeb, PhpMyAdmin, etc. The method would work by looking at the resources that have been created and if it finds a DbGateResource, it'll to it, otherwise it'll create a new one.

Sure, i think i know how to do that, like nice too. But i think using that user cannot configure the DbGate resource (like changing port for example)?

The other advantage of this is that we can consider if it'd be right to create "extension" packages for these Aspire resources, like we have for Node.js and Python, in which we could roll in other future extensibility to them.

I'm not sure about that, DbGate is not really extension for the database itself, more like separate ui that is using the database

[Alirexaa]

I agree with @aaronpowell. We can extend each resource using the WithSomething extension method.

I reviewed the DbGate Docker image documentation, which clearly explains how to spin up a container and connect multiple databases to a single DbGate. We can provide database providers and connection strings through environment variables.

@aaronpowell I've already integrated RedisInsight and PgWeb into dotnet/aspire. If you and @jakoss are on board, I can work on this as well.

[aaronpowell]

You mean this, right? I never used such thing, won't there be an assembly conflict when i add 2 nugets (like DbGate.Postgres + DbGate.Mongo)?

Yep. We'd make the types internal so there wouldn't be a conflict since they wouldn't be exposed.

Sure, i think i know how to do that, like nice too. But i think using that user cannot configure the DbGate resource (like changing port for example)?

The common pattern is to provide a callback which you can configure the container resource. See https://github.com/dotnet/aspire/blob/main/src/Aspire.Hosting.PostgreSQL/PostgresBuilderExtensions.cs#L134 as an example.

I'm not sure about that, DbGate is not really extension for the database itself, more like separate ui that is using the database

It is and it isn't. I'm a little apprehensive about shipping around a dozen NuGet packages that are essentially a single method. While rolling them into an Extension package might still see the same problem initially, it lays the groundwork for future evolution.

[jakoss]

@aaronpowell I've already integrated RedisInsight and PgWeb into dotnet/aspire. If you and @jakoss are on board, I can work on this as well.

Fine by me, I'm not complaining about excess of free time :). I do have a spike ready (only for WhoDB, but might be helpful starting point) if you want it

Sure, i think i know how to do that, like nice too. But i think using that user cannot configure the DbGate resource (like changing port for example)?

The common pattern is to provide a callback which you can configure the container resource. See dotnet/aspire@main/src/Aspire.Hosting.PostgreSQL/PostgresBuilderExtensions.cs#L134 as an example.

I know about the pattern, but since we aim to have single instance of DbGate connected to multiple databases - which WithDbGate should i use to for example change the port of DbGate. What i change it in two places? This pattern is fine if there is a clear resource declaration. Here (if i understand correcty) we don't want to declare the resource directly, we only want to do call WithDbGate on multiple databases. That makes the DbGate resource itself somewhat implicit.