FR: Allow file based communication (sendmail-like/unix socket; no networking/ports)

[RafaelKr]

Use case summary

• Starting up many local projects in parallel
• Hosting many projects on one server

Use case 1: Starting up many local projects in parallel

I'm working at an agency and sometimes we need to spin up multiple projects (and mailpit instances) in parallel.

We're using https://devenv.sh/ to configure our projects, so we just have to set services.mailpit.enable = true in our devenv.nix file and have a running mailpit instance. Data is stored in ./.devenv/state/mailpit/db.sqlite, the path is relative to the project. Also see https://github.com/cachix/devenv/blob/d612b77ff73912cd82e58256ab5e84d5904abef7/src/modules/services/mailpit.nix#L47 and https://devenv.sh/reference/options/#servicesmailpitenable

The mailpit instance listens on 127.0.0.1:1025 and 127.0.0.1:8025 by default, so if we want to spin up more than one project in parallel we always need to temporarily change the ports of at least one project and then change them back to defaults later.
I also thought of reserving 2 free ports per project, but I probably worked on 100+ projects by now. Always reserving new ports per project doesn't seem feasible, also sharing the project configuration with colleagues isn't easy then.

Use case 2: Hosting many projects on one server

We're hosting many NixOS servers, they're configured declaratively. Those servers host multiple projects as virtual hosts. Most projects have a dev and staging environment where we're using a mailpit instance per virtual host.

Currently we spin up multiple mailpit instances and always need to search for free ports for the mailpit UI and the SMTP services. Then we need to define the chosen ports in the configuration. The UI is configured in the reverse proxy to be reachable via https://www.example.com/mailpit/ (protected via BasicAuth).

Other Service Examples

For services like the Database or redis I'm able to configure them to listen via unix sockets, so we can specify them as

# one global socket, authentication by system user
DB_SOCKET=/run/mysqld/mysqld.sock

REDIS_PATH=/run/redis-project-1-dev/redis.sock
REDIS_PATH=/run/redis-project-1-staging/redis.sock
REDIS_PATH=/run/redis-project-1-prod/redis.sock

REDIS_PATH=/run/redis-project-2-dev/redis.sock
REDIS_PATH=/run/redis-project-2-staging/redis.sock
REDIS_PATH=/run/redis-project-2-prod/redis.sock

As you can see we can just use a generic project name (project-1, project-2, ...) to automatically reserve sockets. Or in the case of the database, where only one instance is running, the context (accessible databases, database permissions, ...) depends on the system user under which the application is running.

So unix sockets have another advantage from a security perspective: We can set unix permissions on them to control which user has access to them. In our server environments every vhost has an own user. The application is running as that user. And the redis instances (started via systemd) are running as the same user and a socket permission is set to 600, so only the vhost user can read and write to their own socket.

Mailpit Feature Request

It would be nice to be able to configure mailpit in a similar way.

SMTP

For the SMTP configuration it could be a sendmail-compatible wrapper file.

# Wrapper binary (not sure if we could use a unix socket here?)
MAILER_DSN=sendmail://default?command=/run/mailpit/project-1-dev/mailpit-sendmail
MAILER_DSN=sendmail://default?command=/run/mailpit/project-1-staging/mailpit-sendmail

MAILER_DSN=sendmail://default?command=/run/mailpit/project-2-dev/mailpit-sendmail
MAILER_DSN=sendmail://default?command=/run/mailpit/project-2-staging/mailpit-sendmail

# currently
MAILER_DSN=smtp://localhost:1031 # project-1-dev
MAILER_DSN=smtp://localhost:1030 # project-1-staging
MAILER_DSN=smtp://localhost:1041 # project-2-dev
MAILER_DSN=smtp://localhost:1040 # project-2-staging

Configuration examples are for Symfony:

• MAILER_DSN transport syntax: https://symfony.com/doc/current/mailer.html#transport-setup
• sendmail command syntax: https://symfony.com/doc/current/mailer.html#other-options

Mailpit UI

See: https://mailpit.axllent.org/docs/configuration/proxy/
For the reverse proxy (nginx in our case) something like:

server {
    server_name dev.project-1.com;
    auth_basic secured;
    auth_basic_user_file /path/to/nginx-auth/project-1-dev.htpasswd;

    # ...

    location ^~ /mailpit/ {
        # currently: proxy_pass http://127.0.0.1:8031;
        proxy_pass http://unix:/run/mailpit/project-1-dev/mailpit-ui.sock;
		# configure the websocket
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "Upgrade";
	}
}

server {
    server_name staging.project-1.com;
    auth_basic secured;
    auth_basic_user_file /path/to/nginx-auth/project-1-staging.htpasswd;

    # ...

    location ^~ /mailpit/ {
        # currently: proxy_pass http://127.0.0.1:8030;
        proxy_pass http://unix:/run/mailpit/project-1-staging/mailpit-ui.sock;
		# configure the websocket
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "Upgrade";
	}
}

server {
    server_name dev.project-2.com;
    auth_basic secured;
    auth_basic_user_file /path/to/nginx-auth/project-2-staging.htpasswd;

    # ...

    location ^~ /mailpit/ {
        # currently: proxy_pass http://127.0.0.1:8041;
        proxy_pass http://unix:/run/mailpit/project-1-dev/mailpit-ui.sock;
		# configure the websocket
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "Upgrade";
	}
}

server {
    server_name staging.project-2.com;
    auth_basic secured;
    auth_basic_user_file /path/to/nginx-auth/project-2-staging.htpasswd;

    # ...

    location ^~ /mailpit/ {
        # currently: proxy_pass http://127.0.0.1:8040;
        proxy_pass http://unix:/run/mailpit/project-2-staging/mailpit-ui.sock;
		# configure the websocket
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "Upgrade";
	}
}

# ...

Proposed mailpit configuration options

We need to be able to specify a socket path and also need to be able to specify the socket/mailpit-sendmail permissions. My idea would be to have the following configuration options:

• MP_UI_BIND_ADDR: allow configuration of sockets or create an own configuration option (important thing is to be able to disable networking so there will be no EADDRINUSE errors due to port collissions)

• MP_UI_SOCKET_PERM: octal permissions, see redis unixsocketperm (references below), default to 600

• MP_SMTP_BIND_ADDR: see MP_UI_BIND_ADDR

• MP_SMTP_SOCKET_PERM: see MP_UI_SOCKET_PERM, default to 700 if it is a wrapper binary or 600 if it's somehow possible to use a socket

Socket owner (user) and group should be determined from the mailpit instance user and group.

References:

• redis unixsocket and unixsocketperm, also needs port 0 to disable networking: https://redis.io/docs/latest/operate/oss_and_stack/management/config-file/
• MariaDB Remote Client Access: https://mariadb.com/kb/en/configuring-mariadb-for-remote-client-access/
• MariaDB Unix Socket Authentication: https://mariadb.com/kb/en/authentication-plugin-unix-socket/
• Caddy Webserver bind syntax for sockets: https://caddyserver.com/docs/caddyfile/directives/bind#examples
• Dovecot/Postfix LMTP (Local Mail Transfer Protocol): https://doc.dovecot.org/2.3/configuration_manual/howto/postfix_dovecot_lmtp/
• gitea Issue Allow SMTP over unix socket: Allow SMTP over unix socket (& rework mailer settings) go-gitea/gitea#18901

Advanded: Abstract Namespace Sockets

Maybe interesting for an implementation of the mailpit-sendmail wrapper if it needs to be a binary instead of a socket. The to communicate with the mailpit instance process could be implemented with that.

• Linux's abstract namespace for Unix domain sockets: https://utcc.utoronto.ca/~cks/space/blog/linux/SocketAbstractNamespace
• The net package of Go already supports Abstract Namespace Sockets out of the box: https://golang-examples.tumblr.com/post/92025745979/pitfall-of-abstract-unix-domain-socket-address-in
• nginx Unit also supports reverse proxying to abstract namespace sockets, so maybe it would also be interesting to support them for the mailpit UI socket: https://unit.nginx.org/news/2022/unit-1.28.0-released/#abstract-unix-domain-sockets
• They even have the best performance: https://github.com/mschwager/socket-speed

[axllent]

Thanks for your detailed feature request @RafaelKr ~ very useful.

This is a very interesting idea as it does provide an alternate approach to the whole "port issue" ~ which has been raised a couple of times before by others implementing Mailpit in large multi-user/customer environments with shared hosting.

Forgetting briefly that Mailpit is not limited to just *nix platforms (ie: I do not believe unix sockets are supported on Windows)..... theoretically this idea could work. I just did a (very crude) test running the Mailpit HTTP server via a unix socket which Nginx connected directly to (as per your example) and it worked great (including the websocket).

But ... I'm not sure about the SMTP server because the mhale/smtpd module (that Mailpit uses) does not currently support sockets (it's hardcoded to use tcp). I would need to fork and test to see if that would even work with a unix socket.

Assuming it all worked, to implement this properly will be quite a lot of work as there are several moving parts, including of course the SMTP server & SMTP client, plus testing & of course documentation. I'm not sure when I can make the time available as I'm scheduled to have shoulder surgery in about 6 weeks time which will put me "out of action" for a couple more months.

To give me some idea of your need for this feature, could you please tell me what kind of scale are you wanting to use this for (ie: how many projects would be using this if it was implemented), and how & if you (or your company) could potentially contribute? Thanks.

[axllent]

@RafaelKr I have started work on this feature and it seems possible using HTTP & SMTPD sockets instead of TCP, as well as the Mailpit "sendmail" sending directly via the SMTPD socket- so it will be possible to run multiple instances of Mailpit without using any TCP ports.

It does require quite a number of internal Mailpit changes, as well as some heavy modifications to the SMTPD library (meaning I will need to bundle the modified library into Mailpit's source code), and I'll still need to do a lot of testing - so there's a lot more work for me to do.

It would still be great if you could get back to me on my previous questions, thank you.

[RafaelKr]

@axllent Wow, didn't expect it to go that fast after your "I'm not sure when I can make the time available" response, but I'm positively surprised! First of all, I wish you all the best for the surgery, I hope everything goes well.

So to your questions:

• Our scale isn't that large, it's much more a convenience feature which I thought about while being in the works of improving our internal processes with more automations. The more is configured automatically (and reliably of course!) the less can fail. A rough scale are 15 different Servers with 3-6 environments (2-4 using mailpit each). And the local development environments for each project where it's also sometimes convenient to just spin them up in parallel without adjusting configuration first.
• My schedule is a bit tight right now, it starts getting better in ~2,5 weeks. Then I could maybe contribute with testing and documentation. I don't feel my experience with Go is enough that I could do actual coding. Besides from that I started sponsoring you with a small amount but hopefully better than nothing.

But I just came up with another idea that wouldn't require modifications to the smtpd library. When we request address "localhost:0" the OS automatically assigns a free port, so we also don't have collisions. I will try to come up with a quick PoC if we can get the port from smtpd then.

[RafaelKr]

I got a working PoC. Here's the code: https://gist.github.com/RafaelKr/ff101e0b505caf4150146a34e569e1a4

Usage:

mkdir mailpit-smtpd-port-poc
cd mailpit-smtpd-port-poc

go mod init smtpd-port-poc
# Download GitHub gist PoC snippet
curl -O https://gist.githubusercontent.com/RafaelKr/ff101e0b505caf4150146a34e569e1a4/raw/814f60c56e4510d78a5e2b3a4407317773930f10/mailpit-smtpd-port-poc.go
go get github.com/mhale/smtpd

# Start Server
go run mailpit-smtpd-port-poc.go

Now we got a server running on a free port chosen by the system. Also a new shell wrapper script mailpit-sendmail.sh was created in the project directory.

In a new shell (running in the project dir) we can now create a mail.txt file:

printf 'Subject: Terminal Email Send\n\nEmail Content' > mail.txt

And send mails via

./mailpit-sendmail.sh [email protected] < mail.txt

Now we should see a log line in the shell where the server is running.

[RafaelKr]

I also just checked if that works inside a Symfony project. It does!

MAILER_DSN="sendmail://default?command=/path/to/mailpit-smtpd-port-poc/mailpit-sendmail.sh%20-bs" bin/console mailer:test [email protected]

And I got a log!

2024/10/19 16:47:48 Received mail from [email protected] for [email protected] with subject Testing transport

Unfortunately I can only test on Linux, I'm pretty sure it will also work on macOS - but I don't know if or how it will work on Windows.

Edit: I used the -t argument first, which also worked. But I updated the command to use -bs instead which is the Symfony default: https://github.com/symfony/mailer/blob/69c9948451fb3a6a4d47dc8261d1794734e76cdd/Transport/SendmailTransport.php#L36

-bs: Use the SMTP protocol as described in RFC 821 on standard input and output. This flag implies all the operations of the -ba flag that are compatible with SMTP.

[axllent]

Hey @RafaelKr - thanks for both the sponsorship and the thought you have put into this! It's really useful to have two brains thinking of a solution instead of one 😄

Whilst your approach and PoC for a dynamic port plus a "sendmail wrapper" is an interesting one (and which works), I personally do not think it's the right solution as it still requires TCP which is not always ideal - I'll explain my thinking:

• If we're supporting sockets for HTTP, then we may as well provide the same approach for SMTP. One goal I have Mailpit is to try my best to keep the startup flags / env variables as constant as possible, and only introduce new ones when absolutely necessary. So far (in my modified branch) I am able to re-use the both the --listen and --smtp for the unix sockets (eg --listen unix:/var/run/mailpit/http.sock:0666 --smtp unix:/var/run/mailpit/smtp.sock:0666) which provides the path & permissions without introducing any new flags.
• Unix sockets can easily be mounted into all containers (assuming the user is using containers). One issue I have had in the past (with a completely different project) was the need for two separate Docker containers to communicate without knowing the docker IP of "the server" -Unix sockets solved that issue perfectly.
• The SMTPD library is really only one file, so maintaining Mailpit's own customised version within the project isn't an issue because that library is not updated often at all. The SMTP protocol won't change any time soon, and updates to the upstream library can always be manually ported across if need be.

So the goal I have here is to be able to:

1. Connect to the HTTP socket like your example earlier - tested & works.
2. Allow mailpit sendmail to send via the socket - tested and works. In your example this would mean we'd just configure the Symfony mailer with something like "sendmail://default?command=/path/to/mailpit%20sendmail%20-S%20/path/to/socket%20-bs" rather than a wrapper (alternatively if the MP_SENDMAIL_SMTP_ADDR environment is set to the socket location then it wouldn't need to be manually set, but this depends on the infrastructure of course) . Whilst I haven't tested this in Symfony yet, I'm sure this will work just fine.

I'll keep working on this (I'm trying to get it done as soon as possible because of the planned surgery), and I'm fairly sure I can get something out within the next week or two that will allow you to start testing the new experimental feature 👍

[Corvan]

Just to chime in here with a very little detail: postfix uses LMTP delivering mail to local mailboxes, AFAIK LMTP is simply an smtp implementation over local sockets, maybe something like this also exists in Go. If this comment is not fitting or does not help, in this situation, please just ignore ;-)

[axllent]

Thank you for the "chime in" @Corvan - I'm always open to ideas ;-) Until now I hadn't actually even heard of LMTP.

I've done some reading but cannot see any advantages (for what I'm trying to achieve here) over SMTP. LMTP (which uses a slightly different "language" compared to SMTP) apparently works by default over TCP on port 24 and is designed for local mail delivery which doesn't use authentication like SMTP does. Some examples I saw (Dovecot + postfix) appear to use a Unix socket to communicate, so that must be an option too, however I strongly suspect that this is very similar to the current implementation I'm working on.

Whilst I appreciate your suggestion, however I'm going to pursue my previous approach for now as that appears to be working so far as expected without needing to introduce another communication dialect or libraries (only to bundle a modified version of one). The way I see it, being able to configure Mailpit's SMTP server to listen on a Unix socket (and its sendmail implementation to communicate with that socket) will enable large-scale and automated setups with less headaches. This approach would for instance allow Docker containers to communicate between each other (or the host) without needing to know IP addresses or reserve TCP ports in advance. Of course everyone's setup is different, so this is likely for a very niche (but potentially large) "market".

[Corvan]

@axllent sure, go as you see fit, my thought was only, that you might be able to piggyback, but if course knowing that it could come with different repercussions and dependency. Just wanted to mention it, to have it mentioned, with my fractured knowledge of past times 😉

[axllent]

I am just in the process of releasing v.1.21.0 which includes this experimental feature. I say "experimental" as it's currently undocumented and pending testing from you (please).

The new syntax for setting a Unix socket instead of a TCP address for either (or both of) the HTTP and/or SMTP is unix:</path/to/socket-file>:<unix-permissions>. An simple example would be:

mailpit --listen unix:/var/run/mailpit/http.sock:660 --smtp unix:/var/run/mailpit/smtp.sock:660

This applies to environment variables too if you use those instead.

To use Mailpit's "sendmail" to send via the socket, you should use the same syntax (but without the permissions), so in this example:

mailpit sendmail -S unix:/var/run/mailpit/smtp.sock -t < message

It should also work (although I did not test it yet) with the -bs flag, such as the one Symfony likes to use. It would be great if you could test this with the MAILER_DSN="sendmail://default?command=/path/to/mailpit%20sendmail%20-S%20unix:/var/run/mailpit/smtp.sock%20-bs" (adjust to your environment of course).

Please let me know how this works for you, thanks 😄

[RafaelKr]

Awesome, thank you so much! Unfortunately I'm currently ill, I will look into it as soon as I'm feeling good again.

[axllent]

Sorry to hear that, and not a problem. Get better soon!

[RafaelKr]

I tested it inside my project using devenv.sh and it just works! 🎉

Also I built a little demo using Symfony + devenv: https://github.com/RafaelKr/mailpit-socket-devenv

[axllent]

That's awesome @RafaelKr, thanks for the feedback!

I started writing some documentation yesterday but got stuck on proxying using Apache. It seems Apache can proxy to a Unix socket, however I'm not having any luck proxying the websocket (I suspect this may just not be supported).

I'm going to leave this ticket open for now until I can release some documentation.

[axllent]

@RafaelKr I finally worked it out (Apache) which requires a recent version of Apache and a different config syntax than the one I am used to 🥳 I'm not even going to try the other proxies (Trafiek, caddy etc) as I have no experience with them.... anyway...

Please take a look at the documentation I've written with an example for both Nginx, Apache and "sendmail", and feel free to send a PR (to the website project) if you feel there is anything I should add or change. I have stated that this feature is experimental, both in the Mailpit release notes as well as on the website, but rest assured I won't be removing it. It just needs a lot more testing "in production" before I can state it's no longer experimental, so I would really appreciate your feedback (once you have started using this method in your infrastructure) to whether this has made it much simpler for you, and of course any issues.

I'll close this issue now, but feel free to comment on it when you can.