memory storage

[elee1766]

renames existing memoryStorage -> testingMemoryStorage, as it is a special storage only used for testing

creates memoryStorage, which can only be created through calling NewMemoryStorage

memory storage is thread safe, supports lock keyed by file

[mholt]

Thanks! Is there a reason we need two? Like, why not just use the new memoryStorage for testing too?

[elee1766]

Thanks! Is there a reason we need two? Like, why not just use the new memoryStorage for testing too?

i'm not sure.

// testingMemoryStorage is an in-memory storage implementation with known contents and fixed iteration order for List.

i assume those things matter for testing, and the map based in-memory store can't do things, so my guess is we can't.

[mholt]

Hmm. I still wonder if we can just use the one for both. (In other words, keep the new memoryStorage type, and just use it instead of the new testingMemoryStorage type.) I haven't had a chance to explore this but could we possibly try it if anyone has a chance?

[elee1766]

i can try to look for some time to investigate. i am usually afraid of changing existing tests.

[ankon]

The "original" memory storage was added with #283, and indeed the predictable order of the its List method is used in one of the tests (specifically the one that checks that broken accounts are correctly skipped). See also #283 (comment)

That being said, I guess the "new" memory storage is intended not so much for the purpose of unit tests, but rather actual deployments: In that case I wonder whether using a tmpfs-backed file storage might be a better approach anyways, as it would give the ability to actually "look at" the stored contents?

[ankon]

I think this needs updating?

[ankon]

Suggested change

	// memoryStorage is a Storage implemention that exists only in memory
	// memoryStorage is a Storage implementation that exists only in memory

[elee1766]

@ankon

using actual tmpfs may be an issue because of portability, is why i did not opt for this. especially on windows hosts, i don't know how to create a ramdisk, and afaik there is no tmpfs-like thing that you can easily access that exists in page cache.

it is true though, using tmpfs would reduce the memory overhead on linux (in exchange for possibly higher cpu usage, i think)

if you had thoughts for anything cross-platform that might work, would be happy to consider and implement.

I think matt would have to approve new dependencies if they were required. personally i use https://github.com/spf13/afero memory backed fs for cases like these - but i didn't want to pull the dependency in.

[ankon]

Well, indeed tmpfs would be the "Linux-thing", but the point here would be to have it out of the scope of certmagic: It simply is a question of "configure the thing you're using certmagic in such that the storage points to a suitable file system" (which, in the case of "testing" and "linux" could be a tmpfs mountpoint).

I did a quick look now, and it seems that Windows doesn't come out-of-the-box with any such functionality, so when running on Windows that wouldn't work.

I guess what I'm trying to say is: Adding a in-memory storage is an interesting idea, but I wonder whether that is something certmagic needs to have, or whether that could be easily done outside of it -- for instance a library that provides a in-memory storage (using github.com/spf13/afero, for instance) could just live in a separate repository, and certmagic could point to it in the README.