Suggest: add key expiration

[cye1024]

write key with expiration. while the expire time arrives, delete the key file.

[peterbourgon]

This would require an active component in diskv that doesn't currently exist.

[dertuxmalwieder]

In my opinion, there already are enough Go built-ins for this functionality. In rssfs, I solved it by calling a timer after each .Write().

[GwynethLlewelyn]

@cye1024 if what you have in mind is a garbage collector, I guess you could always delete the files on a separate goroutine (not tested!) and let diskv give an error if a certain key has no file on disk — and use Import to get it again.

An alternative would be to use keys that embed the expire time in it, and do a check before Getting the item from disk: if the current time is bigger than the expire time on the key, well, discard the key and remove the key file.

[peterbourgon]

@dertuxmalwieder There are a lot of problems with this approach: it leaks goroutines, it's not cancelable, there's no way to know if it will or has already executed, and so on.

[benyanke]

@peterbourgon Additionally, that does not work for things like a CLI, where there will not be one consistent execution of the binary. The storage of the TTL really ought to happen alongside the value being stored. And even for long-lived executions, you lose all your TTL functionality if you ever need to redeploy your application or restart the service.

[peterbourgon]

One consistent execution of diskv for a given directory is a core requirement.

[r--w]

I've emulated TTL by using an additional layer of in-memory caching. Usually, these caches have a user-defined function running on eviction, so by using that, you can remove objects from diskv.

I use github.com/patrickmn/go-cache with OnEvict method.

Implementation details are at github.com/coinpaprika/echo-http-cache/blob/master/adapter/disk/disk.go.

One note: we don't need the cache to survive rollouts, so there is no need for embedded TTLs.

[dolmen]

@dertuxmalwieder 's workaround could be implemented more efficiently using time.AfterFunc.

However that simple solution doesn't handle the case where a key is refreshed (the key will be deleted after the expiration of the first timer for that key, and not kept even if the value has been refreshed before expiration).

[s290305915]

You can refer to the following code, which is from another project I wrote:

type Config struct {
	ExpiryTime string `json:"expiry_time"`
	Data       string `json:"data"`
	Key        string `json:"key"`
}

func WriteCache(key string, data string, expiration time.Duration) error {
	cfg, err := json.Marshal(data)
	if err != nil {
		return err
	}

	configData := fmt.Sprintf(`{"expiry_time":"%s","key":"%s","data":%s}`, expiryTime.Format("2006-01-02 15:04:05"), key, string(cfg))
	keyFile := fmt.Sprintf("./models/db/db_%s", key)

    ...
}

func ReadCache(key string) (*Config, error) {
	keyFile := fmt.Sprintf("./models/db/db_%s", key)
	data, err := os.ReadFile(keyFile)
	if err != nil {
		return nil, err
	}

    var config *Config
	err = json.Unmarshal(data, &config)
	if err != nil {
		return nil, err
	}

    ...

    now := time.Now()
	expTime, _ := time.ParseInLocation("2006-01-02 15:04:05", config.ExpiryTime, time.Local)
	if now.After(expTime) {
        // delete the file
		return nil, nil
	}
	return config, nil
}

The generated data uses JSON

{"expiry_time":"2123-05-05 01:04:22","key":"ddd123","data":"78"}

Mainly, this may cause some performance issues.
Also, there's a suggestion to put the expiry time on the file name. When getting the file through "key_expire_time", we can know whether it has expired or not. Moreover, we can asynchronously delete these data.