I've been using bkt (https://github.com/dimo414/bkt) for subprocess caching. It has some nice features, like providing a ttl for cache expiration. In-pipeline memoization looks nice, I'm not sure it supports that
I was not aware of bkt. Thanks for the link. It seems very similar to memo, and has more features:
- Explicit TTL
- Ability to include working directory et al. as context for the cache key.
There do appear to be downsides (from my PoV) as well:
- It's a rust program, so it needs to be compiled (memo is a bash/zsh script and runs as-is).
- There's no mention of transparent compression, either in the README or through simple source code search. I did find https://github.com/dimo414/bkt/issues/62 which mentions swappable backends. The fact that it uses some type of database instead of just the filesystem is not a positive for me, I prefer the state to be easy to introspect with common tools. I will often memo commands that output gigabytes of data, which is usually highly compressible. Transparent compression fixes that up. One could argue this could be avoided with a filesystem-level feature, like ZFS transparent compression. But I don't know how to detect that in a cross-FS fashion.
