We have run foundationdb in production for roughly 10 years. It is solid, mostly trouble free (with one very important exception: you must NEVER allow any node on the cluster to exceed 90% full), robust and insanely fast (10M+ tx/sec). It is convenient, has a nice programming model, and the client includes the ability to inject random failures.
That said, I think most coders just can't deal with it. For reasons I won't go into, I came to fdb already fully aware of the compromises that software transactional memories have, and fdb roughly matches the semantics of those: retry on failure, a maximum transaction size, a maximum transaction time, and so on. For those who haven't used it, start here: https://apple.github.io/foundationdb/developer-guide.html ; especially the section on transactions.
These constraints _very_ inconvenient for many kinds of applications so, ok, you'd like a wrapper library that handles them gracefully and hides the details (for example count of range).
This seems like it should be easy to do - after all, the expectation is that _application developers_ do it directly - but it isn't actually so in practice and introduces a layering violation into the data modeling if you have any part of your application doing direct key access. I recommend people try it. It can surely be done, but that layer is now as critical as the DB itself, and that has interesting risks.
At heart, the problem is, the limits are low enough that normal applications can and do run into them, and they are annoying. It would be really nice if the FDB team would build this next layer themselves with the same degree of testing but they themselves have not, and I think it's pretty clear that it turns out a small-transaction KV store is not enough to build complex layers in actuality.
Emphasis on the tested part - it's all well and good for fdb to be rock solid, but what needs to be there is that the actual interfact used by 90% of applications is rock solid, and if you exceed basic small-size keys or time, that isn't really true.
- If you’re a developer wanting to build an application, you should really use a well designed layer between yourself and FDB. A few are out there.
- If you’re a dev thinking you want to build a database from scratch you probably should just use FDB as the storage engine and work on the other parts. To start, at very least!
(One last thing that I think is a bit overlooked with FDB is how easy it is to build basically any data structure you can in memory in FDB instead. Not that it solves the transaction timeout stuff, etc. but if you want to build skip list, or a quad tree, or a vector store, or whatever else, you can quite easily use FDB to build a durable, distributed, transactional, multi-user version of the same. You don’t have to stick to just boring tables.)
I think i would say that for me the biggest issue is that what little there is in well-written layers are all java. Nothing against java but I'd be looking for Go and Rust, not Java.
We did use fdb to backend more complex data structures (b+ and a kind of skiplist) and it's very cool. fdb basically presents the model of a software transactional memory and it's kind of wonderful, but it's not wonderful enough.
Another issue that I forgot to mention is that comprehensibility of keys is your own problem. Keys and values are just bytes - if you don't start day one with a shared envelope for all writers, you _will_ be in pain eventually. This can get kind of ugly if you somehow end up leaking keys that you can't identify.
Not to take away from your main point, and I appreciate it, but I am interested in one minor you made which is - you wrote, "and insanely fast (10M+ tx/sec)"; When you say that, what does it mean, what's the context? Is it for the cluster, is it for one machine (what kind of cluster and networking, which machines, what machine), size of transactions, is there acknowledge after each, are they truly transactional or batch in one go..
Medium size multi-key transactions of substantial read-dependency complexity (many dependent keys loaded as part of the tx) and moderate key size; each tx of modest total size on the set side; This is an own-AWS cluster which means crap networking vs. a purpose-built on-prem network, instances with NVME storage.
fdb transactions are real transactions. These aren't batches.
CockroachDB is many cool things but not even remotely in a the same category as fdb in terms of the transaction rates it can deal with per unit cost and you wouldn’t be mapping complex data structures into cdb; it’s not what it is for.
They just aren't the same thing. It’s like comparing a binary tree to json. If you squint you can see how they could be similar but really aren’t.
That said, I think most coders just can't deal with it. For reasons I won't go into, I came to fdb already fully aware of the compromises that software transactional memories have, and fdb roughly matches the semantics of those: retry on failure, a maximum transaction size, a maximum transaction time, and so on. For those who haven't used it, start here: https://apple.github.io/foundationdb/developer-guide.html ; especially the section on transactions.
These constraints _very_ inconvenient for many kinds of applications so, ok, you'd like a wrapper library that handles them gracefully and hides the details (for example count of range).
This seems like it should be easy to do - after all, the expectation is that _application developers_ do it directly - but it isn't actually so in practice and introduces a layering violation into the data modeling if you have any part of your application doing direct key access. I recommend people try it. It can surely be done, but that layer is now as critical as the DB itself, and that has interesting risks.
At heart, the problem is, the limits are low enough that normal applications can and do run into them, and they are annoying. It would be really nice if the FDB team would build this next layer themselves with the same degree of testing but they themselves have not, and I think it's pretty clear that it turns out a small-transaction KV store is not enough to build complex layers in actuality.
Emphasis on the tested part - it's all well and good for fdb to be rock solid, but what needs to be there is that the actual interfact used by 90% of applications is rock solid, and if you exceed basic small-size keys or time, that isn't really true.