Prioritized load shedding works well as a last resort [0]. The idea is simple -

- Detect overload/congestion build-up at the database

- Apply queueing at the gateway service and schedule requests based on their priority

- Shed excess requests after a timeout

[0]: https://docs.fluxninja.com/blog/protecting-postgresql-with-a...

Fascinating historical insight!

We have a team of 20 engineers currently working on solving this problem in the context of API requests and service chains. Do you know JMS @ Penn? Asking because he did some work in ATM networks, QoS etc. He is advising us on the project.

This is the link to the project: https://github.com/fluxninja/aperture

We built a weighted-fair queueing scheduler as well - https://docs.fluxninja.com/concepts/scheduler/load-scheduler

We built a fair scheduler for APIs and it's in the open source - https://docs.fluxninja.com/concepts/scheduler/load-scheduler

I wish more people knew about this project!

You should check out the Aperture flow control system - https://github.com/fluxninja/aperture

We built a weighted fair queueing system in the Aperture flow control system to help alleviate the API load pressure - https://docs.fluxninja.com/concepts/scheduler/load-scheduler

And in addition, we are investing in the graceful-js library to handle 429 and 523 codes returned by the Aperture system - https://github.com/fluxninja/graceful-js

Very interesting blog post! Our team has been working intensively in this area for the last couple of years - flow control, load shedding, controllability (PID control), and so on.

We have open-sourced our work at - https://github.com/fluxninja/aperture

We also did a Twitter Spaces discussion with Kelsey earlier today - https://twitter.com/kelseyhightower/status/16893552848026296...

We would love feedback from folks reading this blog post!

Disclaimer: I am one of the co-authors of the Aperture project. There are several interesting ideas we have built into this project, and I will be happy to dive into the technical details as well.

Very interesting blog post! Our team has been working intensively in this area for the last couple of years - flow control, load shedding, controllability (PID control), and so on.

We have open-sourced our work at - https://github.com/fluxninja/aperture

We would love feedback from folks reading this blog post!

Disclaimer: I am one of the co-authors of the Aperture project. There are several interesting ideas we have built into this project and I will be happy to dive into the technical details as well.

AI helps as well. For instance, our company has extensively used coderabbit.ai since its inception.

Came across this very interesting GitHub pull request of an AI bot reviewing another AI bot’s generated code!

Is this the way of the future?

Aperture project's policy language is indeed inspired by Control Systems, especially PID Control.

See: https://docs.fluxninja.com/concepts/advanced/circuit

It’s a blog post, it will be judged by its content. It does feel like at a minimum it was heavily edited with an AI assistant, verbose and elegant but light on specifics. And if the goal is to get people hooked on how to implement adaptive rate limiting (and maybe adopt fluxninja) it doesn’t seem to be hitting the right note.

Thanks for the feedback :). I will try to be more specific in next blog.