It's a blog on an open-source project that precisely tells you how to implement adaptive rate limiting.

Just click around a bit:

- https://github.com/fluxninja/aperture

- https://docs.fluxninja.com/use-cases/adaptive-service-protec...

Note: I am one of the authors' of this project.

Companies that pioneered the stateful reconstruction of APIs from raw network events -

- Netsil [0] (acquired by Nutanix) - Academic spin-out from UPenn

- Pixie Labs [1] (acquired by New Relic)

[0]: https://thenewstack.io/netsil-visualizes-performance-microse...

[1]: https://techcrunch.com/2020/12/10/new-relic-acquires-kuberne...

why do you think these companies, though successful in their own right, never became bigger standalone observability platforms?

While the tech these companies built (stateful reconstruction) is quite hard to engineer, large-scale observability platforms like Datadog were much more successful as they could cover a much larger surface area with minimal incremental effort. Even APM companies like New Relic and AppD struggled to justify the value they were delivering with their well-engineered code-level agents in the face of large-scale server/process metrics collection done by the Datadog agent.

In addition, these techniques have been prone to event misses (partial reconstruction) in case the ring buffer overflows and are ineffective in the face of SSL traffic.

concise summary. thank you! i guess simple is best.

Good observation. We were able to prevent issues due to limited context by sending the entire file in the prompt and the results were pretty amazing. We eventually reverted the change due to 2 reasons -

1. Limited tokens: 8K tokens is sometimes not enough to hold the entire file. Perhaps 32K tokens (or 100K Claude2 tokens) can help circumvent this.

2. Cost: GPT-4 is super expensive. Our current usage is roughly $20 a day but when we send files the usage shot up to $60 a day or so.

Our hope is that both the cost and token limit improve in the future so that we can send the entire file in each review request.

It’s all about providing relevant context in the prompts and using a better model where reasoning is needed, eg gpt-4.

If you are curious, some of our prompts are in the open source.

Here is the link to our open source version - https://github.com/coderabbitai/openai-pr-reviewer

Deploy Aperture for runtime protection.

https://github.com/fluxninja/aperture

Look at all these super knowledgeable “engineers” down voting this.

Load shedding is exactly what would prevent this in the future, scaling up capacity and adding a cache is only going to help buy some time.

Wish I could have a dollar for each time SREs duct tape rather than solve problems with systems thinking.

This is a purely informative comment. I do not mean any harm.

I believe you are getting downvotes because it's difficult to give a direct solution like you did without having more context on the matter.

> Look at all these super knowledgeable “engineers” down voting this.

Maybe people thought "look at this super knowledgeable 'engineer' shooting a one-off solution just by looking at an RCA"

We use Jsonnet extensively in Aperture project for generating control policies.

- Policy spec is expressed in protobuf format

- GRPC gateway plugin is used to generate Swagger (OpenAPI v2) spec from proto files

- Jsonnet bindings are generated from Swagger spec.

- Blueprints are implemented using Jsonnet bindings. The users generate policies from blueprints by providing configuration in yaml (using aperturectl CLI) or via jsonnet mixin.

Relevant links:

- Aperture Blueprints (Jsonnet): https://github.com/fluxninja/aperture/tree/main/blueprints

- Generated doc example: https://docs.fluxninja.com/development/reference/policies/bu...

- CLI for generating policies from blueprints: https://docs.fluxninja.com/development/reference/aperturectl...

- Policy spec: https://docs.fluxninja.com/development/reference/policies/sp...

I'm interested in the code that processes the swagger/openapi into jsonnet, can you link to that?

It’s customized to our policy spec. But you can learn from this and adapt it to your spec.

https://github.com/fluxninja/aperture/blob/main/scripts/json...

Multi-prompt[0] approach taken while summarizing large number of files and diffs. Prompts has been tuned for a concise response. To prevent excessive notifications, this action can be configured to skip adding review comments when the changes look good for the most part.

[0] : https://github.com/fluxninja/openai-pr-reviewer/blob/main/ac...

This is probably what HN operators should use! They keep crashing due to overloads and logged-in users.

DoorDash team talks about types of metastable failures in microservices and how observability-based control techniques can be used to mitigate them.

They introduce a project called Aperture[0] that provides techniques such as adaptive concurrency limiting, prioritized load shedding and graceful degradation to mitigate cascading failures.

[0] https://github.com/fluxninja/aperture

[1] https://docs.fluxninja.com/