> If you can convince the lowest white man he's better than the best colored man, he won't notice you're picking his pocket. Hell, give him somebody to look down on, and he'll empty his pockets for you.
"I'll tell you what's at the bottom of it. If you can convince the lowest white man he's better than the best colored man, he won't notice you're picking his pocket. Hell, give him somebody to look down on, and he'll empty his pockets for you."
what gets me is trying to make the argument that market economies are not necessarily capitalist economies. it seems plain over time that capitalism works to destroy markets. As an American I'm pretty pro market, but that means at this point I'm an enemy of capitalism.
Ah yes, and every consumer should just do this in a while (true) loop as producers write to it. Very efficient and simple with no possibility of lock contention or hot spots. Genius, really.
I've implemented a distributed worker system on top of this paradigm.
I used ZMQ to connect nodes and the worker nodes would connect to an indexer/coordinator node that effectively did a `SELECT FROM ORDER BY ASC`.
It's easier than you may think and the bits here ended up with probably < 1000 SLOC all told.
- Coordinator node ingests from a SQL table
- There is a discriminator key for each row in the table for ordering by stacking into an in-memory list-of-lists
- Worker nodes are started with _n_ threads
- Each thread sends a "ready" message to the coordinator and coordinator replies with a "work" message
- On each cycle, the coordinator advances the pointer on the list, locks the list, and marks the first item in the child list as "pending"
- When worker thread finishes, it sends a "completed" message to the coordinator and coordinator replies with another "work" message
- Coordinator unlocks the list the work item originated from and dequeues the finished item.
- When it reaches the end of the list, it cycles to the beginning of the list and starts over, skipping over any child lists marked as locked (has a pending work item)
Effectively a distributed event loop with the events queued up via a simple SQL query.
Dead simple design, extremely robust, very high throughput, very easy to scale workers both horizontally (more nodes) and vertically (more threads). ZMQ made it easy to connect the remote threads to the centralized coordinator. It was effectively "self balancing" because the workers would only re-queue their thread once it finished work. Very easy to manage, but did not have hot failovers since we kept the materialized, "2D" work queue in memory. Though very rarely did we have issues with this.
Yeah, but that's like doing actual engineering. Instead you should just point to Kafka and say that it's going to make your horrible architecture scale magically. That's how the pros do it.
Kafka isn't magic, but it's close. If a single-node solution like an SQL database can handle your load then why shouldn't you stick with SQL? Kafka is not for you. Kafka is for workloads that would DDoS Postgres.
Kafka is really not intended to improve on this. Instead, it's intended for very high-volume ETL processing, where a classical message queue delivering records would spend too much time on locking. Kafka is hot-rodding the message queue design and removing guard rails to get more messages thru faster.
Generally I say, "Message queues are for tasks, Kafka is for data." But in the latter case, if your data volume is not huge, a message queue for async ETL will do just fine and give better guarantees as FIFO goes.
In essence, Kafka is a very specialized version of much more general-purpose message queues, which should be your default starting point. It's similar to replacing a SQL RDBMS with some kind of special NoSQL system - if you need it, okay, but otherwise the general-purpose default is usually the better option.
Of course this is not the same as Kafka, but the comment I'm replying to:
> Ah yes, and every consumer should just do this in a while (true) loop as producers write to it. Very efficient and simple with no possibility of lock contention or hot spots. Genius, really.
Seemed to imply that it's not possible to build a high performance pub/sub system using a simple SQL select. I do not think that is true and it is in fact fairly easy to build a high performance pub/sub system with a simple SQL select. Clearly, this design as proposed is not the same as Kafka.
No, I implied that implementing pub/sub with just a select statement is silly because it is. Your implementation accounts for the downfalls of this approach with smart design using a message queue and intelligent locking semantics. Parent of my comment was glib and included none of this.
It's one of my favorite patterns, because it's the highest-impact, lowest-hanging fruit to fix in many systems that have hit serious scaling bottlenecks.
I was in the midst of writing a snarky reply and then realized my actual issue with Kafka is that people reach for it way too often and use it in ways that don't really make sense.
Kind of like how people use docker for evrything, when what you really should be doing is learn how to package software.
Agree on the Kafka thing though. I've seen so many devs trip over Kafka topics, partitions and offsets when their throughput is low enough that RabbitMQ would do fine.
The people distributing software should shut them damn up about how the rest of the system it runs in is configured. (But not you, your job is packaging full systems.)
That said, it seems to me that this is becoming less of a problem.
Nothing inherently wrong with the core product IMHO. The issue is more with Confluent, who have been constantly swinging from hot buzzword to hot buzzword for the last few years in search of growth. Confluent cloud is very expensive, and you still have to deal with a surprising amount of scaling headaches. I have people I consider friends that work there, so I don't want to go too deep into their various missteps, but the Kafka ecosystem has been largely stagnant outside of getting rid of Zookeeper and simplifying operations/deployment. There have been some decent quality of life fixes, but the platform is very expensive, yet if you are really all-in on Kafka, you would be insane to not get support from Confluent- it can break in surprising ways.
So you are stuck with some really terrible tradeoffs- Go with Confluent Cloud, pay a fortune, and still likely have some issues to deal with. Or you could go with Confluent Platform, still have to pay people to operate it, while Confluent the company focuses most of their attention on Cloud and still charges you a fortune. Or you could just go completely OS and forgo anything Confluent and risk being really up the river when something inevitably breaks, or you have to learn the hard way that librdkafka has poor support for a lot of the shiny features discussed in the release notes.
Redpanda has surpassed them from a technical quality perspective, but Kafka has them beat on the ecosystem and the sheer inertia of moving from one platform to another. Kafka for example was built in a time of spinning rust hard disks, and expects to be run on general purpose compute nodes, where Redpanda will actually look at your hardware and optimize the number of threads its spawns for the box it is on- assuming it is going to be the only real app running there, which is true for anything but a toy deployment.
This is my experience from running platform teams and being head of messaging at multiple companies.
So much that we presume in the modern cloud wasn't a given when Apache Kafka was first released in 2011.
kevstev wrote just above about Kafka being written to run on spinning disks (HDDs), while Redpanda was written to take advantage of the latest hardware (local NVMe SSDs). He has some great insights.
As well, Apache Kafka was written in Java, back in an era when you were weren't quite sure what operating system you might be running on. For example, when Azure first launched they had a Windows NT-based system called Windows Azure. Most everyone else had already decided to roll Linux. Microsoft refused to budge on Linux until 2014, and didn't release its own Azure Linux until 2020.
Once everyone decided to roll Linux, the "write once run everywhere" promise of Java was obviated. But because you were still locked into a Java Virtual Machine (JVM) your application couldn't optimize itself to the underlying hardware and operating system you were running on.
Redpanda, for example, is written in C++ on top of the Seastar framework (seastar.io). The same framework at the heart of ScyllaDB. This engine is a thread-per-core shared-nothing architecture that allows Redpanda to optimize performance for hardware utilization in ways that a Java app can only dream of. CPU utilization, memory usage, IO throughput. It's all just better performance on Redpanda.
It means that you're actually getting better utility out of the servers you deploy. Less wasted / fallow CPU cycles — so better price-performance. Faster writes. Lower p99 latencies. It's just... better.
Now, I am biased. I work at Redpanda now. But I've been a big fan of Kafka since 2015. I am still bullish on data streaming. I just think that Apache Kafka, as a Java-based platform, needs some serious rearchitecture,
Even Confluent doesn't use vanilla Kafka. They rewrote their own engine, Kora. They claim it is 10x faster. Or 30x faster. Depending on what you're measuring.
I think Age of Disclosure is coming out. Might be interesting to watch if only to scratch your head about how many military and defense people are trying to convince people the opposite, that there are craft with extraordinary capabilities and they ARENT ours.
I made fun of UFO people the same as I did the religious and bigfoot believers, but after the video in the Times in 2017, and all the military people who have come forward with roughly the same story.. I would very much like to know whats behind it.
People in the military saying this stuff doesn’t really change anything. People in the military are just a cross-section of the other morons that make up this country.
Over 40% of Americans believe that some UFOs have been alien spacecraft according to Gallup.
In other words, 40% of Americans are fucking morons. [1]
How many people believe that the virgin Mary magically got pregnant without having sex? And that’s something we know is scientifically impossible, even more so than the wildly improbable phenomenon of an alien spacecraft visiting us.
A whole bunch of people in the military believe the Virgin Mary was a virgin, too.
[1] This 30-40% number seems to also line up with a host of dumbass political opinions like being pro-authoritarianism. At any given point, around 30% of people in the world seem to be dumb as fuck.
As someone formerly a member of the the Center for Inquiry and still a believer in its mission, I understand what you're saying.
I would still point you to, there is something to this. And theres not been any whistleblower mention of anything "visiting." What they report aligns with the most probable scenario that the things we're seeing are automated Von Neumann probes, most likely sitting at the bottom of the ocean, and they've probably been here a long time.
I agree with that, which is why its pretty interesting the whistle blowers who went before Congress used the term "non-human intelligence" pretty extensively.
In any case the point the article makes and the one I'm most interested in, is it looks like we've been lied to.
I don’t get this conclusion at all, please explain. The article makes estimates without showing their work, then arrives at some probability of a coverup. A coverup where the government releases footage and documents. They just don’t agree that the reports and footage constitute evidence that is compelling.
if you believe like i do that gpg is an awful dead and outdated tool you'd never enforce on your worst enemy, its a good time to mention sops supports age.
"they" dont detect it light years away. from what has been published in the public domain, like in the congressional hearings, there are machines that have been here possibly a very long time. think at least 500 years. theres no indication the "they" behind those machines have ever made contact, or are on this planet, or are even still in existence. but there is something mechanical here that was not made by us.
as far as I know, the very few publicly identified records of speeds we have suggest a really big power source and the probable manipulation of fields we cannot yet (mass/gravity), but nothing breaking the speed of light.
so many people make leaps beyond the evidence we have and then declare them not plausible.
Having read what's in the public domain from the hearings, I'm having trouble squaring these two things:
> there are machines that have been here possibly a very long time. think at least 500 years. theres no indication the "they" behind those machines have ever made contact, or are on this planet, or are even still in existence. but there is something mechanical here that was not made by us.
> so many people make leaps beyond the evidence we have and then declare them not plausible.
Why isn't the first one exactly what you're complaining about in the second statement? Have I missed something?
GOP apparently had enough votes to pass a budget by reconciliation. Which means the riders they want to add on, dropping Obamacare funding expansions are at minimum important enough to shut down over.
They do not seem to be acting in good faith, not sending people to negotiate any of this. Combined with the leaking presidents comments about being able to force through things under shutdown they wouldn't be able to otherwise, I think a reasonable interpretation is this shutdown is intentional and part of someone's plan.
edit: since subtext is dead its called Project 2025 and it's supposed to be a "bloodless coup" of the federal government. And if that isn't obvious by now please wake up.
Also the current OMB director along with the President have apparently decided they can just carry out rescissions of any spending they don’t like, even though the spending is congressionally mandated. Republicans in the house don’t care to address the fact they’ve ceded the Purse to the White House so why would Democrats negotiate a spending bill when the president can decide he’s not going to follow-through on D priorities after the bill is approved?
True! But I think in general, those were the exception, not the rule. Gingrich changed the party as a whole to grab power more than anything else, including respecting democracy or honesty.
