As someone who has worked in the media industry, I can tell you that the problem with unbiased news goes much deeper than simply presenting facts objectively. The very first step mentioned in your breakdown - scraping the latest headlines from publicly available news sites - is inherently biased, because it relies on a small selection of sources to determine what is considered newsworthy. But by whom? In reality, editors play a crucial role in shaping the news by deciding which events to cover and which to ignore, based on their own biases and interests.
From my own experience, I can tell you that the most important news stories are often the ones that go unreported/underreported by most news sites. Unfortunately, there are also many insidious methods used to shape public opinion, such as distraction etc. But it all starts with the decisions made by editors about WHAT YOU, as a reader, should SEE and CARE about. So if we truly want to achieve unbiased news, we need to start by reevaluating the very definition of what is news.
I find the closest to an unbiased news source is Reuters.
I removed all other news apps from my phone and only leave the Reuters news app installed. [1]
I never read the news anymore.
The combination of the uncurated deluge of articles about stuff I don't know anything about (and realize I am not going to ever really know anything about) and them not really poking my buttons with editorialized headlines makes me just ignore it and come here more often instead.
Edit: I fathom its because the Reuters business model is fundamentally different - they are just trying to supply as many articles for use as raw base material factoids to all types of media organizations who can then add their own "secret sauce/value" by curating which articles their audience "sees" and handle the "interpretation" of the factoids (i.e. insert the editorial opinion with headlines, quips for their audiences will appreciate). That said - I do recall reading about some instances that Reuters is not perfect (article rushed to press and not entirely factually correct).
Reuters isn't so much a source of news, it's a (top level) news aggregator.
A source would be an experiencer telling their story about the events considered new, i.e. unexpected.
News are updates of our collective model of the world.
Human communication has the aspects of content, sentiment and style. People look at style first, as they take it as an indicator for the social standing of the speaker. Sentiment is a close second, as it tells you about the attitude that speaker has towards the reported events. Content lastly is just the events themselves as they (reportedly) happened.
People find that content part hard to follow on its own, mostly because they usually don't really know the context anyway. Focusing on it as supposedly important "unbiased" "true" news is a gross misunderstanding.
Like yourself, people generally have a very incomplete idea of the world they live in.
Like you, they don't seem to take much interest in it, as they view it as "not relevant to their lives".
That is an absurd statement to make, if one has no clue about that part of the world to begin with, like you observed.
> “ In 1969, Reuters needed money to further expand in the Middle East and Western powers such as Britain wanted to bolster their influence against the Soviet Union by expanding news services across the world, the documents showed.
The secret government financing of Reuters - as set out in the documents - amounted to 245,000 pounds ($317,838 at current exchange rates) per year before 1969 but then reduced to 100,000 pounds per year in 1969-1970 and nothing in 1972-1973.”
I don’t grasp what the article’s writer meant by “current exchange rates” and “before 1969” here, but in any event it sounds like only a few million real constant 2023 dollar equivalents, total, were involved, which is less like a secret think tank and more like travel and bureau support, which is basically what is being admitted to.
> I find the closest to an unbiased news source is Reuters.
Maybe for American news it is unbiased across conservatives / liberals, however from a non-American point of view Reuters is very VERY much biased towards US / CIA (even though it is incorporated in England).
Underreporting on violence in certain communities vs others is biased. Who determines what is uncommon vs common? Who decides the value of 4 people in Riverdale vs 4 people in Beverly Hills? Just going by “frequency of shootings” is a form of bias.
"Unbiased" is like absolute zero. You never actually get there, but don't think there is no distinction between putting in an effort to make something decently cold, and hurling it into the sun.
Imagine, you had some tool affording you complete information about everything happening on earth, aided by some AI that only tells the truth about anything when asked.
You would still have to pose the right questions or risk dying in ignorance.
People's complaints here about bias is really about being manipulated successfully by parts of society working actively in their own, markedly different, interests.
Those won't just go away. Aligning interests doesn't happen by providing facts.
I realise now that my messaging with this project is off.
What would more accurately explain the purpose of this project is to reduce the effects of sensationalism and indoctrination in the news.
In other words, provide a platform for readers to make up their own opinions on news stories by filtering out the excessive positive/negative language.
Even if NewsNotFound articles are only 90% neutral, that's 90% BETTER than reading the same story from a sensationalist source.
If I understand correctly, you want to help people get an objective baseline of information about the world, they then can compare their individual bias against. Only thing is, I don't think most people care about that.
Humans provide their sentiment about developments to further their individual interests. Other people want to be informed about those motivations. Taking that away makes news subjectively meaningless for most.
There is a minority of truth-seekers who essentially individually engage in OSINT activity (if only by consuming more than one source).
An AI "doing the thinking for them" isn't the right way though (trust is an issue, remember). You want to teach people, how to be an intelligence analyst and give them the tools to avoid the drudgery.
Isn’t the aspiration to be unbiased worthy already?
The goal or call it “aspiration” to serve unbiased news is admirable to me, even while my rational mind knows the aspiration is an unattainable ideal. I will give them a try. Then I can judge their sincerity and performance relative to their aspiration.
It sounds to me like you want to rely on quantifiable science in all things. Yes science is founded upon math which is founded upon logic. Both math and logic are axiomatic systems, and axioms are assumed or declared. They are not independent of a person's mind either. You cannot escape it. David Hume sorted it out for us long ago. Rational certainty is a pipe dream. But that need not prevent us from navigating this world and enjoying life.
Back to the topic: we seem to be talking about a human being choosing between competing news sources. As that consumer, you get to define "unbiased" for yourself. Wise people have given it a lot of thought, so you might consider consulting what they said.
And you can draw the line at something being measurable to matter to you. That's your choice. I have found the things that matter most are fictions in my brain: love comes to mind.
It is also not necessary to measure to detect a distance. I love my dog, and I love my child. I cannot measure love, yet I detect a significant distance between my love for my dog and my love for my child.
Like love, "bias" has meaning even if it cannot be measured. Like love, one can detect distances between biases, even if they cannot be measured. Science (measurement), math (quantity), and logic (relations of ideas) are just as ephemeral, because they are founded upon unprovable axioms and norms.
Objective observation is really hard, but amenable to systems and rules that help get closer to this ideal. I'm convinced it's possible to do far better than existing news organisations do just through approaching it with an engineering mindset.
I do believe in science, I think perhaps you have a very biased, or at least limited, view of what “believing in science” means.
There is no way to judge whether or not you are close to some objective view of “unbiased news”. So it is not a worthwhile pursuit.
As others have noted, what is “newsworthy” is an entirely subjective prioritization of events and information. Are four deaths in Riverside news? Are four deaths in Palm Springs? Are both? Are neither? It depends on your subjective priorities, as a consumer, and no single news organization is ever going to achieve universal, subjective, but somehow unbiased news prioritization.
Labeling anyone person or group’s prioritization of information as “truth” is a dubious endeavor. Dubious to the point of being laughable.
I wonder if this is true in reality. Isn’t it possible and maybe even likely that a non-neutral party is providing the most accurate version of the story? For example, a person or reporter living in a war zone, experiencing the actual war is likely biased in some fashion, but their recollections of what is actually happening is quite accurate compared to a “neutral” journalist sitting in their comfortable office in New York City, referring to various anonymous sources and reports from multiple perspectives.
Also, filtering out the positive and negative language might hide the bias of the article, making it seem more authoritative and believable. Many times when I read an article, and I see the journalist making conclusions with bombastic and unbalanced language, it is helpful information toward allowing me to detect bullshit and avoid that source.
3. Information that doesn't affect me but I still care to know
4. Information that won't affect me
5. Information that won't affect me and I don't care to know
6. Information I don't care to know even if it affects me
Surely there must be a way to monetize information delivery. I would pay for an editorial-less stream of information that is relevant to me, and that isn't that I still want to know.
I used to think that unbiased news is an intractable problem purely based on information theory: to have an unbiased picture of the world, you need to have all the facts, which is physically impossible; and any layer that simplifies those facts (be it a journalist or an LLM) will add bias (either by journalist, journalist’s employer, LLM’s training data or operator preferences, and so on). In short, when your selection of facts to report is already biased, there’s not much you can do. In even shorter, map is not the territory (and all you have is a map).
However, I realized that’s not really true.
I mean, of course you cannot have unbiased news, and reevaluating the definition of “news” is not going to help you.
It’s just that there is actually a very good mechanism for combatting this—and it’s called “be aware of your bias and upfront about it”. There is any other way around it.
To add: where I wrote “be aware of your bias and upfront about it” I of course meant journalists and news organizations.
Corollary: since this awareness of the bias requires self-honesty and introspection, it’s drastically more difficult with an LLM because you also need to know all about your LLM developer and operator’s accidental and intentional biases.
Technically speaking, if every day you take an uniformly random selection of news articles, on average what you read will not be biased. Or, to be precise, it will follow the same biases as the population of news articles you sample from.
It's almost impossible to arrive at a pure random sampling like that. As an easy example, if you limit yourself to English news articles, you're going to have a pretty large bias towards "the West", which right now probably includes a strong bias against China or Russian's viewpoints on things (not to say that including those viewpoints would be more truthful necessarily, but certainly less biased).
The difference is that facts are harder to consume than news articles, as the latter is essentially packaged and distributed collections of the former.
Furthermore, the intent of parent's bias averaging, as I understand it, is...
1) There are a finite number of major biases axes in mass media (e.g. political parties, wealthy-poor, conservative-liberal, etc.), ergo averaging these out is a helpful reduction of major biases
2) It assumes the underlying material has a bias, and seeks to counteract it. If you build a fact-based method, you leave yourself open to treating biased opinions as facts, if they're sufficiently masked
Indeed, the first challenge to unbiased approach would be the impossible in practical terms task of separating facts from fiction.
I don’t know if it is clear but I really meant it that unbiased news is not a possibility due to fundamental limitations of information theory and finiteness of individual information processing capabilities; self-awareness and bias transparency is the only way.
> helpful reduction of major biases
Only if those major biases are opposite between media.
Imagine biases as vectors pointing in arbitrary directions. Best case is when bias vector in one cluster is compensated by exactly opposite bias vector in another cluster, but how often do you expect that to happen? A more typical scenario is a vector pointing in merely a different, rather than opposite, direction. A worst-case scenario, and also common (because bias manifests itself in selection if facts to report, and most sources report on the same events), is when all bias vectors point in a similar direction.
In your original (upthread) argument that "unbiased news is not a possibility", are you stating that as a fundamental consequence of the news' lossy compression of {all facts} into a smaller product... or something else? It's unclear.
As for averaging out, yes, the vector model was what I was using internally.
Disclosure: I'm US-based, so our politics are traditionally bipolar as a result of our election system (in contrast to Europe et al.).
I would be surprised if vectors don't cancel often though, because think what we're really talking about in a capitalist information space: popular perspectives that are profitably marketable.
What is the easiest way to find an audience? Do what your competitors are not.
So on the whole, high revenue and production cost media (tv, web) in a capitalist information space will converge into bipolar pairs on any given issue, in order to maximize market share.
In lower cost media or operations, it will more likely be a random scattering of vectors.
> Disclosure: I'm US-based, so our politics are traditionally bipolar as a result of our election system (in contrast to Europe et al.).
There's a whole lot of bias that's "baked in" with US politics / media. There's broad consensus on neoliberal capitalism, personal freedoms vs collectivism, common ground on opposing states like Russia and China.
On a few wedge issues you will average out to a common ground, but ultimately you're not eliminating bias, you're just eliminating a relatively small bias between left-wing and right-wing US and replacing it with an "overall US" bias.
From mass market sources, true. Probably because they're aiming towards the common American, who generally has similar feelings. Ergo, they're that + whichever politically-polarized biases.
As for whether averaging that out is useful...?
I'd be happy to have news from a "overall US" bias, without the right/left political spin.
1. Identify all clusters of power and opinion, eg. Republicans, Democrats, Chinese Communist Party, UK Labour Party, Green Party, etc.
2. Associate each cluster with news organizations that tightly align with their policies.
3. Evaluate stories - headlines in only one cluster (under reported elsewhere) represent a view. Headlines that have different takes, likewise, give you a spectrum of thought on an issue or topic.
4. Collect fine points and build a report. Bonus for providing contrasting opinions and associating them with the party/power responsible for that thinking.
I want this tool. It'd be okay if it wasn't perfect. Just getting 80% of the way there would be incredibly valuable. I'd pay for it.
1. Will your cluster identification be unbiased? How do you avoid providing more fine-grained taxonomy for clusters that hit close to home for you (or this tool’s operator) and shoving the rest into as few buckets as possible (because you don’t care)? 2. What about biases shared by all news organizations, regardless of cluster?
When I started working on Zeitgaist [1] I immediately recognized that biased information is the main problem. I'm currently thinking about attaching additional information to the sources I present like ground [2] is doing with political spectrums. I really like your idea for the news algorithm. If you or someone elsw wants to build something like that on top of Zeitgaist with me, don't hesitate to contact me.
I don't see why you can't. Throw religion in while you're at it. Even interest groups around other classes: disability, socioeconomic, etc.
You can absolutely say a newspaper (or individual columnist or editor) is pro-LGBT, pro-Israel, pro-Palestine, pro-Mormon, pro-Amish, pro-Atheist, etc. That shouldn't be controversial. Wikipedia can and does do this in a relatively neutral way.
If your classification is meant to disparage a group, then I'd see the trouble. Don't do that.
I want an up to date, well rounded view of how everyone sees the world. A system capable of getting me to see geopolitical events in other perspectives is useful, even if I ultimately choose to uphold my existing beliefs.
FWIW, I subscribe to all sorts of diverse political view subreddits and Twitter handles. Unfortunately, it's still hardly comprehensive is a lot of work to analyze and distill.
What we're really talking about here is boting any deviation from a tabula rasa appraisal of reality, then clustering that into an affiliation/quality, right?
I.e. Christians (as an averaged whole) can look at the same situation as a blank slate and have a statistically significant and consistently different perspective
As can scientists. As can maybe microbiologists and industrial chemists.
The point of the clustering isn't that the tags are pejorative, but that they're predictive metadata on any source's bias.
And often times that's currently unexposed. I see a post by echelon, but only by the content of that single post (and maybe some prior HN contact) can I reconstruct his typical perspectives.
Because the project is small right now (and makes no money), I've included a very limited list of sources for each category, however at scale we would have alot of options for getting past this.
For example, one way in theory is to scrape all known publicly available news sites and create an article for every single detected story, but this would be very expensive.
As someone with experience in the media, what steps would you take to find these under-reported stories? Maybe the steps can be replicated with AI to increase coverage?
I think the The idea of this is interesting but I agree with you, Using titles to correlate results that are fact based isn't a greatest way to find unbiased information. You would still need some kind of variable that was not so well known. echo chambers are created by repeating information in similar ways. Its been my experience that echo chambers usually are the least reliable.
Your comment reminded me of this quote from Howard Zinn (who had his own biases, of course):
But there is no such thing as a pure fact, innocent of interpretation. Behind every fact presented to the world—by a teacher, a writer, anyone—is a judgment. The judgment that has been made is that this fact is important, and that other facts, omitted, are not important.
What are some of your thoughts on the subject?
How can somebody achieve unbiased news without editorial supervision? Would a quorum solve the issue with editorial biases creeping into news?
Well, I just tested if quoting the tweet that cannot be retweeted is quotable. The results are in: we can still quote the unquotable! Looks like the 'free speech absolutist' needs to go back to the drawing board. https://twitter.com/aiaaidan/status/1644493999061729280
I feel like quoting generates way more engagement than retweets, it's something active where you have some kind of "skin in the game", while retweets are passive. Often the worst tweets I see are through quote tweets, because they're made to enrage, to make you react.
If you're interested in exploring the beauty of symmetry further, and want to dive deeper into the world of math, I highly recommend checking out the book "LOVE and MATH: The Heart of Hidden Reality" by Edward Frenkel.
Here's a long quote from "LOVE and MATH" that gives a taste of the book:
"The story of Galois is one of the most romantic and fascinating stories about mathematicians ever told. A child prodigy, he made groundbreaking discoveries very young. And then he died in a duel at the age of twenty. There are different views on what was the reason for the duel, which happened on May 31, 1832: some say there was a woman involved, and some say it was because of his political activities. Certainly, Galois was uncompromising in expressing his political views, and he managed to upset many people during his short life.
It was literally on the eve of his death that, writing frantically in a candlelit room in the middle of the night, he completed his manuscript outlining his ideas about symmetries of numbers. It was in essence his love letter to humanity in which he shared with us the dazzling discoveries he had made. Indeed, the symmetry groups Galois discovered, which now carry his name, are the wonders of our world, like the Egyptian pyramids or the Hanging Gardens of Babylon. The difference is that we don’t have to travel to another continent or through time to find them. They are right at our fingertips, wherever we are. And it’s not just their beauty that is captivating; so is their high potency for real-world applications.
...
What Galois had done was bring the idea of symmetry, intuitively familiar to us in geometry, to the forefront of number theory. What’s more, he showed symmetry’s amazing power.
...
The general formula for the solutions of the quadratic equations was already known to the Persian mathematician Al-Khwarizmi in the ninth century (the word “algebra” originated from the word “al-jabr,” which appears in the title of his book). Formulas for solutions of the cubic and quartic (degree 4) equations were discovered in the first half of the sixteenth century. Naturally, the next target was a quintic equation (of degree 5). Prior to Galois, many mathematicians had been desperately trying to find a formula for its solutions for almost 300 years, to no avail. But Galois realized that they had been asking the wrong question. Instead, he said, we should focus on the group of symmetries of the number field obtained by adjoining the solutions of this equation to the rational numbers – this is what we now call the Galois group.
The question of describing the Galois group turns out to be much more tractable than the question of writing an explicit formula for the solutions. One can say something meaningful about this group even without knowing what the solutions are. And from this one can then infer important information about the solutions. In fact, Galois was able to show that a formula for solutions in terms of radicals (that is, square roots, cubic roots, and so on) exists if and only if the corresponding Galois group has a particularly simple structure: is what mathematicians now call a solvable group. For quadratic, cubic, and quartic equations, the Galois groups are always solvable. That’s why solutions of these equations may be written in terms of the radicals. But Galois showed that the group of symmetries of a typical quintic equation (or an equation of a higher degree) is not solvable. This immediately implies that there is no formula for solutions of these equations in terms of radicals.
...
Galois’ work is a great example of the power of a mathematical insight. Galois did not solve the problem of finding a formula for solutions of polynomial equations in the sense in which it was understood. He hacked the problem! He reformulated it, bent and warped it, looked at it in a totally different light. And his brilliant insight has forever changed the way people think about numbers and
equations.
There are too many countries in the world with oppressive regimes where people seem willing to accept the status quo without much resistance to be as optimistic as you about people's desire for freedom. Even in countries where extreme suffering and censorship exist, like N.Korea or, less known, Turkmenistan (which, by the way, is the craziest country of the former USSR - the cult of personality in Turkmenistan has been taken to the extent of idiocy, censorship works like in the worst times of Stalin, and people suffer from hunger, poverty, and constant fear) protests and uprisings are rare. I personally know people who demand even more control in some of these countries, and after hearing about their extreme complacency (think mobilization in russia), I'm not as hopeful that people truly want to be free anymore. IMHO the only form of freedom that is attainable is personal "nomadic" freedom, as in Sovereign Citizen book. We need to vote with our feet and wallets, and choose to live in places where our individual freedoms are respected and protected.
I'm sorry if this all sounds a bit bleak, it's personal...
Meanwhile, it takes the human brain months to adapt to visual input after blindness: "This research suggests that the brain is able to adapt to getting visual input, even after 10 years of blindness. That said, it takes a little while before this ability to really process the images kicks in. Right after the second surgery, patients are getting input to the brain from their retinas, but their visual memory is still poor. After about a month, though, the brain has found ways to process this visual information, and, a year later, their brains do just about as well as those of people who had sight from birth."
Source: https://psychologytoday.com/us/blog/ulterior-motives/202210/....
I liked the example in the article of teachers learning to read documents upside down (presumably because they do this a lot when students are in front of them).
> "After about a month, though, the brain has found ways to process this visual information, and, a year later, their brains do just about as well as those of people who had sight from birth."
Hey, that's still pretty impressive! The brain starting to receive a signal that it hasn't before and still being able to eventually adapt and figure it out is wonderful!
