I’m not sure his solutions are sound. I don’t think you should violate your own morals, but the real issue isn’t that your code can be used for wrong doing, it’s that there are no consequences when it is.
I think weapons manufacturing is a better example than software, because it’s much clearer. Weapons are necessary to defend free society, and when they are misused, we hold the misusers to justice.
Or at least we used to. Because today America is actively bombing civilians in 7 different countries, and no one is ever going to be held accountable for the outright human rights violation. I mean, it’s a war crime to kill civilians. I don’t think you can blame the weapons manufacturers for this though, and I don’t think we should have to. Because it’s our common ethics that should prevent it.
I guess, though, that in a world without accountability, your individual morals is all that’s left, but it’s our ethics that need to change if we actually want change. Because if you personally refuse to write the software, someone else will. Like I said, I think you should absolutely refuse to write the software because you won’t be able to live with yourself, just don’t expect the violators to stop unless we stop them.
Mistakes are going to happen and the world has changed. The battlefield is everywhere now. It's not good, but from a high level the US leading the world led to fewer combat deaths.
Very few countries have nukes and that is part of the US-leading thing. The MAD thing might work out for us, but it is too early to really tell. Might just be pushing the risk to the tails.
It is an improvement considering the last time we had multiple powers we saw minor countries get carved up by major countries and the whole thing eventually flare up in a war that spanned continents, between major countries, and killed millions.
Even the two power order of the Cold war just meant double the problems - the USSR had its fair share of adventures.
Too broad a statement. Civilians can be killed. They can knowingly be killed, even deliberately killed in some circumstances. Civilian deaths can happen when one attacks military targets. Bombing a military airfield can kill civilians. The fact that a civilian cleaner or cook will be killed does not make the attack a war crime. This is very important from a human rights perspective. If the killing of civilians in war was always criminal, armies will surround themselves with civilians. So, ironically, the possible killing of civilians is necessary to prevent them becoming human shields.
Remember too that the definition of "civilian" is complicated these days. Criminals are 'civilians' and a terrorist is a criminal. They may be an enemy combatant in bunker one day, and a civilian criminal defendant the next.
That argument just doesn’t make a lot of sense when you’re bombing a school in Kenya. I mean, why were you bombing anyone in Kenya to begin with? Has Kenya attacked the US?
I guess you could make a long line of explanations, but the war on terror is killing “combatants“ that weren’t even born in 2001. That’s just crazy.
Source? I did a good faith search to find it, and the closest I got was the 1998 bombings of the US embassies and an attack on a college by Al-Shabaab.
These "combatants" are often non-combative, the logic our government here in the US uses to classify combatant purposefully mis-classifies a huge chunk of the populace.
I agree with everything you've said, but the frequency and scale of the killing of civilians by US bombs and missiles is incredible - it's literally at the point where only some of these events (and I guarantee we only know of a fraction of them) get even cursory attention from popular media. And when they do, it seems nobody cares.
We're bombarded (pun intended) with constant scenes of war and fighting. We're continually warned, cautioned and kept frightened about the bugbear de jour (North Korea, Russia, China, Daesh, terrorists, whoever). All this is used to justify the terrible actions and atrocities, of our military forces, used to justify ever creeping mass surveillance of the general population, because hey, we're the good guys! We need to do this to keep us safe! The enemy is worse! Etc
It hit saturation point a long time ago, and we know we can do nothing but continue sleepwalking into the dystopian nightmare that awaits us.
I don't think this is anything new. The US was casually bombing people during the Vietnam era with little or no attention from the media. What got attention was the campaign as a whole, mostly after the fact. Individual actions rarely got a mention. I would also be careful about conflating foreign military adventures with domestic spying. While there is some crossover, balling everything into one big problem makes change impossible.
A standard defense by those who support a status quo is to link everything, to state that any change in one area must involve other changes to any number of other programs. It is a delay tactic. Debate spirals upwards until people are talking about bringing down the entire military-industrial complex, not the specific of how drones are being used in a particular conflict. If you see only an edifice it is because they want you to see an edifice. It keeps them safe from specific questions.
In your weapons example the problem is nobody will admit it’s a war crime. People will see it t their own way and to this point there’s no official legal consensus about what to call it. So this is left entirely to the morals of the individuals. This isn’t always the case.
There are plenty of situations where legal barriers are put in place to prevent supplying abusive regimes (for example) with certain types of equipment, software, weapons, etc.
The fact that no one will admit that it’s a war crime is exactly the point I’m making. How do you stop that with individual morals? I probably wouldn’t work in weapons manufacturing, just like I wouldn’t write spysoftware for the NSA, but a lot of people would.
If our ethics as a society allow it, then our individual morals don’t really matter.
Even in weapons manufacturing there are “no go” weapons like around biological warfare where even the simple production is seen as bad.
I am starting to feel like this makes sense in other domains (namely, facial recognition has almost only applications with bad side effects in practice)
I don't understand this guy at all. Two things confuse me:
First, the purpose of a military is to be able to break things and harm people as effectively as possible. Did he think his job would involve making the military less effective?
Second, why is it bad to work on software that's used to kill people? Killing isn't necessarily bad. If this software helped kill Abu Bakr al-Baghdadi or Ayman al-Zawahiri, it would have a hugely positive impact upon the world.
To me, the post comes across as a bunch of meandering moral grandstanding. The core thesis is either utterly pedestrian ("Think about the consequences of what you’re building.") or totally fringe (that helping the US military is immoral).
> First, the purpose of a military is to be able to break things and harm people as effectively as possible. Did he think his job would involve making the military less effective?
That's his point. He was distracted by the pay, the work, etc. and put thinking about what his software would do to the side. Of course it's obvious, but it's easy to "get distracted" and not think about it.
> Second, why is it bad to work on software that's used to kill people? Killing isn't necessarily bad. If this software helped kill Abu Bakr al-Baghdadi or Ayman al-Zawahiri, it would have a hugely positive impact upon the world.
This is inaccurate for two reasons:
1. It assumes moral absolutism: that certain killing is bad and certain is good, but good or bad for whom? The CIA tried to kill Castro multiple times, but he's a hero to the people of Cuba. I'm sure people in the USA would've applauded this, but Latin America would've mourned and been furious.
2. Is the US government the best judge to decide who lives and who dies? Moreover, is the single individual in the military who will ultimately call the order to kill the person who should decide who is good to die and who isn't? And on that note, where does that ethically place the enabling of certain people to wield such power?
> It assumes moral absolutism: that certain killing is bad and certain is good, but good or bad for whom? The CIA tried to kill Castro multiple times, but he's a hero to the people of Cuba. I'm sure people in the USA would've applauded this, but Latin America would've mourned and been furious.
Considering the horrific things Castro did to the Cuban people[1] and how he almost started nuclear armageddon[2], the world would have been better off if he had been overthrown (and possibly killed) as early as possible.
> Is the US government the best judge to decide who lives and who dies?
No, and nobody in this thread is arguing that. The US military is an extremely flawed organization, and any military is a blunt tool. Even when the US goes into a country with the best intentions (a rare occurrence), lots of innocent people will be killed. It's good that we continually raise the standards we hold our military to. It wasn't that long ago that people were fine with carpet bombing population centers. Now we recognize that such acts are abhorrent. The same is not true for other governments and leaders. So to answer your question: I don't think the US government is the best judge, but it's a lot better than the alternatives (such as the PRC or Russia). It's also a better judge than the people it tries to kill, such as Osama bin Laden, Saddam Hussein, Abu Bakr al-Baghdadi, and Ayman al-Zawahiri.
> Considering the horrific things Castro did to the Cuban people[1] and how he almost started nuclear armageddon[2], the world would have been better off if he had been overthrown (and possibly killed) as early as possible.
Considering the horrific things US presidents have done to the American people [1] and how they invented nuclear armageddon[2], the world would have been better off if they had been overthrown (and possibly killed) as early as possible.
(In fact, if we're going into the territory of human rights-- the US has not a page, but a whole category in Wikipedia [1] on violations).
> I don't think the US government is the best judge, but it's a lot better than the alternatives (such as the PRC or Russia). It's also a better judge than the people it tries to kill, such as Osama bin Laden, Saddam Hussein, Abu Bakr al-Baghdadi, and Ayman al-Zawahiri.
From your point of you, an American, sure. From the point of view of Chileans, I doubt the murder of democratically elected Salvador Allende aided by the CIA was good judgment, or that American judgement was better than his. From the point of view of Guatemalans I doubt the US's involvement in the coup against democratically elected Jacobo Árbenz was good judgement. The list goes on. As someone who is not American I can assure you the US government is not a better judge than the PRC or Russia, is equally as bad and in some cases, from my point of view, worse.
I frequently see this argument used to justify everything from ever creeping surveillance of civilians, to sanctions that primarily affect civilians, to the bombing, killing and maiming of civilians.
Surely we should aim for better than "yeah, but they'd be even more terrible than us"?
> I don't think the US government is the best judge, but it's a lot better than the alternatives (such as the PRC or Russia). It's also a better judge than the people it tries to kill, such as Osama bin Laden, Saddam Hussein, Abu Bakr al-Baghdadi, and Ayman al-Zawahiri.
And there it is: the blind ideology that drives this madness. Any rational analysis of the last 50 years would show the US is far, far more dangerous than China, Russia and terrorism combined. The US has slaughtered millions of innocent people across the globe in the last twenty years. China has lifted hundreds of millions out of poverty in that time.
The real lesson here of course is that true evil is absolutely convinced of its own righteousness and totally ignorant of real world facts. That evil is something many are all too happy to join.
You literally have no idea what the people of Cuba feel about Castro. Neither do I. There is no freedom of the press in Cuba. And the reason for that is that anybody who says he was anything less than a saint gets thrown into a dungeon-like jail cell at best, or executed at worst.
> You literally have no idea what the people of Cuba feel about Castro. Neither do I.
I don't. I personally haven't been to Cuba, but I'm not American so I know people who have (unlike the US, we could always travel to Cuba) and in fact have worked with people from University of Havana.
From what I hear, it's not a paradise, but its people are also not "tormented by the constant fear of a tyrannical regime" like some people seem to claim. Much like most other nations in the world, I'd say.
Right, but if you are an absolute pacifist, you should never join a military contractor. No matter what you work on, you are helping an organization whose mission involves killing people. Even if you were just making them lunch, you would be indirectly helping them do THEIR job of making killing machines better.
> What about killing to protect others, as part of your duty?
How do you know if it's really going to protect others? How do you know you can trust the motives of those that say "it's all to keep us safe?".
Did the wars in Iraq and Afghanistan (which let's not forget, also gave rise to IS) really save more people than if we hadn't gone to war? Given the numbers killed, I very much doubt it.
Overthrowing democratically elected governments, murdering leaders, drone strikes on hospitals, Abu Graib, Guantanamo Bay... are these really the actions of a benevolent leadership?
> "We sleep soundly in our beds because rough men stand ready in the night to visit violence on those who would do us harm"
Churchill, I think? Those were rather different times, and Hitler's armies were a very different threat to those we're told we've faced since and now. These days, our leaders speak much the same words, but I like it would be naive to believe them.
According to Quote Investigator[0] it's actually Richard Grenier of the Washington Times (not to be confused with the Washington Post) paraphrasing something he believed Orwell might have said.
The basic sentiment has been around since at least the 1890s, and Orwell did during and after WWII criticize pacifism by essentially saying that the only reason you even can be a pacifist is because people that aren't pacifist are willing to protect you. But that particular phrasing is modern.
The US military is, by and large, concerned with killing people who are demanding policy changes under threat of violence. To commit to nonviolence requires making those changes, and any others that someone with a gun demands. Is that better?
Ok now I'm really confused. If killing is necessarily bad, then how can it be justified in the case of self defense?
And it's a distraction to point out that capital punishment is banned in some countries. Those countries still have cops that shoot criminals and they have military forces that would kill other soldiers in the event of war. Capital punishment is about depriving someone of life after they've been captured. Capturing is rarely an option in a warzone. Given that constraint, it's better that the military kill someone like Abu Bakr al-Baghdadi rather than let them go free.
By that logic, America was not justified in attacking Germany during WWII.
Countries have several good reasons to invade other countries. They can come to the defense of their allies. Or they can invade to stop genocide, such as in the Kosovo War. There's also the idea of a preemptive war, though in practice that is usually a pretext.
Ugh, please don't use Kosovo as an example of a "good" kind of intervention, without looking more deeply into historical context and consequences afterwards...
Well, not everywhere, and not even for that reason.
The morality of killing a prisoner who has done horrific things is not the only reason to end capital punishment. I have no moral problem with someone putting a bullet through the head of some evil thing that murdered one or more of our fellow citizens, but an honest distrust of the system we use to convict said person keeps me from supporting capital punishment.
You cannot undo capital punishment. You at least have some hope of going on if you wrongly convict someone and let them out of jail.
I think you are being too harsh on the guy. He says at the outset that this is his first "real" job and it is clearly the first time he has worked on something where he was able to see a larger picture than the one directly in front of him.
When you are doing a problem in differential equations where you are calculating the inflow of a contaminant and water and trying to find the point where the equations solve for a steady state level of contamination. Nobody tells you this is what you do if you want to poison a lake and kill the fish with out killing off possibly other marine life. Its just a numbers problem.
People reach an adult understanding of the world at different times. In my experience and in stories others have told me, it all rings very similarly. Suddenly one's actions in the "real" world have real implications and effects and that opens up an entirely different way of looking at things. It is simultaneously exhilarating and terrifying. I think of this as one step past the one where you realize while driving that you could just simply drive into the oncoming traffic and create a huge pileup (and no doubt die in the process, no I'm not suicidal, it is the realization that it is only your will to not do that, that prevents it from happening.) This next step is "what I'm doing in my day to day life can be used to, or enables, this wide range of things, not all of which are good or moral. One of my cohorts at Google (the group of people that all start the same week have a built in social group) came to me in a panic when they suddenly understood the implications of what they were building. (It was an internal tool). And they didn't like it at all, and they were conflicted because they loved the project and the team, everyone was super smart and great fun to work with, but the tool ... that was not something they felt they could be associated with. I shared with them choices like that which came up in my career and how I chose to deal with them. (in general I've been able to tolerate usual and cursory sleaziness but have a line where I won't cross. At that point I choose to work elsewhere). This person transferred to a different group and was much happier, and it must have worked as they are still at Google as far as I know.
So when I read the article I heard someone who was waking up to the realization that "oh crap, the world isn't run by other people, its run by me and my friends here." and being overwhelmed at the sudden responsibility of it all. Give the guy credit for opening his eyes, and offer him a beer as he joins the rest of us trying to navigate life choices knowing that nearly every one we make impacts others in both positive and negative ways.
I like how charitable you are, and that is not meant to be sarcastic. However, he knew precisely who was signing his paychecks. No one pretended he was working in inventory management for the toy section of Walmart.
I once talked to a retired general who had been in Afganistan. In his view the purpose of a military is to reach a truce. Once you have a truce, politicians have the job to reach peace. Breaking things and harming people might sometimes be the most efficient way to reach a truce, but if it is not then a good military will avoid it.
This view might be a german peculiarity though. After WW2 they had to distinguish the new "Bundeswehr" from the previous Nazi "Reichswehr". They developed the concept of "Staatsbürger in Uniform" [0] (citizen in uniform).
Some countries overuse conscripts as a measure of population planning in additional to its intended purpose. This is an example of deliberate inefficiency in a military.
I think a lot of people are missing the point of this post, especially because it offers an opportunity to nit pick on specifics. But the point here isn't "don't write code for the military" or "don't write code for killing" or anything else, it's that we need to think about if we're okay with the code we're writing. We need to think about the implications of the things we build.
We're never going to find a hard line we all can agree on, aka "don't work on missiles, but airplanes are okay" or something similar. But we can all stop to think about what that line is for us personally, and if what we're working on crosses it. What would we do if asked to violate that boundary we've created for ourselves?
When I was in grad school, we had to take an ethics in engineering course. We split into groups and discussed the ethics of building certain things, in my group's case building predator drones. While I didn't agree with the person who said they were okay building them because we need weapons, I was far less concerned with that individual than the one who looked at me, straight faced, and said "I don't get what airplanes have to do with morals".
> it's that we need to think about if we're okay with the code we're writing
I don't disagree, but feel that it's rather difficult to actually discern this.
For me, the stand-out problem that most coders face is that of compartmentalization. That is, you are a small part of a bigger churning wheel. You are told to do something, so you do it.
Unless you are working completely for yourself, or a small start up with specific goals in mind, you will not have the oversight necessary to actually objectively determine whether your code is going to be used for the greater good or not. The majority of bigger commercial ventures (whether it be defense related or fb) are heavily comparmentalized. In my experience, the more deliberate and the more legalistic the compartmentalization is, the more intrinsically morally corrupt (or even illegal) it possibly will end up being.
For instance, look at Google's eroding "don't be evil" construct [0], or FB's experiments on mass psychology [1]
This is why I left the corporate world. Too often, my moral compass was being desensitized by sociopaths.
I was far less concerned with that individual than the one who looked at me, straight faced, and said "I don't get what airplanes have to do with morals".
They're arguing that morality attaches to the user, not the tool.
See any gun-control argument in history, basically.
Let's accept for the moment that tools are morally neutral. Notwithstanding that, when you're thinking in a consequentialist fashion, you don't just ignore the knowledge you have of how the tools you build are going to be used. You can say "well it's not up to me, it's not my responsibility to think about how they'll be used", but you have to be upfront with yourself about the fact that you're not taking moral responsibility for your actions.
but you have to be upfront with yourself about the fact that you're not taking moral responsibility for your actions.
My concern isn't so much the nature or purpose of the tool I'm building, as it is with whose hands the tool will ultimately end up in.
I'd be OK working on military drones, for instance, if I knew they wouldn't end up in the hands of avaricious neoconservatives, LBJ-school paleoliberals, religious nuts, or basically anyone who isn't someone whose judgment I personally trust.
So in practice I wouldn't want to work on a military weapons project with no valid civilian uses. But it's not because I believe drones or land mines or atomic bombs are inherently evil. It's because I have no idea who will eventually end up in control of those products, and because history suggests that they will be used in ways I don't approve of.
I have the same attitude towards patents, basically, but that's a different debate.
I think a lot of people here are getting hung up on some imagined implication that any code that could ever be used for evil is wrong to make. Yes, this is a grey area and there's no clear line of what the correct ethical choice is in many situations. But the point is that it's barely discussed professionally at all. Other disciplines have quite a bit of exploration into their professional ethics, but software engineering seems to gloss over it. Yes, there is some precedent, but it seems disproportionately small when considering that it is becoming an increasingly ubiquitous role in our society.
It seems like it's not even in our vocabulary as engineers to comprehend the ethics of our work. There's no framework for analysis or disclosure. Sure, there aren't any easy answers to most situations, but have we even tried?
Killing people is at the extreme end of ethical concerns, but a lot more people are working on code that's designed to tighten corporations' control over users or otherwise take away their freedom and privacy and increase authoritarianism. Things like DRM, "security" ("because who doesn't want to be safe and secure?"), removing functionality that allows extensibility/interoperability, etc. I've heard it phrased thus: "Do we want to help them build better nooses to put around our necks?"
I would rather write code to kill people, than to control what they watch or consume, and how they consume content. Killing a few people is small in the grand scheme of things. Restricting knowledge or access to knowledge can destroy entire civilizations. Restricting content produces a world I would not want to live in.
A _lot_ of folks here in the comments are missing the point and "getting distracted" by the word "kill."
Sure, it was probably a bit short-sighted of the speaker to not connect the dots and see that this software would be actively used to seek out and kill people. But to me the core message of this talk applies to the very large grey area that lies in between fully ethical software and stuff the DoD makes to kill people (yes killing _can_ be necessary I know, I know, don't get distracted!). And that grey area is mostly social media and ad-tech.
These are two domains which require software to be written that is actively harmful to people's privacy and mental health. We see Twitter being used to target and harass people to the point of suicide. Instagram has been precisely designed to the point of addiciting its users into a fake world that makes them feel like they are nothing and that everyone is happier than they are and it depresses them.
The examples I just cited are prevalent criticism and can start to echo in our chamber here (HN). But how about a real, recent example: Facebook's use of two-factor authentication phone numbers in advertisements. Some engineer at Facebook was given these requirements to implement this super shady and deceitful functionality and chose to implement it anyways without pushing back. It takes advantage of folks who were simply trying to improve the security of their account, but now it is being used to target them with advertisements.
Most of us will have long careers that don't involve writing software that will kill people, but a stunning majority of us will be somewhere in this grey area at one point or another, and you must think about what you make at that point still.
Yes, it's important to understand what you're doing.
However: a country without a military (or an allied country with a military) is very quickly not a country. If the US and Europe stopped having a (working) military, they would be immediately taken over by totalitarian regimes who would be delighted to trample over all the rights and privileges their citizens currently enjoy. South Korea, Taiwan, and many other countries/regions would instantly be destroyed by powerful and dangerous neighbors. Free countries are not free because everyone around the world is nice; they are free because people are willing to die to protect them.
A military must have weapons that can kill people. The real goal of such weapons in a western democracy is not to kill people - it's to be able to kill people so that no one will take over or threaten the country and its allies (at least not without consequences). It is entirely ethical to enable self-defense, and self-defense is the purpose of the Department of Defense (remember, its very name is "Defense"). The ACM code of ethics doesn't forbid this, because it focuses on unintentional harm, not intentional harm from a lawful order to protect a country. The author seems to think it's unethical to enable self-defense, and that's just nonsense. Weapons (and anything else) can be misused, but we need to hold the misusers responsible - not pretend that they aren't needed. It's a good thing that military personnel are willing to risk their lives to protect others, even those who don't appreciate their sacrifices to do so.
A counterexample: Costa Rica (where I'm from). We haven't had an army for the last 70 years and still remain a country. Superpowers (e.g., US, China, Russia) might need armies as means of mutual deterrence. Smaller countries not so much.
I did specifically note that it's okay if your country doesn't have a military, as long as your country has an ally with a military. I guess I should add if there's absolutely no threat then you also don't need a military - but that's a rare situation.
"Countries Without Militaries" by Kathy Gilsinan (Nov 11, 2014) at https://www.theatlantic.com/international/archive/2014/11/co... specifically discusses the case of Costa Rica: "After 7,000 U.S. servicemembers deployed to the country in 2010 to assist in counter-narcotics efforts, Bolivian President Evo Morales said Costa Rica did have an army: the United States military." Indeed, that articles notes a lot of the complexities when you dive in. The vast majority of countries have obvious militaries, and of the rest, "Virtually all the countries the CIA lists as having no formal militaries have some form of security service, and the degree to which they focus exclusively on internal threats varies". If there's absolutely no threat then you don't need a military, but for the vast majority of countries, that is not reality.
> So, Who Protects Costa Rica?
Costa Rica maintains its military-free status and does not command any military units or house any war weapons. However, the country does maintain alliances with other countries, such as the United States, that can be expected assist in the event of war within Costa Rica.
https://qcostarica.com/costa-ricas-military-abolition-histor...
Hm... not exactly sure how your quote stating that Costa Rica maintains a military-free status falsifies the grandparent's comment stating exactly the same.
It doesn't directly falsify the fact that Costa Rica lacks a military, but shows that their solution does not scale in a useful manner.
Edit: It directly contradicts the usage of the claim that small countries do not need a military. Essentially borrowing deterrence from another doesn't result in the world being military free.
it doesn't directly rebut the post, but the poster you are responding to is emphasizing that costa rica is only able to maintain their military-free status by allying with one of the most powerful militaries in the world.
it's probably a good thing that not every country in the world needs to maintain a full-scale military, but this only works if you have a powerful ally whom you can trust to defend you. the fact that costa rica is able to exist without a military doesn't imply that the US could also disband their military.
And the next time there is a large scale conflict, some neighbour country may feel tempted to use that opportunity to send a single division to your country and absorb it.
Then you can say how wonderful it is to save a fraction of a percentage point of your GDP on defence and how it worked so well until the day it was needed.
That argument could be made about pretty much any small country using a superpower as the "neighbor". Take for example Russia annexing Crimea.
At the end of the day, it isn't the presence of a military that makes the difference, but the strength and spread of the country's treaties with other nations.
But at the end of the day, somebody still needs to spend money on a military. Being put under the umbrella of the US/Russia/China/etc just alters where it is based out of, and makes you vulnerable to the demands of those that provide it.
In real life the DoD does end up killing lots of people that it probably doesn't need to be killing. I would generally agree that countries need weapons or to at least be friendly with countries that have weapons, but you have to look at how those weapons are used.
Preemptive self-defense absolutely is a legitimate thing, if used properly. If a great deal of violence against civilians can be stopped by a smaller amount of violence against combatants, the second part is justified, even if it takes place "preemptively, on foreign soil on the other side of the world." A significant amount of context based on real world cases is necessary however.
When such capabilities have been misused, we look at why they have been misused and fix those. The Department of Defense has a legitimate usage for defense of the United States. Misuses of that should be handled through the appropriate political channels, rather than destroying something with a legitimate purpose.
> Does it find phones ... This was never about finding better WiFi. We were always finding phones. Phones carried by people. Remember I said I was working for a Department of Defense contractor? The DoD is the military. I was building a tool for the military to find people based on where their phones where, and shoot them.
I got distracted by this utter failure to define the objectives and requirements of the project. If you want to find phones, don't start by using phones to find wifi routers.
But yeah, if you work for the DoD, you should probably be ok with the stuff you are working on being used to kill people. It's a big part of what they do.
And it's not clear that it was a requirement to implement this on an off-the-shelf phone. There are plenty of developer friendly chipsets out there that are easy to sniff with that could've been run off a handheld device.
Where do you draw the line? Almost any significant technology can be applied for military purposes. Likewise, the Internet itself came from a DOD research project; military technologies can be repurposed for peaceful uses.
Yes, this is what I don't understand. I'd wager the most dangerous weapon used by the US military is either Microsoft Word or PowerPoint to produce operations orders.
It seems that if a product's utility isn't directly tied to a military outcome, then the outrage never materializes.
Project Maven caused outrage because of the direct link to military drones. However, advancements in AI and machine learning haven't caused the same outrage but can be used for the same purpose.
I feel like this underestimates the complexity of the problem.
Both markets and open-source software run on abstraction. (The official open source definition doesn't even allow restrictions based on field of use.)
If people want to implement "know your customer" like the banks do, and make decisions based on their own political values, your customers need to share a lot more information and there isn't going to be very much privacy. Buying services is going to require a lot of hoop-jumping.
And then consider the effect on product design. Unless countermeasures are built in, a copier can be used to counterfeit money. And that's an easy case.
Now we have a simple service for distributing text messages making front page headlines for its effect on society.
It was naive, but the assumption that customers are responsible for their own actions was a useful fiction. A society of mutual distrust makes everything difficult.
Maybe software that wouldn't be used to bomb civilians, without visual confirmation, in countries that the US is not in war with?
I mean I would be okay with producing firearms to protect my county. I wouldn't be okay with the CIA sending those guns to kill squads in South America.
A lot of posters here are treating all warring and killing as equal. But there is a huge span from proportional and moral use of deadly force, to doing what would be treated as a war crime if it wasn't carried out by the world's foremost superpower.
Great article. I believe it's important to understand purpose of everything you do.
This story crystalizes that into a very compelling advisory.
Often we do things without that understanding in mind and that can lead to many problems, including miscommunication, errors and possibly what this article alludes to.
I only skimmed it but this doesn't really make sense.
a) Obviously if you work for the military, killing people is going to be involved somewhere. That's what the military does!
b) Why would put so much effort into locating wifi routers when they wanted to find phones? They have no need to hide that objective - it's a perfectly obvious thing for the military to want to do.
c) I didn't get that far but is he assuming that "find a phone" = "kill the owner"?
The military is an absolutely essential institution, but for the sake of the civilians and the soldiers, we shouldn't forget its function: Kill people and destroy their creations. I saw a recruiting commercial for the U.S. military showing an aircraft carrier and calling it a 'global force for good'. That misleads recruits: It's a global force of death and destruction, and that will be your job if you sign up. We don't like to think that, and we can't let that result in rejecting all use of the military - which is just as irresponsible - but we must face the reality of a very serious subject so we can think seriously about it.
When we imagine that the military is something else, especially something glorious, not only do we risk the worst evil of humanity, war, but we also harm the soldiers (and sailors): War is very damaging to them, and not just the dead and the physically wounded, but causing and experiencing death and destruction results in great psychological harm. Humans are not cut out to do it: IIRC the details, on D-Day in WWII, half the soldiers didn't fire their weapons when they should have. After every war, you can read that people 'were not the same' when they returned; many are damaged. Suicides are (or recently were) very high among U.S. soldiers, and the current wars are relatively very low risk for them. I've read interviews with elite special forces soldiers who talk about how hard it is, psychologically, to kill.
Another consequence is that we put soldiers in positions to fail: We send them to wars that they cannot win, usually because we ignore the essential requirement of a stable political outcome - Afghanistan and Iraq are only the two most recent examples. Many in the U.S. like to imagine an invincible force, a panacea for international problems, but just a brief glance at history shows otherwise: since WWII, there has been one clear victory (Gulf War), two endless stalemates (Korea, Afghanistan), one ongoing quagmire of mostly negative results (Iraq), and one loss (Vietnam). We also ask soldiers to do jobs they are not trained for, such as policing: Police are there to bring and maintain peace and public order; soldiers are trained to do the opposite, kill and destroy.
When we have a clear idea of a military's function, we can align outcomes with our values, minimize the use of soldiers precious lives and health, and put them in a position to succeed. To ignore the reality of the military's function, of killing people and destroying their civilization, is immoral IMO.
> of killing people and destroying their civilization
You should exchange these two parts, making it "destroying civilization and killing their peoples". It is ambiguous currently and suggests a dangerous act of directing a military against its own people, which happened in history and brought bad results every time.
LOL @ "North Virginia" . Also, when he says R^2, is he talking about Pearson's correlation coefficient? That paragraph is confusing, and in the next one he admits being mystified by the idea of gaussian distributions... I guess I should stop trying to make sense of the technical elements of this article.
I don't see this as a "don't help the military" post. This is a post about needing a profession of software engineering.
There are, clearly, ethical lines. Leaving aside where the line and the military cross, it is important to think how we will build such a profession - and enforce membership (which is the whole point)
Also worth noting is a common medical ethics quiz: You are a ER doctor, and a college football player comes in, RTA, spine shattered, internal bleeding, conscious and not in pain but needs operation to stem the bleeding.
He clearly and openly states that as he will never walk again, he does not wish to live.
Do you operate?
Most doctors it seems operate, and oddly that is a violation of most ethical board recommendations.
if i understand correctly, it would probably be illegal (at least in the US) for a doctor not to operate in that situation. the only circumstance i know of where a doctor can willfully allow a patient to die is when a notarized do not resuscitate order has been filed. even in this situation, they would still probably be obligated to try any other means at their disposal to save the patient.
I don’t think the point is that there is an obvious right answer, but rather that the existence of formalized (ish) ethics and codes of professional conduct frequently distinguishes the medical profession from software engineering.
sure, but it seemed worth pointing out that the ethical behavior prescribed by the board is probably illegal. i don't want a formalized code of ethics in my field that might require me to break the law.
This is a wonderful article and interestingly one of the more poignant bits is a quote from someone in the film industry (to paraphrase):
"We meet with a lot of startups and the only question is 'Can this be done?' Nobody is stopping to ask questions regarding the ethics."
That being said however, it is really tricky to come to any easy conclusions. We live in a complex world and it's not clear even after exhaustive questioning what damage could be done with the work that one is doing.
Fire was one of man's greatest discoveries - but if you stop and look at all the dangerous uses it could have been put to, it is quite possible we would not be sitting here today on HN...
The article is interesting, but not a new problem. Any technology can be used for good or bad (within a given morale framework). Yes, we should all keep it in mind, and ensure governments representing us regulate technologies, or do not use them for unethical purpose. The article doesn't even address this.
Unfortunately, the author writing 6 or 7 long paragraphs to build tension, repeating the same sentence for drama effect, to finally get to the the author's pretend shocking discovery that the DOD and its contractors are producing lethal technologies, ruined it for me.
I'd have worked on greyball if it was pitched as a tool for disrupting justice, but would feel bad if it was for people who threw up in the car too many times.
I see these kinds of articles constantly. Are there really that many people out there who aren't aware of their own responsibility for their actions?
He seems to be saying don't build something if it's possible that some people could do bad stuff with it. That would mean we shouldn't build end to end encryption, because terrorists could use it to hide. And that we shouldn't build Tor, because child pornographers could use it.
The job said “Department of Defense” right on the tin. He was not misled in the slightest. He said it paid half again as much as other internships, but I see no mention that he returned any of his bloodsoaked gains.
Its both. Absolving yourself of hunting jews for nazis because you didnt personally pull the trigger is morally and ethically bankrupt. Your line of thinking is exactly the problem.
how many levels of indirection does it take before you are no longer "morally and ethically bankrupt"?
is the janitor at CIA partially responsible for extrajudicial killings? is it morally bankrupt to pay federal income tax, a portion of which will finance the killing machines?
If you look back far enough, nearly every on-land government still extant today has a history of violence; be it that they conquered the land they inhabit, that they invaded their neighbors so that they might not be themselves invaded, that they were established due to their predecessors being violently overthrown, and so on.
> is it morally bankrupt to pay federal income tax, a portion of which will finance the killing machines?
To be fair, it could easily be said that we all engage in the morally bankrupt act of voting for politicians whose foreign policy ideals aren't as well-thought-out as Ron Paul's.
I said "and their chain of command". Yes, I get that in addition to the trigger-puller following an illegal order, someone would have to have provided that order - and at worst, the buck stops with the Joint Chiefs and the President.
Unless this blog-post's author is a serious multitasker, I highly doubt that he/she was anywhere in that chain-of-command.
Or do you mean to insinuate that - for instance - if the Navy had been using OpenBSD to run their carriers' computers, then that makes Theo de Raadt responsible for everything the Navy does that you think is bad?
(And of course, there's also the facts that our military aren't the Nazi-led Wehrmacht and the terrorists aren't Jews ... but by Godwin, that's the darnedest angle to take here.)
This was my same experience in the DC area, anyone with a Computer Science degree is getting scooped up by the intelligence community, and you will get interviewed by spooks because your friends are interviewing with the NSA.
I did some contracting for a Department of Defense subcontractor too, and then got the hell out of that town.
Those people are twisted. Their ideology is twisted. And your parents are just excited that you got an interview at any job.
I never really get why military has such a bad press in the programmer's crowd.
I mean sure, killing people is bad in a society but it's precisely because the world has been so far a succession of ruthless wars between groups of people that having a military is a good thing to protect you.
I know the "military kills unethically etc.." and "we are a peaceful world etc.." arguments but maybe people are getting a bit too, and wrongly, "certain" that it will last. If anything, history has shown that it's a succession of cycles until the next wars. Better be able to be the strongest ones.
I've been in my country's military (Europe) and most of the people are not psychopath (a few, sure). they are normal people thinking their job is an important one to protect peace at home.
So yeah "just don't go to job that makes you uncomfortable". It's the only takeaway i get from that article but i don't share the military shaming. Would have been more convincing with a oil industry or "on-demand market" example
>I've been in my country's military (Europe) and most of the people are not psychopath (a few, sure). they are normal people thinking their job is an important one to protect peace at home.
If you live in Switzerland, that's a reasonable expectation for those people to have. If you live in the UK, it's naivety to the point of intentional ignorance. If you join a military that has a habit of starting foreign wars, you can't hide behind the make-believe motive protecting your country from invading forces. You know what you're getting into.
No. Military is designed to kill some people for the sake of some others' lives. Enemy detection is vital to a military's function. Otherwise it will obliterate itself.
This is pretty dumb. Maybe I have a good imagination for malice but if I thought through every possible bad use of some code I could never write any code or produce nearly any item in the world. I suppose this is parallel to believing if guns themselves are bad or not.
I think weapons manufacturing is a better example than software, because it’s much clearer. Weapons are necessary to defend free society, and when they are misused, we hold the misusers to justice.
Or at least we used to. Because today America is actively bombing civilians in 7 different countries, and no one is ever going to be held accountable for the outright human rights violation. I mean, it’s a war crime to kill civilians. I don’t think you can blame the weapons manufacturers for this though, and I don’t think we should have to. Because it’s our common ethics that should prevent it.
I guess, though, that in a world without accountability, your individual morals is all that’s left, but it’s our ethics that need to change if we actually want change. Because if you personally refuse to write the software, someone else will. Like I said, I think you should absolutely refuse to write the software because you won’t be able to live with yourself, just don’t expect the violators to stop unless we stop them.