Among other things, it suggests that there are some incentives for complexity: That engineers just enjoy complexity, and that making simple designs might not be good for the respect of your peers/bosses/career: "A simple design might make it seem like you’re not really doing your job."
While that's true, I think the essay isn't clear enough about the fact that simplicity is just plain hard. Complexity is sort of the "natural" outcome of "adding things". It takes time, effort, and skill to create software that can do what is asked for it while maintaining an internal simplicity. Even if everyone involved really wants it. I do not think software complexity comes only, or mainlly, from people not wanting it enough (explicitly or implicity), from incentives to complexity. It's just plain hard. It's a mistake to think just wanting it harder (or having the right incentives) is sufficient.
It may be the key challenge in software engineering -- how do we make software that does everything that's asked for it (powerful, complex things), while maintaining an internal simplicity? You can not accomplish it by cargo-culting any "design patterns" or other approaches. It's still a skilled craft. That comes from years of experience with the intent of developing this skill (years of experience alone aren't enough), looking at other people's code who are also trying to do it, and just some aptitude. And domain knowledge, if you understand the nature of the problems you are working on better, you can design simple solutions for them better--contrary to some current practices treating developers as interchangeable.
Yes, you need your organization to allow you to do it (including taking the more time it will take), but that's necessary but not sufficient for the outcome.
There is the well-worn Pascal quote about having no time to write a shorter letter[1]. There is also, I think, an important point hidden in the “domain knowledge” part: the freedom to change the spec; the effusive texts on Charles Moore’s approach to Forth programming[2,3], however much scepticism they deserve otherwise, are very direct about that freedom being essential.
But that means simplicity is a cross-cutting concern, like security, and does not particularly fit into any environment that would attempt to separate design and implementation. It needs “redraw all the module boundaries” to be a plausible (if improbable) response to “this one 300-line part of this one module is awkward and three times longer than it really needs to be”. Aside from the obvious administrative problems, suggests that the complete edifice needs to fit into the mind of a couple of people at most or the iteration simply will not converge most of the time.
[3] There was an example about changing a negative-feedback control loop, written into the spec by hardware engineers emulating analog circuitry, into a couple of fixed-point operations that behaved differently but did the actual job just as well; I can’t find the reference right now but this a minor domain-specific change among the kind I have in mind.
> Aside from the obvious administrative problems, suggests that the complete edifice needs to fit into the mind of a couple of people at most or the iteration simply will not converge most of the time.
> I don’t see a good solution to this.
I think this is probably true.
Some people take that to basically give up, and say, okay, our software is always going to be a mess, but look at all it's done as a mess, so be it.
Otherwise... the commonly understood solution seems to be composing the thing of independent units (very very very decoupled) each of which can fit into the mind of a couple of people at most. And then the whole, built of these independent units considered as black boxes, can perhaps also fit into the mind of a couple of people at most. I know you said "the complete edifice", which is not the same thing, but this is what we've got: and indeed is "abstraction", basically the entire basis of computer science and what makes possible anything we do. (Even the simplest program of 20 years ago -- can probably only fit into the mind of one or two people if you exclude the hardware all the way down, which you can because it is well abstracted and decoupled; it wasn't always, 50 years ago).
And we can come up with all the challenges and barriers of that -- of course! If it were easy, we'd have simple software. :)
But that's the task, I guess. But a lot of software isn't even built realizing that's the task -- to have one or two people "driving the bus" who have a mental model of the thing they're building, at any level of abstraction (I feel like Fred Brooks wrote about this), meaning that continuity of experience matters and domain knowledge matters, treating developers as commodities to be shifted around continually has a cost to complexity too, as does continually switching technologies and platforms so you can't build up a solid repertoire of encapsulated abstracted pieces to compose.
Among other things, it suggests that there are some incentives for complexity: That engineers just enjoy complexity, and that making simple designs might not be good for the respect of your peers/bosses/career: "A simple design might make it seem like you’re not really doing your job."
While that's true, I think the essay isn't clear enough about the fact that simplicity is just plain hard. Complexity is sort of the "natural" outcome of "adding things". It takes time, effort, and skill to create software that can do what is asked for it while maintaining an internal simplicity. Even if everyone involved really wants it. I do not think software complexity comes only, or mainlly, from people not wanting it enough (explicitly or implicity), from incentives to complexity. It's just plain hard. It's a mistake to think just wanting it harder (or having the right incentives) is sufficient.
It may be the key challenge in software engineering -- how do we make software that does everything that's asked for it (powerful, complex things), while maintaining an internal simplicity? You can not accomplish it by cargo-culting any "design patterns" or other approaches. It's still a skilled craft. That comes from years of experience with the intent of developing this skill (years of experience alone aren't enough), looking at other people's code who are also trying to do it, and just some aptitude. And domain knowledge, if you understand the nature of the problems you are working on better, you can design simple solutions for them better--contrary to some current practices treating developers as interchangeable.
Yes, you need your organization to allow you to do it (including taking the more time it will take), but that's necessary but not sufficient for the outcome.