to a naive kid, this sounds like a great idea - humans are pretty much vision-only, so cars should be able to do the same! makes me feel like there are a lot of naive kids over there. I know they have some geniuses, I'm just saying how the decisions look from the outside.
> I bet if we had lidar/radar vision we could do a bit better too
IMO we do - try driving with the windows on both sides down a crack. Your brain synthesizes traffic noise into your mental model. You can be aware of traffic near you without being able to see it.
Radar isn't hearing. It has different properties than your eyes and your ears.
You know how your eyes can sometimes get things wrong? For example, you can see a mirage of an oasis when you are in the desert? Well, radar has failures like that. For example, in normal everyday driving conditions, like when you go over a pothole, someone passes through the lane in front of you, or you go under a bridge radar tends to hallucinate just like your eyes tend to hallucinate. It tells you things that aren't true about the observed reality.
The most famous story about radar that I know is the time that ignoring it prevented the death of humanity. There was an illusion of a nuclear launch by the United States on Russia. The officer didn't believe the radar and so didn't launch nuclear weapons in retaliation.
If you were combining radar with your eyes while driving, rather than combining it with hearing, you would have times like that - times where you recognized that the sensor was wrong and you were forced to ignore it. Or you would die. And others might very well die with you.
Humans crash cars because humans make stupid human mistakes.
A computer doesn't "forget" to look before merging. A computer doesn't drive drunk, fall asleep, get road rage, or get distracted by a phone. A computer can see in all directions at once with enough sensors.
I 100% believe that a vision-only self-driving system CAN work...but that radar/lidar providing extra signal would make it a lot easier to implement, especially at night when oncoming headlights can be blinding.
This would be true if we could rely on radar being an idealized sensor in which its error follows a normal distribution and in which visions error also follows a normal distribution. Under that world we could just use sensor fusion techniques to combine the two readings. We don't have that sort of normally distributed error though. In practice there are scenarios where radar gets things very wrong, like with bridges. This is is a systematic error, not a normally distributed error.
If you use the average rate of error and you combine it with the vision reading the sensor fusion is much worse, not much better, and in practice it resulted historically in break checks at underpasses, followed by drivers taking over to prevent themselves from getting rear ended.
In theory you could absolutely do sensor fusion, but it isn't trivial. You need to have a network that can (1) understand the scene and (2) understand the error distributions as they relate to each particular scene. But notice this - vision is usually more reliable than radar, by a hell of a lot, so how exactly are you going to be determining the edge cases where you need to assume a different error distribution for radar? Seeing how tricky this gets? Combining with radar relies on vision being reliable enough to help inform you of how radar is going to fail. Ugly circular dependence right there.
Lets say you can compensate for the potential for radar to error dramatically. In theory, this would allow you to have a better reading in some edge cases, but in practice those edge cases don't matter. What are the actual edge cases where it helps?
1. Driving in the dark or while blinded: radar isn't sufficient to drive safely. It can't see lane markings for example. Therefore this isn't a solution. The correct thing to do if you can't drive on vision alone is to not drive. Or - and I'm dead serious here (fully automated logistics is basically so ridiculous for enabling massive wealth that very unrealistic things are worth pursuing to attain it) - modify every road in the world so as to trivialize driving with sensors other than vision.
2. Seeing obstacles that are hidden from sight. This is an edge case that matters and is where you would get the big win, but it isn't actually just a win. It is also a huge complication. Uncertainty matters and you just made it very hard to project a cone of occlusion producing uncertainty because now we are claiming knowledge of occluded areas with an error prone sensor. You need some feedback mechanisms here or you are going to underestimate/overestimate your ignorance in situations involving uncertainty, potentially resulting in bad driving decisions as a consequence.
3. The more complicated you make this, the greater the potential latency to make the decision. Pursue the theoretical best and latency could increase enough so that you delay your decision. However, this is a real time system. It can't afford pure theoretical best, because latency matters a lot. That makes the sensor fusion more useful as something you use in the backend systems that aren't real time - training time and inference time have very different properties.
I actually basically agree with your larger point that in theory these other sensors should be able to benefit. However, I disagree that getting them to the point where they do benefit is easy. I think it is actually pretty hard.
In my other reply I tried to say you were wrong in a way that was rude, which I regret, so I want to try again in a more respectful way.
Initially, it was the thought of most people who thought about the problem that more sensors would be best. This was what Tesla thought, not just what you thought. It was what I thought. I've read Artificial Intelligence: A Modern Approach. I've read about Kalman filters and sensor fusion. I've implemented them. So intuitively, I think it makes a lot of sense that more sensors should be more effective. So it was surprising to me when Tesla decided to drop radar and it was even more surprising to me when they shared that the empirical results of dropping radar led to measurable improvements in vehicle safety and improved their accuracy in determining their position relative to other objects.
In the past I've made that claim and people have been surprised by it. It doesn't seem to be common knowledge. If you didn't know it or you doubt it, then I'd recommend you check out an engineering talk by Andrej Karpathy. This isn't a Tesla marketing piece. It was a workshop talk at the Computer Vision and Pattern Matching conference in 2021. Andrej Karpathy is the senior director of AI at Tesla. He was someone involved in this decision to switch from radar to vision only and in the talk he outlines the engineering reasons which motivate the switch.
If you still doubt that it is a good idea to switch, I can contribute my own anecdotal experience. I have a Tesla. I drive in it. When I do, I use autopilot very frequently. The majority of my driving is done by autopilot. As such I've gotten experience which has informed me a bit about how autopilot tended to fail. One of the ways it could fail was by making me take over when going under an underpass or by breaking suddenly when doing so made little sense. This is a category of error that I have stopped experiencing since the switch away from radar.
So now we're in an interesting position, because if you recall you claim that people who believe vision only are naive. Naivety is usually defined as meaning to be wrong because of idyllic assumptions that are wrong, but which you don't realize are wrong because of ignorance. Yet the way things played out historically is that people assumed that sensor fusion was the best approach, empirical results suggested it wasn't the best approach, and consequent to that people changed their minds. In terms of the progression pattern, this is exactly opposite of what it means to be naive, because the beliefs are contingent on experience rather than a consequence of its lack.
Sometimes we have simple models of reality and they suggest one thing. Then we get experience in the real world, which is much more complex than our simple model, and that experience tells us something else. Telsa, like you, thought radars plus vision was better. They tried a model that dropped radars and the result was empirically measured as being safer.
Should we go back to the thing that we know to be less safe through empiricism? Well, if we do, but we do so without solving the reasons it was bad, then we are going to have the car making bad decisions. Those bad decisions put lives at risk. So I don't think we should go back. If we can address the root causes of why the sensor fusion approach introduces dangerous error, then we can do that, but just adding back radar? That would kill people. Probably the people directly behind a Tesla that breaks because of a hallucinated object detected by radar.
I wish I could delete my other reply, but I think the example in it is extremely important, because when people try to use the toy problems that fit in their head? They are choosing to exclude that very very real non-theoretical situation. That is exactly what the world with radar was generating as an inevitability and we need to be crystal clear on that when we reason out exactly how to avoid that type of error. Blindly saying that sensor fusion is better is really dangerous, because people can really die if we make the wrong decisions -- and because we can measure the results of different models empirically? It is blindness to say it, because it contradicts the evidence.
Radar gives false positives. Pretend you are a computer for a second. You get told by radar that right in front of you there is an obstacle. You are five feet from crashing. The only option to avoid it would be to slam on your breaks. Meanwhile the vision system tells you that you are on a freeway, there is a car directly behind you, and there is a bridge ahead of you.
Do you a) slam on the breaks or b) drive under the underpass that was producing the false senor reading?
You've already declared your answer. You choose option a, because you think radar is amazing and you think people who think otherwise are naive morons. What happens next is that you slam on the brakes, surprising the person behind you. They slam into your vehicle. Their child wasn't wearing his seat belt. He flies forward, slamming through the window. His brains splatter the pavement. His body rolls without his brains into another lane. A horrified person to your left swerves to avoid hitting the kids body as they drive by, plowing into another car.
Congratulations. You are a genius. Everyone else is naive. Thank you for playing murder innocent people.
Of course, this isn't what actually happens. What actually happens is that your decision results in a sudden break, but the person at the wheel recognizes this is a mistake and presses down on the pedal. Your decision making ability is taken away, because you are a moron. They do so, because they trusted their vision system more than they trusted your stupid radar based decision. So the error correction mechanism that stopped your murder attempt? Vision.
Thank God for that, but the person in the vehicle is annoyed. They report the issue, not liking that bridges consistently produce that behavior. Tesla investigates. They realize that the radar sensor is producing false positives. After empirically validating that removing radar is better at driving in this edge case they roll out the improvement. Tesla removes your ability to decide to kill people because you are too obsessed with radar.
Later, even as the empirical results show that self driving is now better than human driving in terms of safety, partly of course due to it being human+computer driving now computer alone, a person names knodi123 goes online and calls people naive for thinking that vision is preferable to radar.
He gets asked this question. If you were the driver of that car, which would you rather trust? The car's decision made on the basis of radar? Or your own decision made on the basis of vision?
They recently ditched radar. They're vision-only now.