Surely 'nothing' is an exaggeration. In the limit of the lens size going to zero, the brightness also goes to zero. (Ah, you're also scaling other things in your mind I guess, so then this is not true.)

If you keep the setup such that all the light that comes through the lens will get to the sensor, and keep the sensor the same size, a larger lens would mean a higher brightness, right? Maybe in practice you're limited by other factors, but 'nothing' can't be right. Especially when we're talking about theoretical limits.

Edit: I think what you're saying is that the lens doesn't matter for the brightness if you scale the focal length and sensor size in such a way that the brightness stays the same. Which is obviously true, but if you leave out the last part, it isn't.

A lens with a smaller focal length will produce a projected image smaller than the same lens with a longer focal length. A lens with a focal length of 1m and an aperture of 50 cm will produce a projected image that will be 4 times bigger than a lens with a focal length of 50 cm and 25 cm of aperture. So even though the amount of light captured in the first case is 4 times, the effective brightness is exactly the same because it is spread in an area 4 times bigger compared to the second lens. For this reason the only way to increase the brightness is to reduce the focal ratio, simply increasing the aperture maintaining the same focal ratio won't help. Obviously if you have an aperture of size 0 then all the light is wasted because you can't have a sensor with size zero. In an empirical way you can see this concept in the real world with the abysmally small lenses in the smartphones that have the same brightness of the much bigger lenses in SLR cameras or in refractor telescopes.