The brain has a lot of skip connections and is massively recurrent. In a sense, the brain can be thought of as having infinite depth due to recurrent thalamno-cortical loops. They do mention thalamno-cortical loops in the paper, so I think a more concrete definition of what is meant by "depth" would be helpful.
The "infinite depth" seems to be a matter of definition. It's practically infinite if you include feedback loops via learning.
If you exclude learning, then it's far from "infinite". Activations linger for up to 15-30 seconds, so at oscillations of around 30 Hz that would result in about 450-900 loops (times an unknown small multiplier for the actual number of layers). But the brain presumably only backprops/optimizes a few layers at a time and not much "through" time.
