> I haven't read the underlying paper so maybe they addressed this, but: couldn't this just be because sedentary/unfit people tend to drive everywhere?

It could be and that's the point - the exact causal effect of fitness might be overstated because some mortality reduction might have stem from - for example - driving less.