Well, if you discard the phase of the original response anyway, then you can shave a few milliseconds of latency by switching to minimum-phase (which is causal, not symmetric) instead of linear-phase. The math is in scipy.signal.minimum_phase.