Or you could use voice authentication... or require that known phones are within range, or best yet don't link your smart speaker to your door lock (when do the pros outweigh the cons here?)