"Just magically logging in" is more of a nice side-effect than the intended purpose, in my opinion. SSH keys allow you to let multiple people log into a server without needing to set up complicated user accounts and without sharing a password that quickly becomes difficult to change.
You can have the best of both worlds by storing the key itself in a place that's not readable by many programs. TPMs and other such tech can store a key securely without risk of FunnyGame.app sending it to a remote server. In this model the key would be stored inside a safe, sandboxed place, only readable by an SSH agent or similar, which will prompt for permission to use the key every time. With fingerprint scanners and other biometrics being available even in cheap devices, this process can be relatively seamless.
If you run sufficiently modern SSH software, you can also use external key stores like Yubikeys to authenticate with plain old OpenSSH.
You can have the best of both worlds by storing the key itself in a place that's not readable by many programs. TPMs and other such tech can store a key securely without risk of FunnyGame.app sending it to a remote server. In this model the key would be stored inside a safe, sandboxed place, only readable by an SSH agent or similar, which will prompt for permission to use the key every time. With fingerprint scanners and other biometrics being available even in cheap devices, this process can be relatively seamless.
If you run sufficiently modern SSH software, you can also use external key stores like Yubikeys to authenticate with plain old OpenSSH.