That very much depends on your security posture. For a desktop computer, with a single user, panicking the kernel is at most a mild inconvenience. As it's said, it's a denial of service, but the service being denied is your own service.
Security, as the reunion of Confidentiality, Integrity, and Availability (CIA), definitely includes vulnerabilities where a malicious user can trigger a kernel panic, as it impacts Availability directly, and potentially Integrity as well in an indirect way.
The distinction you make is a separate step down the line: you do a risk assessment and decide that, in your particular context, a specific vulnerability is not a threat worth defending against.
For that same vulnerability, others will reach different conclusions in their respective risk assessments.
If you can install a program that runs at system startup and panics the kernel, that's beyond mild inconvenience - it's potentially a major IT scramble for a whole company's fleet, for example. Even for a server machine, it might require complicated debug procedures and/or physical access, since you can no longer just SSH into the system.
This has to do with their policy on assigning CVE numbers, which is that pretty much any bugfix might be security-related because it’s the kernel, so it doesn’t take much to get a number assigned. See https://docs.kernel.org/process/cve.html.
I seem to recall that Linus Torvalds has the opinion that he doesn’t much treat security bugs more differently than he does regular kernel bugs. Perhaps this is why?
Yes, but it became more than just Linus and Greg’s view that couldn’t be overcome by outside argument, and became more formally Kernel Policy once they became a CVE number assigning authority.
If a kernel panic is considered a security issue, anyone using Nvidia's drivers should fear for their lives.