This is indeed the bread-and-butter, but there is use of all sorts of standard linear algebra algorithms. You can check various xla-related (accelerated linear algebra) folders in tensorflow or torch folders in pytorch to see the list of what is used [1],[2]
[1] https://github.com/tensorflow/tensorflow/tree/8d9b35f442045b...
[2] https://github.com/pytorch/pytorch/blob/6e3e3dd477e0fb9768ee...