You don't have to do anything. But as you say, they are the one in the position of power. If you are working on some side quest they don't see as valuable, it may not end well for you. Doubly so if you are shirking what they see as a high value task for your side quest.

It's not about what is right or who "should" do what, it's about securing the best outcome by making sure you and your manager have the same understanding, even if your manager isn't doing a good job of making sure you have the same understanding. (Also known as "managing upwards.")