Playing to Lose
One of the hardest things a GM has to do is lose. We can say, "Oh, the point of running a game is to make sure everyone as fun, so as long as everyone has fun, that's a victory!" And while that is true, it also can be frustrating to build things up only to watch players knock them down. Your plots - ruined. Your dramatic moments - turned to comedy. Your favorite NPC - the laughing stock of the party. Being a GM sometimes means herding cats, and sometime means letting the cats go wile.
Being flexible does not mean being a push over. Games are supposed to be fun for everyone - GM included!
How can we be flexible while also accomplishing plot and narrative goals without burning out?
There is no single answer, and usually I rely on a whole toolbox. I pick and choose the tools and methods that work for me and my group in the moment. And sometimes, the answer is to step out of the role of GM, and talk with the group as a tired person who is doing their best - because at the end of the day, we're all tired, we're all people, and most of us are doing our best.
Tool One - Fixed Points in Time
I think about campaigns like seasons of Dr. Who.
Your players are the Doctor and the companion(s). They are going to bounce around all over the place and try to save people when it's the wrong thing to do, and do nothing when you need them to do something. They are going to stick their noses into things you didn't plan anything for and ignore the nicely put together plot that you spent all week prepping. But there can be fixed points in time that are immutable, and that allows for some control and consequence!
Have large events that are absolutely going to happen whether the players are there or not - then you can "script" the narrative.
As the players engage in adventures, let them alter outcomes. Save prince A and he will defend town B, but let prince A die and the players will have to defend town B, or abandon it. Whatever happens, town B is going to be attacked, so the GM can plan that out and decide what happens if town B makes it and what happens if town B falls into enemy control. Having multiple endings to each encounter allows a GM to be more flexible! That flexibility will make the game more fun. As a bonus, the GM never wins or loses, because all that hard work and planning still comes into play.
Tool Two - Move Things
Instead of trying to get the players to go to town X, have them decide where to go, and wherever they end up, put in your plots for town X. The illusion of choice is as good as choice itself for the players.
If you want the characters to go into a cave at the bottom of a mountain and they choose to head to the coast, then move the cave to the coast. When you're GMing, you can move anything in the world. Think of it as setting up a chessboard by placing pieces that you want to be captured. Instead of defending your king, you are setting up the order in which you want your pieces to be captured leading up to the capture of the king. And whenever you want, you can just pick up a pawn and move it into range of the players.
Tool Three - Human to Human
I have, on many occasions now, taken time to talk with my players out of game to let them know when I am struggling. Sometimes it is difficult to tell if players are not engaging with the plot because they don't care about the plot, or because they don't know what they're supposed to be doing. It can feel like a failure to admit, as a GM, that you are struggling to keep things going. But a good group will want to support you as you figure things out. It's okay to be honest, and have those out of character, meta conversations. Sometimes players miss things, forget clues, misremember who did what, and this can lead to confusion, inaction, and what looks like chaos for chaos' sake. Sometimes that is fun for the players, and sometimes it is just as frustrating for them as for the GM.
So, if you're feeling frustrated and can't figure out how to get your players to engage with the story, consider being open and honest about your struggles. You may be surprised by how your players respond!