Game lags if alt+tab out of game and then back into it

Game run wells on my pc except if I end up alt+tabbing out of the game to another window then go back to the game it lags, like dropping frames or something. Does this on all game speeds. Only solution I have found is to save the game then instantly reload the save and the lag is gone.