Back to Haemar's Shame, in the shrine, I guess.
thanks thats very helpful :)
Yes, if you lose sight of him upon "stop following" before the quest is over and go into any loading screen area, he will go back to Haemar's Shame. I wish he had a "wait here"...