There is an easy way to steal a horse and keep it while outside of a city. A pedler is random encounter. Usually being attacked by bandits you can save him or not, sometimes he survives other time hes doesn't. Either way kill him. He has no faction so no bounty will be put on your head.
Steal his horse.. seeing its master is dead the horse will not walk away though I've found entering a building with a quest the horse will dissapear.
Killing the Peddler and stealing the horse will be tallied against you as far as crime done but otherwise you will face no consiquences, save for gold, pelts, and a free horse.Read more >