While he was Jarl, due to the Empire's resources being focused on the war with the Thalmor, the Breton former rulers and natives of The Reach seized the opportunity and raided Markarth, expelling Hrolfdir, Igmund and Raerek. Eventually, the son of the Jarl of Windhelm, Ulfric Stormcloak, a strong opponent of the White-Gold Concordat, came to the rescue, using the Voice to force the natives off the walls, and massacred members of the native government.
Before allowing the Imperial forces to return to Markarth, Ulfric and his men demanded that Hrolfdir repeal the ban on Talos worship in The Reach. This put Hrolfdir in a difficult position - as repealing the ban on Talos worship would violate the White-Gold Concordat and place the Reach under the scrutiny of the Aldmeri Dominion. Ultimately, Hrolfdir agreed to Ulfric's terms, hoping the Thalmor would not learn of it. Unfortunately for Hrolfdir, they did, and threatened the Reach with war. Hrolfdir found himself and his family expelled from Markarth once more, and they were forced to swear to abandon Talos worship and uphold the White-Gold Concordat before being permitted to return to Markarth and resume governance, although as a consequence, the Thalmor cracked down on Talos worship across Tamriel and maintained a presence in Markarth, which lasted at least until the reign of his son Igmund.
As for the native Bretons, they refused Hrolfdir's offers of peace and fled into the wilderness under the command of King Madanach, dubbing themselves the Forsworn, and began working once more to overturn Nordic rule of the Reach, an uprising that continued for some years.
Hrolfdir was ultimately unable to quell this uprising and was murdered by the Forsworn while attempting to negotiate with them, and was soon succeeded as Jarl by his son Igmund.