When the continent of Yokuda sunk into the sea, Tava the Bird God guided a large bulk of the Yokudans to the island of Herne. The vanguard of warriors known as the Ra Gada arrived on Hammerfell and wiped out the Nedic villages in the area for the refugees in Herne to inhabit in. The people waiting on Herne also consisted of the Na-Totambu, the ruling kings and queens of Yokuda.
The isle of Herne is considered a barony in Hammerfell in 2E 582. It's possible some Yokudans stayed in Herne rather than moving to Hammerfell.[OOG 1]
The Daedric Prince of Madness, Sheogorath had considered taking Herne along with Eyevea. However, Sheogorath wants to have something with a unique shape. He despises regular looking shapes, especially rhombuses.[OOG 1]