They seem to respawn after a couple of minutes, and even if all the men in the caves are slain, they will keep coming. This can be an excellent way to improve battle skills, but be careful not to be overwhelmed. They may be quite weak on their own, but they respawn infinitely so being surrounded is a possibility.
Keep in mind, doing this will deteriorate armor, and so a shield spell might help.
If the Hero is unlucky, the Brethren will climb up a trapdoor in one of the houses and chase them away from the village. If they don't kill them, they'll keep chasing them farther away, even to other regions.
There are a lot of bones lying in the cell down in the cave, which probably means Dar-Ma is not the first to fall victim for the Brethren.
The Brethren are made to resemble the people of Innsmouth from the H.P. Lovecraft story The Shadow Over Innsmouth. This is a tie in to the Oblivion quest "A Shadow over Hackdirt ".