Challengers want the duel to fit all of their standards. They will often complain if the Dragonborn's followers join in. They will also be upset if weapons are brought into the fight. If the Dragonborn does draw his weapon the challenger will draw his own weapon. If the Dragonborn uses any bound weapons, the challenger will complain. This may occur because the bound weapons count as one-handed/two-handed/archery.
If the Dragonborn accepts the duel in a town or city, guards will not help. However, refusing the duel request will cause any guards in the area to come to the Dragonborn's aid.
Warning: the Challenger is extremely dangerous if the Dragonborn allows his skill in any one magical category to reach Adept level early on, before building sufficient Health reserves or anti-magic protections (heavy reliance on Destruction magic or bound weapons in the early game being a case in point).