The Great Collapse was a geological event that devastated Winterhold. On 4E 122, a seemingly never-ending series of storms lashed out at the northern coast of Skyrim, eventually causing most of the city to erode alarmingly fast and be washed away into the Sea of Ghosts. The event crippled Winterhold but miraculously left The College of Winterhold largely unharmed.
Many of the the locals, however, blame the College itself. If the Stormcloaks have been driven out of Winterhold, the Imperial Jarl trusts Arch-Mage Aren's assurances that the college was not involved, though the suspicions of the locals remain. Thus, the mages, and their associates, are generally disliked and avoided.
Since the cataclysmic event, Winterhold has never truly recovered. Immediate prejudices emerged surrounding the Dunmer refugees of the College, as detailed in a letter written by the then Arch-Mage Deneth, to the Jarl, Valdimar.  Despite the formal apology, the Dunmer and mages were never truly trusted again.
- The Elder Scrolls V: Skyrim (Mentioned only)