There are some Dwemer ruins very close to the shrine, guarded by a Wispmother.
Unlike most Daedric Shrines, the Shrine to Peryite does not give the player a visual representation of the Daedric Prince, unlike in Oblivion in which the player sees a Dragon-like figure. During the conversation with Peryite however, apparations of Skeevers may appear. These could represent Peryite's familiars, his own conduit on the mortal plane or possibly just hallucinations from the fumes.
Behind the shrine to the east, at the bottom of the mountain are the 'roots' of the shrine. A chest containing random loot, a dead Wood Elf and a random enchanted dagger are next to the pillar depicting Peryite. Similar to the pillars found in Bthardamz.