The mammoth tusk for Ysolda can be found inside Halted Stream Camp, which has several of them. Alternatively, they can be found on mammoths located at giant camps throughout Skyrim. There is also an easy steal if the Dragonborn heads to Jorrvaskr, in a separate bedroom to the hall, belonging to Vignar; or from on top of a cupboard behind Hulda at The Bannered Mare.
The Black-Briar Mead can be found at the Black-Briar Meadery. Alternatively this item can be bought at most taverns in Skyrim. The closest seller is the Dead Man's Drink in Falkreath. Another bottle of Black-Briar Mead can be found in the underground area under the castle in Helgen when you escape with Hadvar (it can be found next to a skeleton and a metallic helmet, which both are right by an empty, wooden wagon). With Hearthfire installed, this quest will become unavailable if you do not acquire it before reaching level 9. If you have acquired it before reaching level 9 but have not yet completed it, it will fail as soon as you reach level 9. You will subsequently receive a letter from Siddgeir, via courier, about working for him in order to earn the chance to own land in his hold (Lakeview Manor).
Song of the Alchemists cannot be obtained without stealing*. After accepting the quest, a copy appears inside Bards College. It is not marked as stolen, but if caught, it will still count as stealing unless the Dragonborn is a member of the college. It can also be stolen from Anise's Cabin. If this quest is taken when the Dragonborn's Alchemy level is higher than adept level then the Dragonborn will not receive the reward (skill increase).