I also had this problem in whiterun and giant inn.
Regarding whiterun, going back a few saves solved it. I modified bijin textures with fair skin complexion, and somehow that made some complications. Luckily I saved before doing those modifications. So going back few saves solved it. And also my modifications were there as I changed the textures manually.
But regarding sleeping giant inn, I couldn't go back few saves because I would have lost too much progress. I did everything, (SKSE modifying, Safety Load, Memory allocation). Nothing solved it. But I could go inside every other house. So, why not the inn. Then I understood because I was doing the horn of jurgen windcaller quest. And delphine was inside giant inn. Sleeping giant inn is part of the quest and I couldn't get in just there? Obviously, it had to do something with the quest.
I think, the quest generates some new data when I enter, but it was conflicting with something. So,I disabled all of the bijin mods then loaded again. Worked like a charm. Now, the data of the quest is loaded. So I saved and again enabled the bijin mods. No problem occured.
I think most of the time, these problems with infinite loading screen? It's most likely due to some texture mods. Disable them and see if it works.