I've read House of Leaves.
For a long time I swore I would never read it again, not because of the difficulty of the experimental layout, but because of the bone deep horror it instilled.
It really shook me. I HOL gives you an Understanding at a gut level that metaphors are based in reality, and using that fact to show how that can lead to the most twisted consequences of reality.
But short answer, no. I've never come across anything like that book anywhere else.
I've just seen that he's written quite a few other books. Have you read any of those?