The first door you need to complete at least half of the nightmare itempedia, as for the second door you need to complete all of the nightmare itempedia.

Dang it Thanks and does that mean I have to find almost every thing in the nightmare

You have to find every item in the nightmare for the second door to open.

The first door you need to only find half of the nightmare itempedia.