Is it possible to make it through the haunted wasteland (from the Spirit Temple) as young Link?

There are a few possibilities to get into Gerudo's Fortress as a kid.

Yes, you can go backwards from the Desert Colossus; however, you don't have the poe guide on the way back. This means that you would have to know your way through the Haunted Wasteland by the mini-map or your own personal landmarks. This is difficult, and the only real way to learn how to do it is by observing some YouTube videos, as well as your own trial and error. If you did complete this task, then you will be met with the river of sand, which you must backpedal over (YouTube instructional videos exist). If you have already opened the wooden gate (from the Fortress to the Haunted Wasteland) as an adult, you can enter the Gerudo's Fortress. If you have not opened the gate yet, you will not be able to walk in.

The other way is over the gate in Gerudo's Valley. This is equally as difficult. There are currently two methods to get over the gate, one being the cucco jump, and the other being the bombchu hover. Through my own personal experiences, I find the bombchu hover is easier to perform, however they are both difficult (as usual, you should use YouTube as a resource to learn these techniques, or post another question about them). If you attempt this task, keep in mind that the guards can catch you if you don't have the Gerudo Membership Card (however if you have the card, you are safe).