Once you have arrived at the desert, turn into wolf form since you have some distance to cover, and head to the east. When you have gone far enough, you will see a large grey block atop a platform. Turn back to human and grapple up to the flying enemy above and let it take you over the platform. Drop down and fight the three shadow beings there to open another portal. Now you will need to teleport to the Bridge of Eldin and bring the grey block with you. If you don’t have that teleport open yet on the map, go to Castle town and run over to the bridge and fight the shadow beings there to open it.
Once you have brought the bridge piece to the Eldin Bridge, teleport back to Gerudo Mesa and head to the north until a scene plays and two boars attack. Use the bow to take down the riders and then commandeer one of the boars to ride. Go towards the gates and use dash to crash through them. Enter the next area and clear out the huge swarm of enemies inside. Continue to the Northwest corner where there is a roast and find the silver key there. Go through the gate you passed on your way and you will now need to fight a rather large enemy with an even larger hammer. Wait for him to swing and dodge and then attack him with slashes. Once you have hit him enough, he will run away and set the building on fire. Hop on the boar and crash through the gate and you will then reach the entrance to the Arbiter Grounds.