You're going to Milford Sound anyway. On the way, take a rest stop to check out these caves. For one of us the best part was exploring the cave, for the other the best part was emerging from the cave.
There is something amazing about parking near a sheep farm, hopping a fence (you're allowed to), and ducking inside the earth. It's not the biggest or most spectacular cave, but it's very special to be able to explore underground in a way that you rarely get a chance to do - by yourself. Plus, there are glowworms!
This small cave system has a trail of reflectors provided by the DOC. Be sure to bring good flashlights, and have back-up lights and batteries. In contrast to much of our travels in New Zealand, there wasn't anyone else here. If you get stuck without a light, don't expect that somebody will be there in a few minutes, or a few days, to help you out. Note also that if you have high water, as we did, you will likely have to turn around at one of the underground pools as there was no way to walk around them or tell how deep they were. There is mostly a single cavern, except for one intersection. But the reflectors show the way. Note that we didn't take very many pictures because one of us was excited about getting back above ground as soon as possible :)
These caves are on the South-western part of the South island, on the way between the Catlins and Fjordland.
The caves and the "trail" of reflectors is maintained by the NZ Department of Conversation. They have information on the caves here. Like many of our favorite adventures, we discovered these via the NZ Frenzy books.