Goat Island, west of the Harbour Bridge, was once the fledgling colony's gunpowder store and the source of the sandstone used in some of Sydney's finest buildings. This 13 hectare island was also the headquarters for the Sydney Water Police who kept an eye out for smugglers and escaped convicts.

Quarried by convict labour, the stone was also used in the island's own structures, including the superb magazine with walls so thick the temperature inside remained steady, whatever the season. Grisly relics of Goat Island's convict origins remain, including a seat carved out of stone by Charles Anderson, who was sentenced to be tied to the rock for two years. Scratching's made in the guardhouse by bored sentries are also in excellent conditions.