It’s midwinter in the South Island city of Dunedin but the sun is high and the thermostat has just pushed past 10C. Settled by the Scots and known as “the Edinburgh of the south”, Dunedin has long had a reputation in New Zealand as the country’s avant garde student haven, a gritty post-industrial city now home to artists, musicians, academics and more than a handful of eccentrics. Odd is acceptable in Dunedin – applauded, even.

The small city of 120,000 sprawls around the pristine Otago peninsula, which hosts significant threatened populations of sea lions, penguins, seals and rare birds, including the world’s only mainland albatross colony. Cars and trucks routinely stop for wildlife crossing the city’s main thoroughfares, while a gothic city aesthetic contrasts with the golden beaches, sheep-dotted paddocks and bucolic harbour views.

No one’s ever in much of a rush in Dunedin, and locals have time to give in spades. Expect to be invited to join communal tables of strangers if you turn up at the local pub and people don’t know your face. Locals will probably want to buy you a drink too and hear your story.

Food and drink

Locals enjoy the sunshine at the Esplanade on the St Clair beachfront, a favourite for casual, flavourful Italian dining. When we visit, we order large glasses of sauvignon blanc, a capricciosa pizza and steaming bowls of Italian pork sausage pasta. With uninterrupted sea views of the Pacific Ocean, a classy yet relaxed interior and reliably delicious food, the Esplanade is always packed, buzzy and good fun.

Up on Māori Hill, the neighbourhood bistro No 7 Balmac has just reopened with a new, Middle Eastern-influenced menu. Specialising in wood-fired cooking and local produce, the food here is perfection. We start with octopus skewers in turmeric oil and coriander, and local clams in spicy ’nduja butter with a carafe of house red, followed by confit duck leg, tender wild venison and pork belly that is so more-ish we order seconds. A favourite with Dunedin locals for special occasions, No 7 also has great bar seating and an extensive wine and cocktail list.

Glenfalloch Garden is a 10-minute drive out of town on the Otago peninsula and, despite it being the middle of winter, the 30-acre garden is already blooming with early rhododendrons, pink camellias and late, lilac hydrangeas. My friend and I have no plans for the afternoon so we settle in and order the popular “trust the chef” degustation menu. This four-course feast makes extensive use of local, organic and seasonal produce and is cooked with finesse by the German chef, Hannes Bareiter. Three hours pass before we call it a day then take a meandering stroll through the gardens. This is a must for a luxurious weekend lunch or dinner.

Music and nightlife

The city is the home of the Dunedin sound; a fervent period of musical innovation in the 1980s which produced the locus of the emerging New Zealand punk and post-punk scene. The Verlaines, the Clean, Straightjacket Fits and the Chills all started off in Dunedin, and the city’s musical roots continue to produce and nurture artists today.

A night out at a live gig is almost obligatory in Dunedin, but ease into it with a pint of local Emerson’s beer at Albar or Dog with Two Tails, before heading to the Cook in the student quarter of town for whatever gig is on. The Cook’s publican, Mike McLeod, is a musician himself and reliably scoops up the best bands. Post-gig, if you have the stamina, you could continue on to the Crown for another gig and a round of pool. Otherwise head back into the city for a nightcap at Pequeno, a darkly lit wine bar with a roaring fire and a snug, late-night vibe. To enjoy the city’s well-established hipster scene, drop into the New New New brewery on a Friday night for a pint and snacks from one of the food trucks parked out front.

Activities

Dunedin’s wildlife and beaches are unforgettable – and no visit would be complete without a brisk walk along Allans beach, Long beach, Sandfly Bay or Blackhead. Depending on the season and weather conditions, you can expect to see wildlife roaming free in these areas. Head up to the Orokonui wildlife sanctuary above Port Chalmers to catch a rare glimpse of New Zealand’s vulnerable native bird population, and expect to see kākā, takahē and bellbirds in their natural habitat. World-famous albatross can be viewed at the Royal Albatross Centre or from the Monarch, an elegant wooden boat that meanders up and down the harbour. A southern skies stargazing tour will give you goosebumps of delight and, in the right conditions, the aurora australis may be visible.

As a university town and a Unesco city of literature, Dunedin has a thriving secondhand book scene. Check out Dead Souls Books, Scribes and Hard to Find, or head to UBS for new reads. The staff are well-read and can be trusted for excellent recommendations.

Dunedin is New Zealand’s oldest city and boasts a rich architectural history, including the country’s best collection of Edwardian baroque-style buildings. A ghost tour of Larnarch Castle will raise the hairs on the back of your neck, and guests can stay overnight at the castle’s lodge to complete the experience. (I was visited by a ghost. Really.) A tour of the genteel Olveston House evokes Dunedin’s time as the commercial and social capital of gold-rush New Zealand, when money poured into the port city from the central Otago goldfields.

A stroll down the once industrial, now increasingly edgy Vogel Street showcases the city’s austere beauty. It’s also an excellent place for coffee aficionados to begin their morning. Check out Heritage, Good Good and Wolf at the Door.

Stay

For a modern edge and creature comforts, try the four-star Distinction hotel. If you want to book a bach, Kiwi-style, check out the Otago peninsula offerings on bookabach.com.

• The writer visited many of these destinations courtesy of Tourism Dunedin

Looking for a holiday with a difference? Browse Guardian Holidays to see a range of fantastic trips