How is it possible for a city to be both blazingly public and intensely private at the self-same time? Somehow Edinburgh manages the feat, even in August with festivals all go and the population almost double the norm.
This quirk is something I've explored many times in my novels, because to me it says lots about the very nature of Edinburgh and how it came to be the way it is. The Scottish capital is bursting with stories, but sometimes you have to tease them out. Its history seems apparent from the moment you arrive, yet there are things you'll never see unless you know where to look – or get lucky. I've lived here more than half my life, yet can't claim anything close to an encyclopaedic knowledge of the place.
Maybe it is all an accident of history and geology. In times past, while invading armies prepared to strike, the denizens would secrete themselves in tunnels dug beneath Castle Rock and the Old Town. You can still get a sense of this underground existence by visiting the Blair Street Vaults or Mary King's Close (where narrow streets with houses on either side housed citizens during the 16th and 17th centuries). When the invaders arrived, they'd find the city empty. It was easy for them to ransack and loot, but they would soon get tired of this and march back out the way they'd come, at which point the citizens would rise up from their underground hiding places.
Edinburgh has always seemed to me a furtive place. Throughout history it has made its money from invisible industries such as banking and insurance. And while the city has been known to celebrate its success stories (the Scott Monument) and flag up folly (the unfinished "Parthenon" on Calton Hill), it is not a place where people flaunt their talents. You don't see many Ferraris – the wealth sits quietly behind the New Town's thick Georgian walls.
It was once called a city of "public probity and private vice" and this still rings true, though the "probity" tag has lost some lustre since the near-collapse of Royal Bank of Scotland, one of the city's biggest employers. But visitors to Edinburgh, if they stick to the main tourist routes, will be seeing only the city's most public side. Travel just a little further afield and you can widen your appreciation. That's why, on a blustery day, I set out from the Oxford Bar for a walk.
This isn't a random starting point. I discovered it as a young writer. I'd invented a character called Detective Inspector John Rebus, and he needed a place to hang out. The Oxford Bar is central (Young Street is a two-minute walk from Princes Street), yet hidden. It is small, but contains the widest possible cross-section of Edinburgh life.
As I walk in, there are a few nods of greeting (nothing too effusive). Kirsty behind the bar has guessed that I'll want a pint of Deuchars India Pale Ale. Edinburgh at one time had more than 40 breweries – the Scottish Parliament sits on the remains of one of them. These days, though, there is just the one. It's called the Caledonian Brewery, and that's where my IPA was made – about two miles from here as the crow flies.
The "Ox" is run by Harry Cullen. Harry used to sing in a folk group (though he won't thank me for publicising the fact), and has a fund of stories of his own. In fact, everyone I have ever met in the Oxford Bar has a story to tell. I ask Harry today if any Rebus fans have been in. He rolls his eyes.
"Two of them took photos – without buying a drink!" He then asks me if I'm having another. I shake my head.
"Things to do," I say by way of apology.
"That's my profits shot," he mutters, polishing a glass.
With a shrug and a wave, I head out, crossing nearby Charlotte Square (home to the First Minister) and emerging on a rain-soaked Queensferry Street. The shops soon disappear as I approach Randolph Cliff. I cross the road and head down Bells Brae, turning right at a signpost announcing that Leith is two-and-three-quarter miles away. This path, deserted apart from the odd dog-walker and jogger, runs along the Water of Leith.
Robert Louis Stevenson once called Edinburgh a "precipitous city", and he was absolutely right. Whether you're peering down on to Princes Street Gardens from the castle, or craning your neck to look up from the Cowgate at George IV Bridge above, you sense that Edinburgh contains an intensity of heights and depths.
I've decided against walking all the way to Leith, so emerge at the Dene. One option here is to cross the footbridge and amble up Edinburgh's loveliest street, Ann Street, with its neat gardens and immaculate Georgian facades. But instead I carry on to Raeburn Place, emerging from one village, Dean Village, into another, Stockbridge. From here, it is a short uphill stroll into the New Town proper. When the Old Town, stretching from the castle down to the Palace of Holyroodhouse, became overcrowded and insanitary, the New Town was proposed, with work commencing in the 1770s.
At this point, I have to admit that I get lost in the New Town. My destination is Kay's Bar on Jamaica Street. Somehow I manage to skirt it, passing the art galleries of Dundas Street and a favourite fish and chip shop, L'Alba D'Oro. I walk down Heriot Row (once home to the young Robert Louis Stevenson), then double back on myself, and stumble upon Kay's almost by accident – the sort of happy accident that makes an Edinburgh walk such a pleasure.
There are some cities in the world where you'll find conversation at every turn, but not here. Edinburgh is quiet and reserved – a place for thinking. Maybe the locals only loosen up when they enter their favoured watering-hole. After walking in silence, it's nice to lubricate (and then exercise) the vocal cords. There is chat in Kay's.
Rested, I start to climb back uphill, walking along Queen Street, past the Scotch Malt Whisky Society, which boasts hundreds of malts and no two alike. A right turn takes me to George Street, the grande dame of the Princes Street area.
Time was, it was filled with banks, but most of these have become bars and restaurants. The Dome, for example, was the Commercial Bank of Scotland headquarters. It is now a place to do lunch, beneath a spectacular glass dome in what used to be the main banking hall.
At the far end of George Street, I turn left and then take a right. I'm now where the New Town meets Leith Walk, and I pause outside another bar, the Conan Doyle on York Place, staring across the road at a modern addition to the city's monuments – a statue commemorating Sherlock Holmes, whose creator grew up in Edinburgh, basing his character on one of his university professors.
The weather has brightened by now, and I see no reason to head home just yet. I have some shopping to do. A friend in London collects LP records, and he visits Edinburgh a couple of times a year, due to the quality of the record shops. There's one on the Canongate, and a couple on Leith Walk. Another stretch of good shops runs the length of South Bridge, Nicolson Street and Clerk Street, and includes (just off the main thoroughfare on East Crosscauseway) Backbeat. Dougie McShane opened Backbeat in 1981, selling mostly blues albums. He now sells pretty well everything, and he reckons he has 65,000 items in stock. An English lad emerges with his finds. He's a blues fan. My own son is 17, and likes vinyl and the blues both. I ask Dougie about this.
"It's great," he says. "Teenagers buying record decks and then coming back to proper records because they just sound better – less harsh, more authentic."
To get to Backbeat, I've walked past my favourite museum, Surgeons' Hall on Nicolson Street, where visitors can view the death mask of the serial killer William Burke (of Burke and Hare fame) and a wallet made out of his skin. So much more interesting than dinosaur bones, I've always thought. Having escaped Backbeat with just the one purchase (of boogie-woogie piano tunes), I notice a couple of early Patricia Highsmith (an American crime writer) novels in a charity-shop window, so I have to have those, too.
I have a plan in mind now, and I hail a taxi, telling the driver to take me to Blackford Glen Road. We get into a discussion about the binmen's strike, the weather and the tram construction. He reckons this unholy trinity will kill tourism stone dead. I remind him that in the past few days, Edinburgh has been voted the best city in Britain.
"Somebody somewhere must like us," I say.
The driver doesn't seem convinced, and drops me at a dead end. But not quite a dead end – a path leads into the wooded parkland called the Hermitage.
This is another steep-sided valley. To the south sits Braid Hills Golf Course and, to the north, Blackford Hill. I could be in the middle of the countryside. A stream runs through the Hermitage and, as usual, a dog comes bounding out of the water, shakes itself dry next to me and deposits a stick at my feet.
The owner apologises, but there's no need.
Here, again, is the hidden Edinburgh. Only locals ever seem to find this place. Their kids wear wellies and play Poohsticks or disappear into the trees. Strangers smile and nod at you. If you're here, you're not really a stranger at all – you are part of a community. By the time I emerge on Braid Road, my batteries have been recharged.
What's more, it's a short stroll down into Morningside, my final stop before home.
The Canny Man's is another famed Edinburgh pub, filled with bric-a-brac and hideaways where you can sup a quiet pint or savour a malt. If you want to be left in peace to read the paper (or start a Patricia Highsmith novel), that's fine; if you'd prefer conversation, that's fine too.
Morningside is yet another Edinburgh "village", and I live on the edge of it. As a student, I rented a shared New Town flat. Later, I made my home near Backbeat Records (coincidence, I assure you). I have also lived in Tollcross and Oxgangs and Peffermill. Each one boasted its own unique atmosphere; each holds memories.
Today's walk has taken me just under four hours. I've come to know my city a little better, but am left with a yearning to know still more.
• For a longer version of this feature, see Lonely Planet Magazine's November issue, which is out now (lonelyplanet.com/magazine).