My Jamaica adventure began with Usain Bolt. Researching a book about the island’s abundance of great sprinters, I planned a visit to Trelawny Parish, where Bolt and several of his teammates come from. Most of my friends who have been to Jamaica had stayed in one of the resorts that dot the coast, particularly around Montego Bay in the north-west, but I wanted to rent a car and tour the island.

“You don’t want to hire a car,” one recent visitor told me. “You want to hire a driver.” This is what he had done and after a few days had spotted something glinting at his driver’s waist, something that wasn’t a belt buckle. It was a gun. I decide to hire a car.

In a pub in St James’s, London, I met Ian Thomson, author of The Dead Yard, a vivid, gripping account of modern Jamaica, who had promised to give me some travel tips and contacts. Over a pint of Guinness, he reassured me; well, he did. His book, which paints a fairly bleak picture, did not. And yet, travelling exclusively by bus, Ian had immersed himself in the Jamaican experience and didn’t just live to tell the tale, but actually seemed to have enjoyed himself. “You’ll love it,” he said.

During my first few days I explored Kingston – the city thrums with an energy that owes something to its dangerous reputation, which explains why it isn’t a mecca for tourists. It is worth visiting – it’s like plugging yourself into the mains. I take a guided tour of Trench Town, the ghetto where Bob Marley grew up, then drive out to the old city of Port Royal, and the Blue Mountains, recently designated a Unesco world heritage site. But the highlight is coming back each evening to the cooking at Neita’s Nest, a guesthouse run by Michelle, daughter of eminent Jamaican author Hartley Neita. Her housekeeper, Charmaine, lovingly prepared ackee and salt fish for breakfast; there are yams, fried bammies, boiled green bananas, cod-fish fritters with callaloo and jerked lamb burgers.

The trip to Trelawny at the other end of the island takes me through an interior which, from the air, looks lush with conical mountains and ferocious waterfalls. Up close it is even more spectacular, the roads narrow and heavily potholed as they weave through the mogul-like hills and dense forest of Cockpit Country. Driving in Jamaica is an intense experience. The roads are not dangerous per se, but some of the drivers are (“Undertakers love careless overtakers,” reads one billboard).

Entering Trelawny’s capital, Falmouth, – with its dusty roads, crumbling buildings and hand-painted signs above shops – it is as if the clock stopped in 1810, when the town was in its colonial pomp. In one direction went sugar and rum to Britain, in the other came slaves from Africa – up to 30 boats a day. The town boasted the finest Georgian architecture in the Caribbean, the pièce de résistance being the courthouse, the first building you would see on disembarking, but the buildings are now crumbling.

I am struck by another sight, one that was utterly incongruous. Sitting in the harbour, towering over the town, are two of the world’s largest cruise ships. That should mean up to 6,000 visitors, yet the streets, and the shops and market stalls, are virtually empty. The local pastor, Devere Nugent – an upbeat man who was also Bolt’s first athletics coach – explains why. The cruise company came to Falmouth, he says, with a proposal to redevelop the pier. Local people were encouraged to open shops and restaurants, but the company then built a replica of the town within the pier complex. A fence now keeps tourists in and locals out.

Which is a real shame, as Falmouth has charm and authenticity. Much of the town’s Georgian architecture has fallen into disrepair though efforts are ongoing to restore some of the grand old buildings; guided walking tours tell of the rich history of a town that was once the jewel of the Caribbean but now feels more like a rough diamond. (Falmouth had piped water before New York; these days there are villages in Trelawny, including Bolt’s, that still don’t have running water.) The area from Water Square to the old harbour has lots of stalls with the local speciality, jerk chicken, and others selling locally made arts and crafts. The businesses on Market Street are wonderfully eclectic and eccentric, among them the Miracle Medical Lab and Dr CAL Behasse, Clinical Christian Psychologist. Despite the lack of tourists, Falmouth feels safe and friendly.

The legacy of slavery is evident throughout Trelawny, though it is also the cradle of emancipation (thanks in particular to William Knibb, a 19th-century English minister who gave his name to the Falmouth Baptist church and the school attended by Bolt). Twenty minutes outside the town, perched on a hill overlooking a coast fringed by beautiful, mainly deserted beaches,, stands one of several plantation owners’ estates, Greenwood Great House. Built in 1780 by the cousin of the poet Elizabeth Barrett Browning, Greenwood is now a museum that shows how the owners lived and the slaves suffered – exhibits include the grotesque, mantraps shaped like a shark’s mouth that were used to catch runaways.

Greenwood tells one side of the story, the Maroon community of Flagstaff another. Its inhabitants are known as Maroons – from the Spanish for “fugitives”. There’s a popular theory that the Maroons hold the key to understanding Jamaica’s great sprinting success. They were the slaves who escaped to the mountaintop communities of Cockpit Country; the toughest of the tough, in other words. There’s a theory that they hold the key to understanding Jamaica’s sprinting success, though as Ian Thomson writes in The Dead Yard: “Much romantic nonsense has been written about the Maroons.”

Flagstaff is at the end of a steep zigzag road that in places is more rubble than tarmac. When I climb out of the car, I am approached by locals offering help and keen to share information. Michael Drizzle, a spry man who introduces himself as the local Maroons chief, rents his own place out on Airbnb and offers guided tours to the heritage walking trails.

It’s funny, I think, as Drizzle leads me through the village: the Maroons were slaves who escaped to inhospitable Cockpit Country to flee the Brits. Now their ancestors are slowly building the infrastructure – information boards, walks – to attract tourists. The facilities are modest but, as Grizzle explains, there is great pride in who they are and what they are trying to do.

I sense that pride in all the Maroons I meet in Flagstaff. And also, given that so many visitors to Jamaica seem to be holed up in their resorts or restricted to their cruise liners, I sense their surprise that I have made it.

Virgin Atlantic flies three times a week to Montego Bay from Gatwick from £674 return. Neita’s Nest in Kingston has doubles from £105 B&B

• The Bolt Supremacy: Inside Jamaica’s Sprint Factory by Richard Moore is published by Yellow Jersey at £18.99. To order a copy for £15.19, including UK p&p, see bookshop.theguardian.com