It was a chance encounter. I was writing a book about sugar (Sugar: The Grass that Changed the World) when I came across Fanny Kemble's story. She was a beautiful young 19th-century actress who travelled from London to America to make her fortune. There she met and wed Pierce Butler in 1834, only to discover that he owned a slave plantation in the deep south. Kemble was forced to live on the plantation and see the abject misery of the slaves at first hand.

Her story became the subject of my latest novel, Sugar Island – but to write the book, I needed to see the former plantation muself.

The Butler estate was on St Simons Island, near Savannah, Georgia, and is part of an African-American heritage trail. Even so, a former slave plantation might not, at first, seem the ideal location for a holiday. Although the entire island was once devoted to the cultivation of cotton and rice, it is now one of the wealthiest zip codes in America. St Simons is part of a chain of small islands called the Golden Isles, all famous for their seafood, beaches and marine wildlife.

St Simons is flat and small – just short of 12 miles; I hired a bike and cycled to Hampton Point at the far end, where the Butler plantation had been. As I biked round Butler Road and Fanny Kemble Street, a suburb of enormous houses set back from the road in between live oaks, the south's famous giant evergreens dripping with Spanish moss, I saw something curious. A ruin on the lawn in front of a holiday condo. There, on someone's front garden, without so much as a plaque, crumbling to dust, trees and creepers growing through them, were the remains of Butler's slave cabins.

You can see other memorials to slave life with marginally more ease: the remains of a hospital lie in the grounds of the Sea Island Golf Club, and at Retreat Plantation, King's Retreat, two slave cabins, now a gift shop, once housed two families of up to 20 slaves. It's perhaps natural that the St Simons islanders would prefer to concentrate on other aspects of their history, such as Fort Frederica (nps.gov/fofr/index.htm) where General James Oglethorpe stationed his troops and defeated the Spanish in 1742 in what became known as the Battle of the Bloody Marsh.

St Simons today is a chilled place. The white sandy beaches are beautiful, and there are many watery pursuits, from fishing and boating to diving and snorkelling. I spent time hanging out in cafes (seafood is obviously a speciality) in the central square near the disused lighthouse (now a museum) and browsing small galleries.

In my novel, Sugar Island is an island off St Simons Island, where more slaves were kept. The island really exists – it's called Little St Simons and is now a 10,000-acre nature reserve and resort. It's laced with trails and has seven miles of secluded beach. Only 32 guests are allowed to stay at a time, although you can arrange day trips.

The island is also renowned for its spectacular birdwatching. Having so few people around helps when you're trying to see some of the 280 species of birds that live there. I saw little blue herons fishing in the marshes, a flock of spoonbills, their feathers the colour of peeled prawns, Carolina wrens chattering through the maritime forest and, on the shore, flocks of tiny sanderlings and terns.

My guide, Scott Coleman, showed me the spot where Kemble had rowed to Little St Simons. Still there were the remains of a cabin that had been home to slaves who had offered Kemble help and refreshment. We followed her path through the dense woodland to the wild beach. Slavery in the southern US was not to end until many years later, in 1865, but Kemble stood here and thought how poignant it was that her slaves were unaware of how close they were to the (British) colonies of the Caribbean, where their brothers and sisters were living in relative freedom.

• Apartments and cottages through hodnettcooper.com cost from around $150 a night. Pretty Ocean Lodge (+1 912 291 4300, oceanlodgessi.com) has doubles from $299. Little St Simons (+1 912 638 7472, littlestsimonsisland.com) has doubles from $600 including food and activities. More information about the islands from goldenisles.com. Fly into Savannah or Jacksonville, Florida

Sanjida O'Connell's novel Sugar Island, is published by John Murray (£8.99)