I first saw the island of Noirmoutier when I was two weeks' old. I think it's probably safe to say that I didn't fully appreciate it at the time; but I grew to love it as year after year I spent holidays there at my grandparents' cottage. This small, sandy island off the Vendée coast has been a home to my family since just after the war, when my aunt Claudine caught pneumonia and was told that she needed a change of climate. The doctor suggested the Riviera – or possibly the small, and mostly unknown, island of Noirmoutier, whose micro-climate makes it significantly warmer than the nearby coast.

In those days the island was free of tourism, separated from the mainland by a causeway, accessible only at low tide. Now there is a bridge, too, though I still prefer the causeway, with its sense of imminent peril and the basket-shaped balises, wooden towers where anyone surprised by the rising tide can take refuge. The bridge – and the influx of tourism that came with it – has made some changes to the island, but not enough to alter its essential charm and character.

It's an island of dunes and salt-marshes, crisscrossed with little canals that make it a haven for bird life. The beaches are long and sandy, lined with tamarisks, blue thistles, dune pinks and rabbit-tail grass. The houses are typically whitewashed, with red terracotta roofs and hollyhocks. The principal town is Noirmoutier-en-l'Ile, home to a castle and the remains of the monastery from which the island takes its name, but much of the rest of the island is made up of small communities, living from tourism as well as fishing, farming and the cultivation of salt.

Each village has a market, selling fruit and veg, flowers, cheeses, bread, meat, oysters and other shellfish. Each village has its church, its beach, its bakery. Island communities are very close: when I was a child there were still people there who had never left the island. Things have changed a little now, but my favourite haunts are still untouched: the rocky beach of L'Anse Rouge and the sea-pine woods of the Bois de la Chaise, thick with gorse and mimosa. There are stretches of beach that I know so well I could paint them from memory: miles of deserted, pale sand, broken by the occasional windmill, or German blockhouse from the days of the occupation, now tumbling like giant Lego blocks out of the dunes towards the sea.

And of all the places I remember the best, the one to which I return the most is a small, unremarkable beach, bracketed by two long dykes to keep the tidal erosion at bay. This is the view from my grandfather's house, just below a sea wall that floods a dozen times a year, sweeping sand across a narrow road that grandiosely calls itself Boulevard de l'Océan. This it's my favourite place on earth, with its view of the white cliffs of St-Jean-de-Monts on one side, and the Pointe de la Loire on the other. This is where I go with a bottle of Gros-Plant and a cornet of crevettes grises (brown shrimps) to watch the sunset from the beach, to listen to the waves and breathe the pungent sea air – a mixture of seaweed and tidal mud – which to those not used to it can seem a little overripe, but to me is the scent of home.

• Return ferry crossings from Portsmouth to St Malo with Brittany Ferries ) cost from £323 for a car and four passengers. It's about a three-hour drive from there to Noirmoutier. Or fly to Nantes, just over an hour away, with Flybe from several UK airports from £60 return. Gites Noirmoutier ) has two flats and 14 cute white cottages sleeping two to seven from €300 a week

Joanne Harris's new novel, Peaches for Monsieur le Cure (Black Swan £7.99) is out in paperback on 28 March. To buy a copy for £6.39 with free UK p&p go to guardianbookshop.co.uk