Staying at La Célibataire is a great tease for those of a republican persuasion. The old mill house – and the surrounding cottages that together form Le Moulin de la Tuilerie – were, from 1952 till 1972, the weekend retreat of the Duke and Duchess of Windsor. In those days it was half an hour's drive from their main house in the Bois de Boulogne.

And, while none of the duchess's hectic decorative schemes survives, the Landmark Trust – which specialises in caring for endangered historic buildings and letting them as holiday homes – has ensured that it is difficult to forget them. The faces of the former king and his spouse leer, in the form of blown-up framed photographs, from almost every wall of La Célibataire, the sweet little guest cottage in which the artist, photographer and wit Cecil Beaton is said to have stayed when he visited the couple.

When my boyfriend disappeared into the downstairs bathroom, there came an excitable cry of, "This is definitely Cecil Beaton's hip-bath!" But then there was silence, and a gloomy eventual exit. "There's even a picture of them above the lavatory," he said. "Must we be confronted with the worst excesses of Britain's constitutional arrangement from every wall? Perhaps we could drape a tea towel over them, or something."

Though the Landmark Trust cares for many historic houses in Britain – and indeed has an apartment to rent on the Spanish Steps in Rome – Le Moulin de la Tuilerie is the first property it has taken on in France. There are plans under way to restore, and let out, further buildings in Britanny and near La Rochelle (the latter property, ironically enough, is a 19th-century fort built as a defence against the British).

Le Moulin de la Tuilerie is on the edge of a little town called Gif-sur-Yvette. About 35km south-west of Paris, Gif lies towards the end of the RER commuter line into the capital, in the Chevreuse valley. Its other distinction, aside from the duke and duchess, is that the artist Fernand Léger lived here – he was still alive, residing in a handsome village house down the road, when the Windsors bought the Moulin from the chic French illustrator Drian. History does not relate what the elderly cubist painter thought of the exiled former Edward VIII and his wife.

Gif is neither quite suburbia nor quite deep countryside – commuter-belt Surrey comes to mind – though when we took a walk in the Bois d'Aigrefoin to the west of the mill a pair of deer leapt and darted in front of us, and at night we heard owls from our roomy bedroom in the eaves. The Moulin itself is a delight: a cluster of mostly 18th-century buildings, handsome without being grand, and not quite what you'd expect, given the duke and duchess's well-known taste for the high life, and the splendour of their Paris residence, which had previously been inhabited by Charles de Gaulle.

La Célibataire (or bachelor's quarters, though it sleeps two) is the smallest of the Landmark Trust's three holiday lets here: La Maison des Amis sleeps four and the main house, Le Moulin, sleeps 11. Behind the buildings, running down to the little river Mérantaise, is the skeleton of the garden that the great landscape designer Russell Page created with the duke. I say skeleton, for the chest-deep pool of blue delphiniums and pink stocks that the duke nurtured is long gone, though Page's layout remains, and it is a lovely spot.

This garden was the former king's passion, and he liked nothing better than getting on his tweeds and having a good dig, bullying his guests into helping him lay a stone woodland path, or screaming orders in German to his Alsatian gardeners. (He never got his French up to scratch, but the family mother tongue was in excellent nick.)

Opening on to the lawn is a great barn of a room, which we peered into. Now rather bare, it was once where the duke and duchess foregathered with all their weekend guests, and you couldn't move for dainty tables and chairs, coffee tables made out of regimental drums, and elaborate knick-knackery of every kind, the more vulgar the better. Photographs of the decor show that the pièce de résistance was a carpet, designed by the duchess herself, in a particularly virulent, migraine-inducing shade of swirling emerald.

"I wanted to have a fling with rich, bright colours," she told a newspaper of Le Moulin's décor. "Every house should have a theme: then the decoration becomes something like a musical composition; each room carries the theme but with variations of mood and pace." Whatever the theme was, it found its apogee in the carpet of the drawing room in the main house: vermilion with a thick tartan ribbon detail writhing across it like a python in its death throes. That duchess worked a Schiaparelli frock and an expensive jewel with aplomb, but she let herself down with the decoration of Le Moulin. (May I reassure you that the Landmark Trust has decorated it in unexceptionable sober taste – with the exception of those photographs.)

The fact that the town of Gif is perfectly ordinary has its advantages. We visited the Sunday morning market, between the mairie and the railway station. Tables groaned with delights of all kind, from oysters and John Dory to ceps and those gloriously evil-looking purplish fungi, trompettes de la mort. "It's the only market in France not full of braying English people," said my boyfriend. (Leaving aside us, of course.)

Those with a car might try undertaking some proper sightseeing in Chartres or Versailles. Without one, we debated walking the 25 minutes to the RER station on the other side of Gif and taking the train to the end of the line where, in the town of Chevreuse, there is a magnificent medieval castle, Château de la Madeleine. Instead we took it in the other direction, and 45 minutes after hopping on it were sauntering happily around the Luxembourg Gardens in central Paris. The RER also takes you direct to the Gare du Nord, so the trip is fantastically easy and quick by Eurostar: with a sharp but not punishing early-morning start we were back in London by 11.30am on Monday after our three-night weekend stay. The canny weekender might consider a Friday-morning train to Gare du Nord, stashing bags in left luggage and lunching in Paris, before taking the RER in the late afternoon out to Gif.

The Windsors also let themselves down badly when it came to putting the property on the market in the late 1960s. Pressed for cash, and eager to maximise the price, they proposed a scheme on the land for 537 dwellings, 560 parking spaces and a tennis club, and pulled every government string they could to see it accepted. Fortunately, the good mayor of Gif stood up against them (vive la république!) and the scheme never came to fruition. No thanks to them, the surrounding area remains green. Even Diana Mosley – who lived a couple of stops up the RER line in Orsay, in a gorgeous little Revolution-era building called Le Temple de la Gloire – thought this was poor form, which is saying something.

The Landmark Trust (whose patron is the Windsors' great-nephew, Prince Charles) is an infinitely more reliable guardian of this pretty plot than the duke and duchess.

• La Célibataire at Le Moulin de la Tuilerie (01628 825925, landmarktrust.org.uk) sleeps two people and costs from £381 for a three-night weekend (Friday-Monday). Eurostar (08432 186186, eurostar.com) fares from London St Pancras to Paris start at £69 return

Charlotte Higgins is the Guardian's chief arts writer