As someone besotted with the Mitford sisters, I'm never short of something to read: their elopements, affairs and extreme politics have kept many a biographer in business since their 1930s heyday. But the recent publication of Debo Mitford's Wait for Me! – she was the only sister not yet to have written her memoirs – made me long to get closer to the stories. Nancy's novels, Decca's letters and those beautiful pictures of the incomparably elegant Diana (in the days before she was revealed as an arch fascist) had all left indelible images in my mind; now I wanted to see the originals.

So I set out to explore the places where Debo – now the Dowager Duchess of Devonshire – has spent her extraordinary life. First stop, Asthall Manor, a sprawling Jacobean pile in Oxfordshire where the Mitfords were living when Debo was born, and home to the eccentric country life that Nancy immortalised in her books – of Uncle Matthew, the explosive pater familias who hunts his own children with bloodhounds, and daughters who convene meetings in a linen cupboard and speak their own private language.

Its well-dressed gardens, bordered by the river Windrush, are opened once every two years for On Form, a modern sculpture exhibition. But next year, for the first time, you'll be able to stay here, because the annexe – a barn converted by Lord Redesdale (Debo's father, and the real Uncle Matthew) to keep his exhausting offspring out of his way – is being opened up for holiday lets. The apartment, which sleeps up to five, still bears the children's handiwork, from the mural Nancy drew in her bedroom to the baffling room names ("Pity Me") painted over each door – although it's been usefully updated with a kitchen and the downstairs ballroom now houses yoga classes.

Five minutes down the road is Swinbrook, the village the family relocated to in 1926 (and which Nancy hated so much she dubbed it Swine Brook). Debo alone loved the place; she now owns the local pub, the Swan Inn, and writes fondly in her memoirs of "the formidable Mrs Bunce" who "kept strict order" there. Mrs Bunce would, no doubt, find the work easier today; I'm not sure how much trouble you're likely to see in a restaurant that serves a starter of spiced soft-shell crab.

I stay in the Duchess's suite at the inn which is, once a month, Debo's own room; complete with dressing table, writing desk and rolltop bath, it's just as elegant as the name suggests. Swinbrook is a tiny village and most of the places Debo writes about (the post office where she bought penny sweets, the blacksmith's forge) are long gone. But its church, where she and her sisters used to lick the pews and giggle irreverently, is well worth the visit, not only for the curious stone effigies of the Fettiplace family, the erstwhile lords of the village, who lie propped up on their hands like Burt Reynolds sprawled on a bearskin rug, but also the more modest graves of Nancy, Diana and Pamela.

With the sun out, the top down, and a thick rug wrapped round my knees, I throw my little car into the Cotswolds lanes. Wanting only a pair of driving goggles to make me feel more Mitford-y, I speed towards Batsford Park, the ancestral seat that Lord Redesdale was forced to sell – damn those death duties! – shortly before Debo was born. Like Asthall, the house remains in private hands, but the land around it has been transformed into an arboretum and falconry centre. As buzzards wheel overhead, the mock-Tudor bulk of Batsford glowers between the trees.

But however imperious it tries to appear, it can't compare with my final destination. Growing up, Debo suffered the ignominies of being a youngest child – her siblings used to taunt: "Who's the least important person in the room? YOU" – but she was the only one of the eligible, Honourable Mitford sisters to make a "good" and lasting marriage. Debo became a duchess in 1950, and it was under her aegis that Chatsworth, the resplendent, 500-year-old seat of the Devonshires, was restored to its former glory and became the major visitor attraction it is today.

Unlike many stately homes, Chatsworth doesn't close for the winter, and while I've missed the special Deborah Devonshire exhibition that ran this year to celebrate her 90th birthday, there's plenty of her legacy on show in the immaculately curated state rooms, the themed Christmas openings, and the portraits that line the sketch galleries (not to mention the farm, which was one of her pet projects: Debo is, rather famously, a great lover of chickens).

Without a title, I'm unlikely to receive an invite to stay in the great house, but I do manage the next best thing – a weekend on the estate with a few friends. When the people who work here talk wishfully of a weekend in Swiss Cottage, they're not talking about a dreary part of London but the park's prize accommodation: a picturesque chalet which Joseph Paxton designed in 1839 and set, in a typical coup de théâtre, on its very own lake. There's even a rowing boat for you and your friends to indulge your Merchant Ivory fantasies (or, if one of your number falls in, your Mr Darcy fantasies. We achieved both).

The cottage is tastefully furnished, by the current duchess no less, with antiques and pictures from Chatsworth, and we're starting to get a taste for the Mitford lifestyle. Pining for the era when people still dressed for dinner, we ask Alan Hill, chef patron of the Devonshire Arms, the estate's award-winning restaurant, to come to the cottage and cook for us, while we stand around louchely in the study mixing gimlets and declaiming "Here's mud in your eye!" until the dinner gong.

Hill serves up a peerless meal, from a meltingly lovely goat's cheese soufflé to a tangy lamb tagine, topped off with a lemon tart he stole from his grandma (the recipe, not the actual dish). We end the evening in the drawing room, sipping scotch in front of a log fire, and talk of Debo, now living in the Old Vicarage in Edensor, just a couple of miles away. "I find 20 roses just as interesting as 200," she writes contentedly of her new home. We raise a glass to her, and her roses.

Essentials

The annexe at Ashtall Manor is available to let from February (bookings@onformsculpture.co.uk). B&B at the Swan Inn, Swinbrook, costs from £120 for a double room (01993 823 339; theswanswinbrook.co.uk). Swiss Cottage, sleeps six, from £1,289 for seven nights (01246 565 379; chatsworth.org). A Russian Christmas at Chatsworth runs to 23 December. Alan Hill is chef patron of the Devonshire Arms (01629 733 259; devonshirebeeley.co.uk)