Driving through Lochinver, a village on the far north west coast of Scotland, I stopped to allow a delivery van to pull out from a side road. It was advertising baguettes, brioche and croissants. "Sacré bleu," I cried. "I don't remember that from my childhood."

What I do remember are the trawlers unloading their catches in the port, and the way that flanks of rough granite rise from the water's edge, and the pebble-dashed houses with their steeply shelved roofs sitting among heather, pine and bog myrtle. And I remember the great sky stretching away across the dark islands of the west. What I don't remember is baguettes.

But a revolution has been occurring in the three decades I've been gone. It seems that Lochinver is turning into a foodie paradise, albeit an isolated one. Last year, the Inver Lodge, sitting on the hill above the village, contracted its kitchen out to Albert Roux of Le Gavroche fame. Across the bay is the Albannach, one of Scotland's best restaurants with rooms. On the main road is Lochinver's Larder, famous for its pies. And up the coast is the Kylesku Hotel, offering the traditional, but all too rare, spread of fish, chips, scallops and langoustines.

There always was good food to be had out at the furthest edges, if you knew someone. I recall watching, for the first time, a freshly caught lobster going into the pot, the rest of my family walking away so as not to witness its agonies while I horrified myself by lingering. There was fresh salmon, so plentiful that seasonal workers would insist on a clause in their contracts that they would only have to eat it once a week. And there was venison.

But, other than raw products, it was what the local Spar had to offer: oily cheese, wizened veg and Fray Bentos pies. When my parents brought us here in the summers of the 1970s, driving for an hour across the rain-swept moors from our farm to the exquisite beach at Achmelvich so that we could dive into the chill Atlantic for sea urchins, it was hard crofting country. Lochinver, if it was known for anything, was infamous for the huge articulated fish trucks that would thunder unstoppable and murderous through our home village on their way south, taking all the fresh fish with them.

Now I was back to spend a couple of days with a family friend from all those years ago, Randal Wilson. He was taught to flyfish on the river Inver and now he was back to teach his own children. We put up at the Inver Lodge.

This hotel is a pure representation of the idea that beauty resides on the inside. From the village below it seems beige and unprepossessing, but its luxuries, and the view across the sea from its rooms, are astonishing. On its walls are mementos of the owners of much of the land round about, the Vesteys, who, as it happened, made all their money in those Fray Bentos pies that my father thought were a well rounded – and round – diet.

I arrived at dawn, preparing for a day clambering up the small river Kirkaig, and walked into the breakfast room to see the first shafts of light break across the islands of Soyea and Glas Leac beyond the windows.

The walk was a reminder that I was in the real Highlands, not soft southern spots such as Ardnamurchan and Skye. I climbed through birch, heather and midge up a river that swept in brief darkness through small glinting pools before frothing white again as it hit cataracts. I stopped to briefly fish, letting the fly stream across the surface. To the west, on the horizon, hunched Suilven, a mountain almost vertical on all sides, and which my mother used to refer to as the "witch's back".

At the top of the Kirkaig is a stunning waterfall, perhaps 30ft high. Clambering down to the edge of the pool, I let the fly sweep over the top on the swirling water and there, for a moment, was the slashing take of a silver salmon; the line drawing tight and my heart beating hard, but then it was gone. It was a "Trevor", a Brief Encounter.

That evening, the meal at the Inver Lodge was a delight, if a touch nouvelle for a hotel where many guests will have been up and down mountains. Roux, who directs from afar given his advancing age of 74, has complained about the difficulty of getting local seafood, touching on the tragedy of the west coast where those lorries carry the best out along the single track roads, leaving only, in the Inver Lodge's case, a ballotine of Achiltibuie organic salmon with sorrel sauce. Maybe there are local politics at play, other restaurants trying to push them out, and in time more seafood will arrive on the hotel's booked up tables. I finished with Roux's own tarte au citron, which was a far cry from the butterscotch Angel Delight I could look forward to in 1978.

The following day I stopped by the Lochinver Larder for some pies. This little cafe at the heart of the village has a reputation, but has yet to fully promote itself. Granted, it's not easy selling pies online, photogenically they are hardly the Kate Moss of the culinary world, but piesbypost.co.uk wouldn't have you reaching for a credit card unless you had been there and tasted one of their delicacies.

If it hadn't been raining, we would have eaten them by the river Inver where Randal's eldest daughter, Annabel, caught a salmon. In that moment came the whisper of the past – her joy, her father's delight – one generation returning, just like the fish itself, to the river of its youth. Here Annabel would remember catching a salmon and perhaps someday return to teach her own children to fish.

And who knows, by then, Lochinver may have become a culinary destination to match Bray in Hampshire or Roses in Catalonia, with some next-generation Heston Blumenthal promoting a new gourmet infusion of butterscotch Angel Delight to bring all full circle. Or perhaps we will have just learned to treasure, properly, what we can harvest from the sea.

That night, after only a few hours sleep in one of the Inverlodge's sumptuous rooms, I crept into the 4am dark, disturbing several deer in the car park and almost giving myself a heart attack. What hasn't changed in the last 30 years are the nights. I remembered that darkness well as I avoided animals on the sliver of road south across the high moor, until a full moon appeared as spooky as a lonely eye, and then came dawn and the 7am flight to London.

• Inver Lodge (01571 844496, inverlodgehotel.co.uk) has doubles from £200 B&B, £280 B&B and dinner, based on two sharing. Three-course set menus in the restaurant cost £40. Dinner at the (01571 844407, thealbannach.co.uk) is £55. Pies from Lochinver Larder (piesbypost.co.uk) cost from £2.85 (minimum order 10). A three-course meal at Kylesku Hotel (01971 502231, kyleskuhotel.co.uk) costs £28.50