Old Mr Fotis turned my question over in his mind while sipping his morning coffee. Below the veranda some youths had been playing noisily on the harbour wall, but now they all dived into the turquoise sea and set off on the long swim to the rocky island in the bay. It had a fragment of crenellated wall on top of it, the ruins of a Venetian fortress. Fotis watched them go, half-smiling.

"We do seem to attract a lot of writers," said the old man eventually. "But that's a name I don't remember."

"Bruce Chatwin, Baroose Chit-win, Chaatwing." I tried a few variations but none struck a chord. "His ashes are scattered somewhere in the hills."

"No, I never heard of him."

"What about Sir Patrick Leigh Fermor? You must know about him."

I'd first heard of Kardamyli because of Leigh Fermor, who had made the place his home. I'd always hoped we might meet, but then the grand old man of British travel writing had died in June 2011 (leaving the literary world praying that he had finished the final volume of his Time of Gifts trilogy). I'd come to the Mani on a sort of literary homage, hoping to find a little of the magic that had attracted first Leigh Fermor and later Chatwin.

The old man shook his head. "No, I don't think so. There was a writer called Robert. Now he was famous – cured himself of cancer by walking around Crete. [Former South Africa cricketer Bob Crisp wrote of his walk around Crete in the 1970s.] He was very famous."

This felt all wrong. Was I in the right place? How annoying that the locals should raise this unknown above the two giants of travel literature.

Fotis leaned back and shouted in Greek to his wife in the kitchen. She came through, cloth in hand. "Robert Crisp," she said, smiling. "What a wonderful man! So handsome! I remember him sitting up at Dioskouri's taverna drinking and talking with Paddy. They were always laughing."

My ears pricked up. Fotis's face underwent a transformation. "Ah Paddy! That is him – your English writer. Of course, Paddy – or Michali we called him. Yes, Paddy was here for years and years. When I was young we used to say he was a British spy and had a tunnel going out to sea where submarines would come."

It didn't surprise me. Leigh Fermor had been, by all accounts, extremely old school, endlessly curious and an accomplished linguist – all well-known attributes of British spies. He produced a clutch of good books and two classics of the genre, A Time of Gifts and Between the Woods and the Water, detailing his journey as an 18-year-old on foot to Constantinople.

"Did you see a lot of him?" I asked.

Fotis shrugged. "Sometimes. He liked to walk a lot. Now Robert Crisp – I used to see him. What a character!"

"Is Paddy's house still empty?" I persisted. "I heard it was now a museum."

Fotis shook his head. "No, no. He left it to the Benaki Museum in Athens (benaki.gr) and they're supposed to turn it into a writers' centre. My guess is nothing will happen for a while."

I could hardly complain about Greek tardiness: Leigh Fermor himself had taken 78 years over his trilogy, and even then no one seemed very sure if he had completed the task. There was, I decided, only one way to get close to the spirit of these colossi of travel-writing: to walk.

"Which paths did Paddy like best?"

Fotis fetched a map and gave me directions. It was already hot when I left him on his veranda. I could see the youths lazing on the harbour wall again, tired by their long swim. Was this really the time of year for walking?

I headed through the ruined village, as instructed, and found a narrow steep path rising up the hillside. Before too long I came across the stone tomb that locals say is the grave of Castor and Pollux, heavenly twins and brothers to Helen of Troy. Kardamyli is mentioned in Homer's Iliad as one of the seven towns that Agamemnon gave to Achilles.

Sweat was pouring off me now, but I kept going. Scents of thyme and sage rose from the undergrowth. Fotis had said there were lots of snakes up here, but I didn't see any. The views of the bay below, however, were becoming more and more magnificent.

The Mani is the middle finger of the three-pronged southern Peloponnese, a 40-mile long skeletal digit that was almost inaccessible, except by sea, until recently. When Leigh Fermor first came here in 1951, it was by a marathon mountain hike across the Taygetus range, whose slopes seem always to be either burned dry by summer sun, or weighted with winter snow.

The people here were different. For a start they had turned vengeance into a lifelong passion, building war towers to threaten their neighbours and generally making life on a stony mountain even grimmer than it needed to be and clinging to weird atavistic beliefs. No wonder that in the 1950s most of the younger people abandoned it for places not as badly infested with saltwater ghouls and blood-sucking phantoms – Melbourne and Tottenham were particularly popular.

Fotis himself had been one of them, settling in Australia for many years before coming home and opening a hotel. Nowadays some parts of the Mani are thick with holiday homes and development, but Kardamyli remains delightfully quiet and understated, the sort of Greek village where old widows in black sit out every morning watching the world go by.

Having reached a good height on the mountain I started to follow the contours, dipping in and out of the shade of walnut trees and cypress, drinking clear cold water from a spring. Further on I came to the village of Proastio, where Fotis had told me there was a church for every family, the ancestors having been sailors, and very superstitious. At the gorgeous little basilica of Agios Nikolaos in the main street I got the priest to come and unlock the door, revealing a gallery of perfect 17th-century Byzantine murals.

I tried the name Chatwin on him, and wondered how to mime death, cremation and scattering of ashes. But the Orthodox Church does not approve of cremation and his face told me I would not get far.

Returning to Kardamyli by a steep cobbled donkey trail, a kalderimi, I passed Fotis's veranda once again.

"Who was that writer?" he called. "My wife thinks she knows."

Anna came out. "Paddy himself scattered the ashes of a writer friend of his, up in Exochori."

That was where I had just been walking, only higher. Next morning I started much earlier and with a water bottle. Fotis was already up and about when I passed his house.

"Exochori is my home village," he said. "But what you see now is just old people up there. The old Maniati culture is gone. We used to grow silkworms and our mothers made all our underclothes from it. Can you believe it? We were peasants in the most remote part of Europe, but we wore silk."

He tried to give me directions to the church, but it got so confusing that I just pretended to understand and resolved to ask along the way. As it happened, this was a useless strategy since the few old people I bumped into spoke no English, and my phrasebook was inexplicably silent on the important line, "Where are scattered the remains of the travel writer?"

In the end I came across a small white-washed shrine with a view of the sea. There was just room to enter, and inside a votive candle burned on a tray with some fresh flowers. A white dog appeared. I elected to call it the Chatwin Church. After a few minutes of contemplation I set off again, southwards past one of the war towers, a gorgeous forest monastery and finally the unspoilt hamlet of Castania, where the taverna owner marched me into the kitchen, pointed out the various dishes and then served a vast quantity of delicious food with a jug of rough wine. It took several strong coffees to get me moving again for the long tramp home.

Back in Kardamyli late that afternoon, Fotis was keen to hear of my walk, but he scoffed at my description of the Chatwin Church. "No, no! That is not it. Come on – I'll take you there."

"I'm a bit tired."

"Good God, we're not walking! In my car."

Soon we were on a longer, twisting route. Fotis pointed out landmarks and patches of land that his family owned. I asked about his ancestors.

"The Mani was always where people came to hide," he said. "Our family are said to have arrived when Byzantium fell to the Ottomans in 1453."

"They came from Constantinople?"

He nodded. "Our family tradition is that we were clowns for the Byzantine emperor." He smiled. "But I've no idea if that is true."

He slowed the car down. "Look. This is where you turn off the main road, between the school and the cemetery."

We pulled up in the shade of two pine trees then set off walking. We picked our way through some old stone houses, their walls overgrown with vines, their shutters closed.

"Holiday homes now," said Fotis. "Exochori people working in Athens."

Then we were on a grassy rill of land and I could see the church, a tiny Byzantine basilica, its rough stone walls and ancient pantiles crusted with lichens. Laid out before it was a wonderful tranquil panorama of the sea, its surface smooth as a sheet of silk. It was obvious why a traveller would want to come to rest here, overlooking the sea Homer's heroes had sailed.

I stood there for a long time. Fotis was searching for the key to the church, normally left in a crack or niche, but there was no sign of it and we gave up. Back in the car, I asked Fotis to point out the house of Paddy Leigh Fermor and glimpsed a low pantiled roof almost submerged in trees on a crag next to the sea.

"Is there no way to see it?"

He shrugged. "It's all shut up."

"Is there a beach?"

"Yes – a tiny one."

I memorised the spot. When Leigh Fermor came to the Mani he did some impressive wild swimming. To honour his adventurous spirit I felt I should swim around to his house and take a look. So next morning, before the heat of day, I entered the sea by the harbour and swam south down the rocky coast hunting for that tiny beach. I swam for what seemed a long time and had given up and turned back when I saw it: a little shingly beach with a single-storey house above. I swam closer until I could stand in the water.

It was a lovely place: deep verandas and stone walls under a pantile roof. Mosaics of pebbles had been made on a flight of steps. I called out but got no answer. The house was shuttered and quiet as though still in mourning. I waded up the beach and sat at the foot of the pebble path. I could see a colonnade with rooms off it, then a larger living room.

I thought of the years that Leigh Fermor had spent here: by all accounts he was a great host and storyteller. When I'd asked one old lady in the village if she had read any of his books, she'd laughed, "Why would any of us read his books? He told us all the stories himself!"

The last story had been that third volume of his epic walk across Europe, but he had never finished it, perhaps never would have. And now a great peace had descended on the place, a peace I didn't want to disturb. I walked back down to the water and swam out into the bay. Without thinking, I found myself heading for the island of Meropi, the one that those youths had swum to. I would explore the ruins of that Venetian castle.

How to do it
The trip was provided by Sunvil (020-8758 4758, sunvil.co.uk/greece), which has seven nights self-catering in a studio at Liakoto Apartments in Kardamyli, including flights from Gatwick to Kalamata and transfers, from £699pp. Walking tours of the Mani with guide Anna Butcher can be booked through Sunvil, which also has a nine-day all-inclusive walking tour of the Mani for £1,755pp

What to read
Artemis Cooper's biography, Patrick Leigh Fermor, An Adventure, will be published by John Murray on 11 October, price £25. To buy a copy for £20, including shipping, go to guardianbookshop.co.uk. Leigh Fermor never did fully finish writing the third part of his epic walk across pre-second-world-war Europe but John Murray will publish an uncompleted version of the book in 2013

"I felt like staying there forever" – Patrick Leigh Fermor on the Mani peninsula

Beyond the bars of my window the towers descended, their walls blazoned with diagonals of light and shade; and, through a wide gap, castellated villages were poised above the sea on coils of terraces. Through another gap our host's second daughter, wide-hatted and perched on the back of a wooden sledge and grasping three reins, was sliding round and round a threshing floor behind a horse, a mule and a cow – the first cow I had seen in the Mani – all of them linked in a triple yoke. On a bank above this busy stone disc, the rest of the family were flinging wooden shovelfuls of wheat in the air for the grain to fall on outstretched coloured blankets while the husks drifted away. Others shook large sieves. The sun which climbed behind them outlined this group with a rim of gold and each time a winnower sent up his great fan, for long seconds the floating chaff embowered him in a gold mist.
The sun poured into this stone casket through deep embrasures. Dust gyrated along the shafts of sunlight like plankton under a microscope, and the room was full of the aroma of decay. There was a rusty double-barrelled gun in the corner, a couple of dog-eared Orthodox missals on the shelf, and, pinned to the wall above the table, a faded oleograph of King Constantine and Queen Sophia, with King George and the Queen Mother, Olga Feodorovna, smiling with time-dimmed benevolence through wreaths of laurel. Another picture showed King Constantine's entry into re-conquered Salonika at the end of the Balkan war. On a poster, Petro Mavromichalis, the ex-war minister, between a pin-up girl cut-out from the cover of Romantzo and a 1926 calendar for the Be Smart Tailors of Madison Avenue, flashed goodwill from his paper monocle. Across this, in a hand unaccustomed to Latin script, Long live Uncle Truman was painstakingly inscribed. I felt like staying there
for ever.
• Extracted from Mani: Travels in the Southern Peloponnese, by Patrick Leigh Fermor (John Murray, £9.99) which is available from the Guardian Bookshop (guardianbookshop.co.uk) for £7.99, including P&P