On the island at the end of the world, there is an old man who plays a lively fiddle and crafts traditional spinning wheels from driftwood. There are women who knit brightly patterned pullovers worn on Antarctic and Everest expeditions, a weatherman who looks like Zeus, and a mailboat skipper whose idea of a good time is battling huge waves in winter storms.

This hotchpotch of characters live happily on barely five square miles of wind-blown rock in the North Sea, memorably described by Robert Louis Stevenson as "an unhomely, rugged turret-top of submarine sierras". Its coast of cliffs rising sheer from heavy seas was the "wildest and most unpitying" he had seen.

Fair Isle feels like the place at the end of the world. When it battens down under howling gales, or disappears in sea mists, the most remote inhabited island in Britain is a world apart. Yet its rugged beauty draws hundreds of visitors every year. Most come to observe a kaleidoscope of itinerant birds that use the island as a flight-path service station, to rest and refuel on annual migrations. In spring and autumn, the likes of rare great snipe and yellow-breasted bunting can be seen posing in the heather for batteries of telescopic lenses.

Thanks to the feathered invasion the island has acquired something approaching a hotel – until now there were only guesthouses and B&Bs – a new bird observatory lodge that opened last July, accommodating 20 guests and 10 researchers in a style and comfort that Stevenson would have envied.

Communications have also improved since the novelist visited in 1869 with his father, Thomas, a lighthouse engineer, in the Northern Lighthouse Board steamer Pharos. I arrive from "mainland" Shetland on a seven-seat aircraft which drones at barely 300m above the sea, banks low over fearsome cliffs and scrunches to a halt on a grass and gravel runway.

There is an invigorating sense of space and freedom in the patchwork of crofting land and peat moors that supports around 70 people, a scattering of sheep, goats, cows, pigs, ducks and friendly dogs, and countless seabirds nesting noisily on the cliffs. The air is filled with the sounds of the sea, the wind, and the cries of gulls.

It takes about five minutes for the tensions of urban life to ebb away, replaced with wonder at this masterpiece of nature. There is only one (single-track) road on the island, and it is never far from a sea surging against a ship's graveyard of islets and rock stacks like monstrous broken teeth.

After lunch at the lodge, I stroll a mile to the crofters' village. I know nothing of birds, but am captivated by the graceful flight of fulmars gliding around me, apparently for the fun of it, and the sweet trilling of skylarks. Passing the first house, I am welcomed enthusiastically by a couple of young Shetland sheepdogs.

The road forms a loop around the crofts, winding through fertile land bounded by dry-stone walls and down to a Stevenson lighthouse that was one of the last in Britain to be manned. On the way, I pop into Stackhoull Stores, the only shop, where there is a notice advertising the lighthouse keepers' golf course – an "adventurous" six holes, with greens maintained mostly by sheep and the weather. Bring plenty of spare balls.

On the face of it, there isn't much to do on Fair Isle. There are no pubs, restaurants, cinemas, theatres or leisure centres, unless you count a tidal rock pool near the south lighthouse. There is, however, the grandeur of land and sea, ever changing with the weather, and a friendly community of crofters, artists and artisans adept at spinning, knitting, crafting traditional Shetland chairs, and building wooden boats to the design of Viking ancestors. There is a community hall where the islanders display their wares when cruise ships call, and a band – whose members are mostly from one family – that performs regularly at the observatory on fiddle, accordion, guitar, mandolin, flute and any other instrument that comes to hand. Its leader is fiddler and spinning-wheel maker Stuart Thomson, 87, whose wife says: "He couldn't find a band for himself, so he bred one."

The observatory serves scrumptious home cooking, its bar is well stocked with Shetland ales, and it has leather sofas by picture windows for gazing at the scenery, reading or dozing. There is a church with beautiful stained-glass windows inspired by island scenes, and a small museum, where curator Anne Sinclair relates the turbulent times of her forebears, whom she traces back to the 1690s.

When they have the time, locals lead guided walks to talk about crofting life, and the wild flowers, history and archaeology of their island. I opt for a stroll with observatory ranger Carrie Gunn to view a colony of puffins nesting on Malcolm's Head, a promontory sweeping majestically to vertiginous cliffs. Puffins are the stand-up comics of the bird world, strutting around with multi-coloured beaks and bemused expressions, and we are given a command performance as clouds of kittiwakes and guillemots wheel and cry overhead.

Elena Mera-Long, an incomer from Portsmouth who is the resident community nurse, knitter for Fair Isle Crafts and church organist, says it is an exciting place to live. "People say: 'What do you do all day on such a small island?' They must be joking. Everybody on Fair Isle works like beavers, and there's never a dull moment. It's a model for a good community. I doubt if there's a more harmonious one anywhere."

It is a view widely shared by families with island ancestry, and by adventurous spirits drawn by the appeal of remote island life. Jane Wheeler, who arrived with her meteorologist husband Dave after five years in South Georgia, says: "This island is like an extended family. There aren't many places where children are encouraged to talk to strangers."

The children will have plenty of strangers to talk to on 18–19 July, when a flotilla of Tall Ships is due to drop anchor off Fair Isle as part of the Tall Ships Races. This has sparked a revival of knitting, with women clacking needles to produce fishermen's "keps" (hats) for bartering with the crews, as their menfolk did of yore.

For all its community spirit, Fair Isle is a place of lone figures. You see them everywhere – crofters working the land, and birdwatchers etched against the sky on moors and headlands. The Vikings called the place Fridarey, meaning isle of truce, or peace. Today, it still offers solitude and good company in equal measure.