Snow is falling as we walk past the RSPB visitor centre at Bempton Cliffs and head towards the horizon. There is no preparation for what myself and my son Conor are about to experience. The hinterland is dull and featureless, the fields monotonous squares of stubble and sheep-nibbled pasture. There is no sign of what lies in store until the last minute.

Then we are teetering above the North Sea on top of a 100-metre tall, white cliff that undulates and swings south, leaving monumental stacks offshore, some tenuously connected by arches, all of it riddled with caves and blowholes. The noise is tremendous: pounding swells and shrieking gulls.

This is northern Britain's only chalk sea cliff, a 10-mile long tilt of rock that takes a giant bite out of the North Sea, culminating in Flamborough Head, one of Britain's greatest coastal sites. Here in the 1860s the destruction of seabirds led to an outcry and the world's first ornithological conservation legislation: the problem was that in fog the bird noise had helped to alert mariners to the dangers of the cliffs.

The birds, however, have become a major attraction. There are an estimated 200,000 in May, the peak of the breeding season when the sea fills up with sand eels and causes a feeding frenzy among gannets and puffins. Even whales get in on the act: Flamborough is said to be one of the best places to see whales in the UK, though not in winter.

Ian, the taxi driver who brought us to Bempton from Bridlington station, is an ex-commercial fisherman and dubious about the whale-watching. "I've seen Minkes about five miles out," he says. "Closer in, you can spot porpoise, sometimes dolphin. What I like is the gannets. There's nowhere else on mainland England you can watch them."

RSPB warden David Aitken is out on the cliffs before us, checking what has come in overnight. "In winter, we can get rarities like the red-breasted flycatcher and red-flanked bluetail," he says, "Then there might be a mega-bird, a brown flycatcher or an olivaceous warbler, the sort that causes a big twitch." In other words, a creature so unusual that birdwatchers from all over the UK will hightail it over to East Yorkshire.

"But what are we to look for today?" I ask.

"In winter short-eared owls, maybe, then there's fieldfares, bramblings and snow buntings."

With the wind whipping in and a bank of snow looming in the south, it doesn't look hopeful for sightings, but we stay optimistic as we take the cliff path south. After a mile, we come across a huge earthwork, a great wrinkle of grassy soil that the map shows running across the base of Flamborough Head peninsula.

Although referred to as Dane's Dyke, this battlement is thought to predate the Viking invasions and go back to the bronze age. A little further on, we find that the path has fallen off the cliff; it's not uncommon in these eroded parts, and we have to make a detour.

Back on the coastal route we reach Thornwick Bay and scramble down to a beach of white boulders and sand. At low tide, we explore the sea caves and reach little niches and coves. Later, down in the bay itself, we fossick through the pools. Crabs scuttle away and fish dart into the shelter of glossy seaweeds, pursued, with lots of splashing, by my dog, Wilf. It's not uncommon to get bigger fish trapped in these pools, even sea trout.

On the south side, there are big caves to explore – good in summer at low tide in a wetsuit – but in winter you need to be careful as there can be rock falls from above. We walk up past the old second world war machine-gun post and around the headland to North Bay. There is a slipway and a handful of traditionally built Yorkshire cobbles, hefty wooden boats designed for severe weather. In the old lifeboat station, I find fisherman Mike Emmerson and his son Matthew working on their cobble ready for spring. "They're good boats for this area," says Mike, standing beside what looks to me like a massive, double-keeled wooden galleon. "We mainly go crabbing, or take people out fishing or birdwatching."

The lifeboat station holds a century of history, the walls plastered with photos and lists of shipping attended to in distress, some of them with an Emmerson on duty.

"At low tide in summer you can get out to the remains of one wreck," Matthew tells me, pointing out the door to the cliff corner. "That's where the Rosa ran aground in 1930. There's just the boilers left now."

Beyond North Bay, you soon spot the lighthouse that helps prevent modern calamities. We leave the Emmersons and head south, again. The plan is to find a way down on to the rocks and follow the beach to Bridlington before the tide comes in, but the OS map is coy about such information. We pass Dane's Dyke for the second time and, deciding to wander up this astonishing handmade ravine, we find the season's first flush of snowdrops and aconites appearing on the wooded slopes.

Back on the cliff path, we follow steps down to the beach, a place littered with smooth white stones that have had neat holes carved into them by the sea. From here, it is a mile stroll to Bridlington, keeping an eye on the incoming tide. Bridlington is one of those seaside towns that gets unfairly rubbished. The glory years brought on by the explosion in Victorian and Edwardian holidaymakers are over and there are many signs of economic failure but there is also a busy harbour with a great fish shop. And there is our final goal: a gem of a cafe near the war memorial.

Katy Wheelwright started the Little Organic Bakery (12 Wellington Road, 07853 303049, open Tue-Sat 10am-3pm) in late 2011, since when it has gathered a serious following, not least for her spicy beanburgers and nut flapjacks. "It was my dream to open a cafe," says Katie, "Everything is homemade, vegetarian and organic."

The café is notable for the quantity and quality of local art on display. What about that most famous local artist, David Hockney?

Katy laughs. "We do own a Hockney."

Really?

"Yes, it's by Paul – his brother."

We sit in the window as the light fades. The walk, all 11 miles of it, has been just about perfect with huge skies, scouring wind and numerous meetings. Now we are ready to tuck into our reward.

• RSPB Bempton Cliffs, 01262 851779, rspb.org.uk/bemptoncliffs