Southampton, in New York state, may be home to Ralph Lauren-clad millionaires, glamorous golf clubs and the Republicans' favourite picnic spot, but I'm here to see the high priestess of punk Patti Smith, The Vaccines, Of Montreal and Graffiti 6. But right now all that can wait as I bathe – like a blissfully tipsy Cleopatra – in a tank of teeny tiny toy pandas.
Slightly confused? I was too at first but surrealism is Freddie Fellowes' (the brains behind Britain's Secret Garden Party) signature party stamp. And having built up his Cambridge-based summer jaunt from a party of 1,000 to 26,000 attendees in just seven years, it's obviously a nifty little formula. But can such British kook fly in the spiritual home of slick, over-sized corporate events? And in an area where the average resident tends to live more like a well-heeled Donald Trump than someone who's up for a three-day bender?
Fellowes admits the chain of events that led to the birth of this bash on this patch of leafy land that belongs to the Shinnecock Indian community was "random".
"They need the money and after years of being treated like shit, they didn't really want Americans to run it," he shrugs. "When we first met, they sat me down and said: "Now you're the party guy: bring us Beyoncé!""
He declined, explaining with so many big successful music festivals already in America, he'd feel more confident running something smaller and boutiquey. And so Escape To New York came into being.
While quirky little "lifestyle events" (where as much attention is given to the quality of the organic cider as to the music line-up) have become the summer norm in the UK, to Americans, the concept is rather peculiar. "But all these bands suck!" tended to be the initial online response to the promotion – which focused on all sorts of fancy affiliations from André Balazs' hip Standard Hotel to Soho House or NYC's trendy Brit eatery du jour, The Fat Radish.
Kicking back, lobster roll in hand, at the mini acoustic stage, I'll admit I was slightly skeptical. The crowd is noticeably thin on the ground with the finger pointing to the high cost of accommodation in the Hamptons and a shortage of normal camping space. The majority of the crowd is glamping in pristine white tipis on a field flanked by two Mercedes-Benz garages (it costs, gulp, $2,250 per person for two days). It's drawn in an outlandish mix of posh English people, Brooklyn trendies and the type of bohemian kids that probably have a very big house nearby.
"We were expecting at least 5,000 people a day here," sighs a friendly, if noticeably bored looking, parking attendant. "On Friday, 1,000 people turned up tops. Most of the guys working the parking lots have just been napping." One guy I meet, who drove seven hours to attend, shakes his head when I ask how he's finding it. "Weird! When the bands are playing, people are just hanging out on rugs and shit," he says, raising his eyebrows in candid concern.
Kevin Balktick, who curated the art installations alongside his co-producer Mark Winkel, is more optimistic. "When things are big, people love to talk about how much better it was when it was small, this is only going to get better. It will be worked out." Mid-way through the day on Saturday, when a little worryingly perhaps, the Portoloos still have hand towels neatly folded by the sink, it emerges that a similar debut festival, Music To Know, scheduled for the next weekend in East Hampton is cancelled due to lack of ticket sales. E2NY offers free entry to all those who have been let down and, by the evening, the pace spectacularly picks up as Edward Sharpe and the Magnetic Zeros hit the main stage.
Winkel and Balktick's quirky installations come to life in the midnight hours too. After a few elderflower sangrias, my initial reservations about that panda tank (it looked like it was for babies) have been totally forgotten as I float around in the weird stuffed teddies, making friends with fellow revellers like it is ancient Rome. Then there's the rose petal dome, which incidentally proves that throwing rose flowers in the face of strangers is a lot more fun than it looks.
It's not until the next day, I recall Balktik saying something that sticks. "The theme of our work is about all about interaction. There is zero per cent chance of meeting anyone in a gallery with white walls."
Opening jitters aside, I suppose E2NY is trying to remove the white walls from America's formulaic music festival scene – and then tie-dye them purple. When I hear Sunday has been cancelled due to "inclement weather", I hope that they can – as they've subsequently announced they will – return next year and get things ironed out.
As Fellowes says: "The Americans have found this completely out of the ordinary. I always see the things that are wrong and want to improve them. E2NY will be back bigger, better, weirder."
I can't help but wonder if 2012's pandas will be living and breathing?
• Escape to New York (escape2ny.com) ran from 5-7 August at Shinnecock Indian Nation