The compact old house with its worn red brick and 300 acres of estate, tucked away beneath the Marlborough Downs, is an unlikely host for such a bold architectural experiment. This is one of those relatively modest English country houses - by no means a palace, rather Jane Austen-ish. The hall table sports a bowlful of pruning tools and the hats and coats on the pegs are practical rather than posh. But the residents are no equivalent of Austen's Bennett family. These are the Keswicks, and they are international traders.

Various Keswicks help run the famous, venerable and bewilderingly diversified Scotttish-run Far Eastern business conglomerate Jardines, which dates back to the 1830s and was instrumental in the founding of Hong Kong and the opening-up of China to Western trade. Henry Keswick, the owner here, is chairman. Meanwhile his wife Tessa Keswick is deputy chairman of the Centre for Policy Studies think-tank, having for years been its director. Another member of the family, the late Maggie Keswick Jencks, had founded the architecturally-ambitious series of Maggie's cancer-support centres. So when they started to ponder an addition to the house they had lived in since 1975, the Wiltshire Keswicks were never going to be orthodox.

Tessa Keswick sums it all up: "Here we are, Scots in the English landscape by virtue of China". We are sitting in her little drawing-room off the hall, which commands a superb view down the gardens to the Pei pavilion in the distance. The 1920s landscaping - part of an earlier expansion of the 1740 house by the wonderfully eccentric architect Clough Williams-Ellis, creator of the fantasy Welsh village of Portmeirion - had terminated the view with one inadequate Lombardy poplar tree. "It's a wonderful landscape, but I always felt it was incomplete, somehow," she says. "And I felt that if we were to do something important in that space, it should be something meaningful for Henry and me."

The 18th century was a time when "Chinoiserie" was in fashion and many a landowner dabbled in Chinese rooms or follies. Why not update that thinking for a 21st century pavilion? A pavilion, moreover, designed by the most famous Chinese-born architect in the world? The Keswicks had visited and admired Pei's masterly Miho museum, set high in the mountains of Japan. A friend in New York knew him. So the introduction was made, and the octogenarian architect called Ieoh Ming but known to all as "I.M", born in Suzhou in 1917, made the first of several visits to deepest rural Wiltshire. One day he arrived with a model in a box. That was it: almost nothing needed to be changed, says Keswick. The model and drawings have now been given to the RIBA Architecture Gallery at the Victoria and Albert Museum.

So we put on our coats and boots and march off down the avenue to see it. At Tessa Keswick's insistence, it is anything but the old idea of a folly that is cold, draughty, musty and not much use except for staring at. In contrast, this is a fully-functioning, properly heated, year-round pavilion that is close to being self-sufficient. It consists of a glass entrance with sliding electric doors, whence a grand staircase rises to a huge living room perched up at first-floor level. This makes it a bit like a tree-house, in fact a bit like a tree, where the ground floor, made of white concrete is like a thick trunk while the main floor above, roofed with a delicate steel and glass structure, spreads out like a tree canopy. Why is it done this way? Because you have to be up in the air a bit in order not to lose the all-important view back to the house. "It's incredibly peaceful," says Keswick, "somewhere to be away from the house and the hubbub of family life. We often have lunch there, or write." They also entertain there: at weekends it has become the main reception room of the estate.

The stair divides to arrive in the centre of the space, so four living areas are laid out around it beneath the complex glass roof with its delicate sun-shading grillage of woodgrained aluminium. So it is four rooms in one really, all floored in oak. There is a lounging area that looks backs towards the house: a television area: a dining section with its own granite bar counter; a work area with a little oval desk. The inevitable Anthony Gormley sculpture is there - is there a wealthy landowner of taste in Britain who does not possess at least one Gormley? Meanwhile downstairs there are lavatories, a fully-fitted kitchen and storage. Sunk into the ground is a plant room for all the heating, ventilation and security equipment. This is far from the kind of 18th century folly where you used to keep a resident hermit in picturesque squalor to amuse your guests.

It is every bit as geometrically strict as you would expect from Pei - a cruciform base supporting a projecting octagonal main floor, with a roof that turns from octagonal to square as it nears its peak. This carries echoes of classically-symmetrical buildings such as a Palladian Temple of the Four Winds - you can slide open four of the glass walls, for instance, leaving waist-height glass balustrades instead. The pavilion also inevitably has a slight touch of the pagoda about it: Pei has got a bit playful in his old age. Inside, alabaster light fittings are placed on a delicate square steel gantry laid out on the plan of the stairwell below, with a cluster of four lights at each cardinal point. What could be a rather austere space is softened by the rather jazzy furnishings by interior designer John Stefanidis. You flop into those big chairs and sofas and just relax, taking in the pastoral views (with sheep) through the enormous, shop-window-sized glass walls. You half-expect a Michael Nyman soundrack to be churning in the background, though Tessa Keswick chooses Willie Nelson instead. Country and Western in Wiltshire? Like the building itself, it works.

It was by no means a doddle to build, and obviously not cheap: an undisclosed number of millions (I reckon somewhere between two and three) were involved to make it as perfect as possible. At one point Pei sent over one of his men from the Louvre to check up on progress. A richly comic scene ensued of bucolic building workers failing to understand a word of French. Luckily the Keswicks employed a trusted Wiltshire firm of architects, Digby Rowsell Associates, to Anglicise everything - not least dealing with the fact that American-based architects such as Pei do not work in metric.

So Pei's only British building, one of his smallest and presumably one of his last, also turns out to be one of his most interesting. It is almost bathetic to report that it is shortlisted for a Georgian society award for "Best new building in a Georgian setting". It should win an entirely separate award for "Most Enlightened and Eccentric Piece of Architectural Patronage in Rural England". Because it's daft, really. But gloriously so.