The builder of the cottage was originally thought to have been Alexander, the 4th Duke of Gordon, a colourful character who died leaving the estate burdened with debts. However historians now suggest it was his son George, the 5th Duke, who was responsible for its commission.

George pursued a successful military career before taking over the estate. He was said to have been highly dedicated in his work improving the castle and its surroundings, even turning down prestigious military commissions to continue his work.

The cottage was created as part of a range of improvements in 1834-5, which included a greenhouse and a porter lodge in the east end of Fochabers.

The unusual design may have stemmed from George’s links with Switzerland. He travelled extensively throughout his military career and later continued visiting Switzerland with his wife, Elizabeth Brodie. He was given the Freedom of the City of Geneva in 1815.

When George died in 1836, the estate passed to the Duke of Richmond and Lennox who later added Gordon to the family name. The Gordon estate remained in their family until 1938 when it became part of the Crown Estate.

Swiss Cottage was inhabited for some time in the later 19th and 20th century by the Reiach family. In 1958, it was sold by the Crown Estate Commissioners to Charles Laing and his wife Margaret, who lived in the cottage until 1995.