The first phase of building at Hever was in about 1270, and the design of the castle reflects the defensive architecture of that time. The oldest part of the castle is the outer fortified wall and the massive gatehouse. To get into the castle, you had to cross the moat by a wooden drawbridge and then pass through the gatehouse with the large wooden doors and portcullis.

In the latter half of the 15th century, the castle was bought, along with Blickling Hall (in Norfolk) by Geoffrey Bullen. Bullen came from Norfolk and began his career as an apprentice mercer. Eventually, he had acquired a large personal fortune, became Lord Mayor of London and was knighted. The Bullen family added to the castle, building the apartments inside the fortified walls. Since castles weren't used much for defense by the 16th century, their additions were comfortable living spaces.

Sir Geoffrey's grandson, Thomas, was born in 1477 and in 1498, he married the daughter of the Duke of Norfolk, Elizabeth Howard. Thomas and Elizabeth had three children which survived childhood: George, Anne and Mary. Some believe that Anne was born at Hever, perhaps in 1501, although the evidence for this is sketchy at best.

After spending time at the court of France and Anne returned to England and lived at Hever until she became a Lady-in-Waiting to Queen Catherine of Aragon, the wife of Henry VIII. After Anne's "affair" with Henry Percy, she was sent from Court to once again take up residence at Hever. By this time, King Henry had had an affair with Mary, Anne's sister. At one point the King noticed Anne herself, and began to visit her at Hever.

Upon Sir Thomas' death, Hever Castle went to the Crown and in 1540, it was given to Anne of Cleves in her divorce from Henry VIII. Anne owned the Castle until her death in 1557.