Governor Edward Nott persuaded the General Assembly to authorize its construction with an act passed October 23, 1705, and building began the following summer. In 1706, the act of the Virginia legislature authorizing the building of the Governor's Palace allocated 635 pounds for the construction of the garden with these instructions, "that a Court-Yard, of dimensions proportionable to the said house, be laid out, levelled and encompassed with a brick wall 4 feet high with the balustrades of wood thereupon, on the said land, and that a Garden of the length of 254 foot and the breadth of 144 foot from out to out, adjoining to the said house, to be laid out and levelled and enclosed with a brick wall, 4 feet high, with ballsutrades of wood upon the said wall, and that handsome gates be made to the said court-yard and garden."
But by 1776, the wooden components of the fences had begun to deteriorate, when note was made in the Virginia Council Journal that they were "Repairing Fodder Houses & paling round the Garden." Twenty five men were appointed "to repair fences of park" in 1777. And it was recorded that "60 foot of plank, 250 nails" were purchased for the task.