The Caleb Pusey House  (15 Race Street) is the earliest surviving English-built house in Pennsylvania, and the only dwelling left in America in which it is known that William Penn visited upon at least one occasion.
Born in Berkshire, England in 1651, Caleb Pusey was a maker of lasts for shaping shoes. In 1681 when plans were being made for the colonization of Pennsylvania, a number of Quaker investors put up money to establish a mill in Upland for the "grinding of corn and the sawing of boards." Caleb Pusey, given one share, was to manage the mill and be its agent. Richard Townsend, an experienced builder and miller, also received shares and was to erect the mill with framed parts sent out from London and to build the miller house. Charles Ashcom's 1683 map of Chester County shows that Caleb Pusey's house was standing by that time.
In 1708, having become so involved in the affairs of the Province and Chester County, Pusey sold his shares in the mill. Among other public services he performed, he was a member of the Pennsylvania Assembly and the Executive Council. Late in life Pusey moved to Marlborough Township where he died in 1727.