Text below was adapted from The Colonial Homes of Philadelphia and Its Neighborhood, 
Solitude was a name not only befitting the former character of the place but also according well with the recluse mood of its builder, John Penn, who went thither to escape the vexings of a perverse and naughty world.
John Penn, "the poet," was a grandson of the Founder and a son of Thomas Penn and Lady Juliana Fermor, daughter of the Earl of Pomfret. He was born February 23, 1760, and proceeded a Master of Arts from the University of Cambridge in 1779. A scholarly man, he travelled extensively in Europe and became a liberal patron of art, something of a poet, and an idealist.
Nervous, near-sighted, of an ardent temperament, he was inclined to be an enthusiastic American. He came over to look after the Proprietary interests in Pennsylvania in 1788 and lived at first in Philadelphia at Sixth and Market Streets. He soon discovered, however, that the State was not disposed to honour his claims, made under hereditary rights, and so decided to remain an Englishman, concluding that the people of this country were not lovers of justice.
He lived here for four years, nevertheless, and purchased for six hundred pounds fifteen acres of the high, west bank of the Schuylkill River where the Zoological Gardens now are. Here, in 1785, he erected his two-story box which he called the "Solitude" after a lodge belonging to the Duke of Wurtemburg.
The house was literally a box, foursquare, twenty-six feet in each direction. Extending entirely across the Schuylkill front was a large parlour from whose windows, opening on a portico, there was a fine view both up and down the river. From here Penn could see the ramparts on the once-wooded Fairmount on the farther shore, the site of the British entrenchments when Sir William Howe was in the city. All the space on the first floor not occupied by the parlour is given over to a hall, nine feet wide, extending across the whole house. In the southwest corner a stairway, with hand-wrought iron railing, rises to the second floor. In this storey is the library, a room about fifteen feet square, with bookcases built into the walls. On the shelves were about six hundred volumes, in which number the classics and English poets were largely represented. To the north of the library is a small bedroom connecting with another bedroom in the centre of the house. In his own room was an alcove for his hours of rest and a secret door by which he shut himself from intrusive friends. On the third floor are several more bedrooms and the roof rises in a hip broken by two dormers. The cellars are deep and roomy for wine, and an underground passage communicates with the kitchen built separately about twenty-five feet distant from the rest of the house. Altogether Solitude made a most comfortable and convenient establishment for a bachelor of quiet tastes.
John Penn loved solitude and spent days in reading his own poems, sitting in his sunny sitting-room, dreaming the summer days away in the companionship of Dante, Chaucer, Petrarch, Tasso, and Anacreon. The stucco work on ceiling and cornice in this room is very beautiful and was brought from England. The chair rail and sub-base are of carved wood. A poem published in London in 1801 gives a view of "The Solitude" with a picture showing a favourite white dove flying close along the lawn, whose death his verses deplore:
Thine, oft I said (nor hoped so near thy end),
Are all things round, the grove, the cloudless sky;
While cheers the enlivening sky, sport and enjoy;
Thine are yon oaks that o'er the stream impend,
And rocks that, as I stray with musing eye,
Or wander from the shed, can never cloy.
It is said that John Penn planted every tree about the house and there are few primeval ones remaining. He had pleasant neighbours. In fine weather the good fellows of the "State in Schuylkill" met at the "Castle" on the Warner farm just north of him on the other side of the point where the Girard Avenue bridge now touches the western shore. His cousin, Governour John Penn, lived at "Lansdowne" just above and farther on was Judge Peters at Belmont.
He seems to have been friendly with these and with most of the best citizens, gay parties coming to his place in boats to spend the week-ends. Washington spent the day with him during the sitting of the Constitutional Convention in Philadelphia.
Here he lived in sweet peace until 1788, when he returned to England and suddenly developed an interest in worldly affairs, erecting a handsome residence at Stoke. He became sheriff of Bucks in 1798, member of Parliament in 1802, and was the royal Governour of the island of Portland in Dorset from 1805 for many years. Cambridge made him an LL.D. in 1811, and he also became lieutenant colonel of the First Troop of the First Regiment Royal Bucks Yeomanry.
While he courted only the muses in the wilderness of the Schuylkill, he formed in his declining years the "Outinian Society," whose purpose it was to encourage young men and young women to enter wedlock. This matrimonial society sent out a blank to be filled in under fifty-one different headings describing the eligible parties. It was called "The True Friend, or a Table showing the Exact Situation in Life and Personal Qualities of Known Marriageable Ladies." Finally, Mr. Penn's social benevolence shifted to the promotion of an invention of lamp labels for street corners and an improved breakfast waiter. He was indeed a many-sided man.
Despite his efforts to land others in the holy estate of matrimony, he very inconsistently died unmarried, June 21, 1834, and the Solitude passed to Granville Penn, his youngest brother, who held it for ten years. It then descended to Granville John Penn, a nephew, who died in March, 1867. Granville John Penn was a great grandson of the Founder and the last private owner of the Solitude.
He came to Philadelphia in 1851, a dapper and well-preserved middle-aged gentleman. The city made much of him, he was lionised by Councils, the Historical Society, and by all who could trace ancestral connection with the Penns in former years. In return for these attentions he gave a grand "Fete Champetre" at the Solitude, with lavishly furnished marquees and a collation to which the quality of the city was invited. This was the last time a Penn was at the Solitude, and it was the last property here of a family that once owned the State. Without a tenant for some years it passed into the ownership of Fairmount Park in 1867 and is now well preserved in its original state as the administration building of the Zoological Society.