Joseph H. Polley, the builder of the house, was born in Whitehall, New York, in 1794.  He moved to Louisiana with his family when quite young, and served as a teamster in the army in the War of 1812.  He was one of the first twenty-two immigrants to come to Stephen F. Austin's Colony in 1821.  This was known as " Austin's Old 300".  Refering to the 300 families that came with Austin to settle Texas.

Polley was appointed by Stephen F. Austin the first sheriff of the colony.  In the "Runaway Scrape", Polley escorted women and children out of harms way when Santa Anna's troops came into central Texas during the revolution of 1836.

In 1847 Polley moved from his home on the Brazos, which he had called "Whitehall" in honor of his birthplace.  He selected a site for his new home on a small hill overlooking the Cibolo Creek about two miles north of old Sutherland Springs, in what was then Guadalupe County and is now Wilson County.

Prior to the Civil War Polley accumulated vast holdings of land and cattle.  In 1859 and 1860 he owned 150,000 head of cattle, more that any man in Texas with the exception of the King Ranch.  His holdings were scattered from Fort Bend County to Marble Falls, from Corpus Christi to Austin and south to the Rio Grande.