Students of the occult tell us that the Atherton House is from time to time haunted by four ghosts: three agressive female spirits who in life had been Dominga de Goñi Atherton, Gertrude Atherton, and Carrie Rousseau and one frail male spirit who had been George Atherton.

The story began in 1834 when Faxon D. Atherton of Massachusetts traveled to Valparaiso, Chile, to seek his fortune. He found his fortune, and in 1860 he purchased six hundred acres of land in San Mateo County to build his estate. Faxon married Dominga de Goñi, member of a prominent Chileno family, and they had seven children including George.

After her husband's death, Dominga built this house and moved to San Francisco with her son George and his wife Gertrude in tow. Both Dominga and Gertrude were strong women who so dominated George that in 1887 he ran away from home to seek his fortune in Chile as his father had done. Poor George had only sailed halfway to Chile when his kidneys failed. The sailors put his body into a barrel of rum to preserve it and shipped the barrel to the Atherton Mansion.

George was dried out and received a proper Christian burial, but he was soon spending his nights knocking on the bedroom doors of his mother and his widow. He became so troublesome that Dominga sold the house and moved out. For the next thirty years, owners came and went, put off by cold spots moving about the house, winds blowing through the rooms, voices in the night, and knocking sounds on doors and in hallways.

Then, in 1923, Carrie Rousseau purchased the house. She lived in the ballroom with her fifty cats until 1974 when, at the age of 93, she joined the three Atherton ghosts.