The Clay Street Center was designed by Julia Morgan for the YWCA. In her design, Morgan mixed elements of traditional Chinese and Tuscan architecture.

The building is much larger than it appears from the Powell Street frontage. It contains ninety-seven federally subsidized, apartments for low-income elderly and disabled persons, operated as The YWCA Apartments.

The Clay Street portion of the building houses the museum and library of The Chinese Historical Society of America which is open to the public.