Hartley married Elaine Violet Starkey (died 1914). George & Charlotte sold the business to W.E. Dance and travelled overseas.

Hartley Sargent opened two bakeries in Pitt Street and refreshment rooms in George Street. At this time George and Charlotte become partners in business with their son.