1821 he moved to Litchfield, Connecticut

the Baker family moved to the Mississippi Territory and by 1811 they had settled at Oaklawn Plantation in St. Mary Parish


Baker owned three plantations: Black Bayou in Terrebonne Parish, Grand River in St. Martin Parish, and Fairfax Plantation in St. Mary Parish

In 1825, Baker married Fanny Assherton in Opelousas. They had three children before she died on August 17, 1831.

He married a second time to Catherine Patton from Fairfax, Virginia in 1832. They had two children.

Colonel Baker retired to Franklin, Louisiana in 1861

Joshua Baker died at the house of his daughter in Lynne, Connecticut on April 16, 1885.