I think you have the wrong parents. John and Mary had a son called James many years before 1829.

1841 census
Market Place, Caistor
John CHANT 65 Labourer
Mary 55
James 35 Druggist
Joseph SWAN? 65 Ind

James was actually bap 23 Nov 1804. In 1851, James was still unmarried and his widowed father was with him.

James CHANT bap 29 March 1829 was the son of James Good CHANT and Mary. There was an 1838 Caistor baptism for Joseph, son of James CANT and Mary, perhaps the same couple.

In view of what crusoe found, the 1829 baptism is interesting.