In the 1861 census Guilford Street, Luton, Bedfordshire.
Head Francis M. Brown, age 47, Organist, born St. Albans, Hertfordshire.
Wife Hannah Brown, age 47 [This differs from her baptism records.] born London, Middlesex.
Son John Brown, age 19, married, flocker?, born St. Albans, Hertfordshire.
Daughter in law Ann Brown, age 21, bonnet sewer, born Littleton, Hertfordshire.
Son Frederic Brown, age 16, shopman, born St. Albans.
Son Philip Brown, age 12, scholar, born St. Albans. 2 lodgers.
Grandson James Brown, age 1, born Luton.
Grandson Frederic Brown, age 9 months, born Luton.

In the 1871 census New Town Road, Luton, Bedfordshire.
John Thomas Brown, age 31, Block Maker, born St. Albans, Hertfordshire.
Wife Ann Brown, age 33, born Berkhamsted, Buckinghamshire.
Son James Brown, age 11, scholar, born Luton, Bedfordshire.
Son Frederick Brown, age 10, scholar, born Luton.
Daughter Louisa Brown, age 8, scholar, born Luton.
Daughter Helen Brown, age 6, scholar, born Luton.
Son Francis Brown, age 5, scholar, born Luton.
Daughter Agnes Brown, age 3, scholar, born Luton.
Son Joseph Brown, age 1, born Luton.

A possible entry in the 1881 census
St. John’s College, New Bedford Road, Luton, Bedfordshire
Frederick Brown, servant, unmarried, age 20, page servant, born Luton, Bedfordshire.
[St. John’s College seems to be a boardings school for boys age 11 to 17.]
[There seems to be another Frederick Brown - Frederick Amos Brown born in Luton at about the same time, so this is only a possibility.]