He was the eldest son of George Villiers, 4th Viscount Grandison and his wife Mary, daughter of Francis Leigh, 1st Earl of Chichester. In 1677 he married the heiress Katherine FitzGerald, through whom he gained substantial property in County Waterford. He adopted the surname of FitzGerald to reflect this inheritance. Their children included:
- John, later 5th Viscount and 1st Earl Grandison
- Mary, who married Brigadier-General William Steuart (d. 1736), nephew of her stepfather
- Harriet, who married Robert Pitt and was the mother of William Pitt, 1st Earl of Chatham.
He died in 1693, so on his father's death in 1699 his son John succeeded to the viscountcy. In the same year his widow was granted the title Viscountess Grandison as if her late husband had succeeded. She later married General William Steuart.
- John Burke, A General and Heraldic dictionary of the Peerage and Baronetage of the British Empire, fourth edition, London 1832, vol 2, p. 10 (online at google books)