An eight-year-old boy and his grandad drifted more than two miles off the Vale of Glamorgan coast.