The creative behinds behind Downton Abbey had a tragic rule when it came to its core actors wanting to exit the series.