Prince Harry’s decision to quit the royal family is reportedly taking a toll on him. A source told New Idea…