Just put him in a room with his toys and litterbox and some food and water. He doesn't like them, and probably feels left out when the children come, plus some cats just don't like other people. Thats normal, but just to keep the kids from getting hurt or scared just keep the cat away. The kitty will be grateful for this too cuz they are probably scared.