Intimacy: forced or creeper?

This isn't a question I can ask without adding a little bit of back-story.
My husband and I have been married for two years today. We have a 1-year-old and a 2-year-old and my husband works while I'm a stay-at-home mom.
Over the past six months I've noticed my husband's physical affection towards me dwindling but I didn't say anything about it until about a week ago when after hugging/kissing/being touchy with him after he got home from work he told me I was being "overbearing."
On the other hand, about a month ago he told me that he was turned off of sex (he says it's stress but he was more stressed out at the beginning of our relationship than he is now and we were having sex five times a day). He had, however, been getting it up, having sex with me, and getting off.
Here's my problem: if I approach him and try to be touchy-feely, I feel like a perverted creeper because I feel like he's cringing at my touch on the inside. If he approaches me for sex/affection I feel like he's forcing the affection to save our marriage (even though he says he's not having problems wanting sex anymore). No matter what he says to reassure me, I can't get past this.
How are we ever going to be physically loving towards one-another if I can't get over my "issues?" How do I get over this? What would you do if you were me?