This combo is not terrible, and thank goodness for some color in a wedding! My only advice would be, if you're going to go for it, really go for it. Also, I'm trying to picture an ivory shirt and vest. Perhaps a plain white shirt and vest instead? Not sure.
I love the color. I wore an ivory dress, but DH wore a white shirt. I would go with white, the tux salesperson actually told me the ivory shirts look kind of yellow. You couldn't tell the difference that he was wearing white and I had ivory.