There is a happy medium here. I don't think this has to be an all or nothing scenario.
I agree with everyone else who said you have to find out WHY. I created my own non-profit on a cause that was very near and dear to me. FH knew this, and actually helped me to start it. We rarely get date nights & so on (we're both workaholics working multiple jobs). He knew how important this was to me, and thus it became important to him.
I do think you need to find out is it because of the fear/discomfort with the topic of cancer that some PP's had mentioned, is it because of the fear of you being taken advantage of like you had said, is it a time issue...SOMETHING is bothering him about this.
If my FH & I need to discuss something that's REALLY important or we're arguing about something, I write my thoughts out so that I know I'm making all the points I want/need to. Maybe you should just write out some key points from your perspective & then sit him down & just talk. Listen to each other & try to find a middle ground.
You can absolutely be a part of this fund-raiser without it killing you (or your marriage). If it's a time issue, see if you can "down-size" some of your responsibilities. There are ways to making something like this work so that it's a good compromise for both of you, and no one gets really hurt.