Does he think that you wont have any time for him? Or does he see that you are being taken advantage of as that usually happens in committees where one person ends up doing all the work. Maybe that is you?
If he just wants to spend more time with you, then maybe the two of you can have a special night every week just for you two. I think that there must be something else going on other than he "just doesnt want you to be on the committee". I hope the two of you can talk more and you can figure out the real reason why he is upset!