Studies comparing the sexual attitudes of men and women with intellectual disabilities (ID) have been limited. A mixed-methods approach was employed to explore attitudes toward sexuality among men and women with ID in Taiwan and to ascertain the disparities between attitudes among men and women with ID. First, fifty-six men and forty-four women with mild and moderate ID completed a face-to-face interview survey. After this, focus groups were conducted for men and women with ID. Results indicated that women with ID were more likely to have negative attitudes toward parenting and non-reproductive sexual behavior than their male counterparts. Qualitative data indicated that men and women with ID had different sexual attitudes and experiences and were subject to different expectations from people around them. Both men and women with ID had very limited opportunities to develop romantic relationships and a healthy sexual identity. Sexual rights awareness and practice should be matters of concern for this group of adults and women with ID in particular.