The communication of a patient with amyotrophic lateral sclerosis is limited and then the quality of lives would be greatly reduced. The patients still maintain the cognitive ability, thus developing an assistive communication interface would greatly help them in daily live. Recently, steady state visually evoked potential (SSVEP) based brain computer interfaces (BCIs) had been successfully developed to help patients. Increasing the accuracy of SSVEP-based BCIs is able to realize the assistive communication interfaces in practical applications. In this study, a modular continuous restricted Boltzmann machine (MCRBM) is proposed to improve the performance of SSVEP-based BCIs. To precisely represent the characteristics of elicited signals of SSVEP, the frequency magnitude, the coefficients of canonical correlation analysis, and the correlations of magnitude square coherence are selected as the features. In the first layer of MCRBM, the continuous restricted Boltzmann machine based neural networks are used as the basic units and applied to accurately estimate by using different types of features. In the second layer of MCRBM, a CRBM is then designed to fuse the decisions and find the final results. The experimental results showed that MCRBM produce higher accuracy compared to CRBM. Therefore, the proposed approach can be adopted in practical applications and then help patients in communicating with others.