Spatial modulation (SM) combined with spatial multiplexing is a newly developed transmission scheme in multiple-input multiple-output (MIMO) systems. The resultant system, referred to as generalized SM (GSM), can use the maximum-likelihood (ML) detector jointly detecting the antenna-subset (AS) index and symbol vector. As known, the ML detector can achieve optimum performance; however, its computational complexity can be prohibitively high when the dimension of the GSM system is large. In this paper, we propose new methods to solve the problem. The main idea is to split the detection into two stages, one for the AS index and the other for the symbol vector. For the detection of the AS index, we develop two methods, referred to as Gaussian approximation and QR projection. Once the AS index is detected, conventional low-complexity ML detectors can be applied for the detection of the symbol vector. The diversity order for the proposed methods are further analyzed and an enhanced method is also proposed to achieve near-optimum performance. Finally, the proposed methods are extended to conduct soft detection of GSM systems. Simulations show that our methods significantly outperform existing ones while the detection complexity remains similar.