Antenna selection is a simple but effective technique to enhance the performance of a spatial multiplexing multiple-input-multiple-output (MIMO) system. The selection criterion depends on the detector used at the receiver. For the maximum-likelihood detector, the criterion is to maximize the free distance. However, an exhaustive search is required to derive the distance, and the computational complexity can be prohibitively high. To avoid the exhaustive search, a lower bound of the free distance derived with the singular value decomposition (SVD) was then developed. This bound only involves the smallest singular value of the channel matrix, and its maximization can easily be conducted. An alternative lower bound of the free distance with the QR decomposition (QRD) was also derived in the literature. In this paper, we first propose a QRD-based selection method maximizing the lower bound. With some matrix properties, we theoretically prove that the lower bound yielded by the QRD is tighter than that by the SVD. We then propose a basis transformation method so that the lower bound yielded by the QRD can further be tightened. As a result, the QRD-based selection method can achieve near optimum performance. Finally, we extend the use of the proposed methods to other applications, such as receive antenna selection, joint transmit/receive antenna selection, and antenna selection in MIMO relay systems. Simulations show that the proposed selection methods can significantly outperform existing methods.