This study intends to develop a methodology of fraudulent transaction detection model. The algorithm of XGBoost integrating the techniques of SMOTE sampling method and Bayesian Hyperparameter Optimization, is proposed to separate fraud transactions from non-fraud transactions. The experimental results based on the public data set of financial statement fraud from Kaggle website show the proposed model is better than the commonly used binary-classification methods, such as Logistic Regression, SVM, KNN, Random Forest, XGBoost without Hyperparameter Tuning and Multilayer Perceptron. The method of establishing fraud detection models assists people who lack the machine learning domain expertise for the modeling and tuning parameter techniques. It can help to detect abnormal transactions as early as possible and carry out risk management for banking industry.