Refined Prediction of Mouse and Human Actions Based on A Data-Selective Multiple-Stage Approach of 3D Convolutional Neural Networks

How to refine and improve the prediction power of a deep learning model is still a challenge. In this paper, we propose a data-selective multiple-stage approach to refine the prediction results from a single 3D convolutional neural network (3D CNN) in terms of precision and recall measurements. In the initial stage of the learning process, a 3D CNN is trained with a pre-trained model and the precisions and recalls for each class are calculated. An over-predicted ratio is also calculated as summarizing the true positive and false negative ratios of each predicted class according to the normalized confusion matrix. From the training result, those with both of the precision and recall larger than a preset threshold and the over-predicted ratio ≤ 1 are directly output as the accepted classes at this stage. For the other classes not satisfied the above three criterions, the corresponding training data are selected as the input for the next stage. If the number of selected clusters is less than that at the beginning of this stage, another new 3D CNN is created and trained with the selected data of unsolved classes. After repeating the procedures until no new 2D CNN is needed to be created, all trained models and accepted classes at each stage are saved. In the testing phase, each data is input to the model at the first stage. If the predicted class is belonged to the accepted class at this stage, the result is output directly. Otherwise, the data is input to the next stage. After repeating the process until the predicted class is belonged to the accepted class at some stage, then the testing phase is terminated. In the results, we have used two video datasets of mouse behaviors in cage and human actions in the wild to examine the proposed method and found that our method can significantly improve both of the precisions and recalls of the predictions in comparison with only using a single 3D CNN model.