This paper establishes a methodology to build hybrid machine learning models, aiming to combine the power of different machine learning algorithms on different types of features and hypothesis. A generic cost-based outlier removal algorithm is introduced as a step of preprocess of training data, we implement a hybrid machine learning model for a crediting problem, and experiment combination of three types of machine learning algorithms SVM, DT and LR. The new hybrid models shows improvement in performance compared to the traditional single SVM, DT, and LR. This new methodology can be further explored with other algorithms and applications.