Task 5: 模型融合

5.1 学习目标

学习融合策略
完成相应学习打卡任务

5.2 内容介绍

https://mlwave.com/kaggle-ensembling-guide/
https://github.com/MLWave/Kaggle-Ensemble-Guide

模型融合是比赛后期一个重要的环节，大体来说有如下的类型方式。

简单加权融合:
- 回归（分类概率）：算术平均融合（Arithmetic mean），几何平均融合（Geometric mean）；
- 分类：投票（Voting)
- 综合：排序融合(Rank averaging)，log融合
stacking/blending:
- 构建多层模型，并利用预测结果再拟合预测。
boosting/bagging（在xgboost，Adaboost,GBDT中已经用到）:
- 多树的提升方法

5.3 相关理论介绍

stacking具体原理详解

5.4 代码实例

5.4.1 回归\分类概率-融合：

(1) 简单加权平均，结果直接融合

1import numpy as np
2import pandas as pd
3from sklearn import metrics
4
5## 生成一些简单的样本数据，test_prei 代表第i个模型的预测值
6test_pre1 = [1.2, 3.2, 2.1, 6.2]
7test_pre2 = [0.9, 3.1, 2.0, 5.9]
8test_pre3 = [1.1, 2.9, 2.2, 6.0]
9
10# y_test_true 代表第模型的真实值
11y_test_true = [1, 3, 2, 6] 
12
13## 定义结果的加权平均函数
14def Weighted_method(test_pre1,test_pre2,test_pre3,w=[1/3,1/3,1/3]):
15    Weighted_result = w[0]*pd.Series(test_pre1)+w[1]*pd.Series(test_pre2)+w[2]*pd.Series(test_pre3)
16    return Weighted_result
17
18# 各模型的预测结果计算MAE
19print('Pred1 MAE:',metrics.mean_absolute_error(y_test_true, test_pre1))
20print('Pred2 MAE:',metrics.mean_absolute_error(y_test_true, test_pre2))
21print('Pred3 MAE:',metrics.mean_absolute_error(y_test_true, test_pre3))
22
23## 根据加权计算MAE
24w = [0.3,0.4,0.3] # 定义比重权值
25Weighted_pre = Weighted_method(test_pre1,test_pre2,test_pre3,w)
26print('Weighted_pre MAE:',metrics.mean_absolute_error(y_test_true, Weighted_pre))

1Pred1 MAE: 0.1750000000000001
2Pred2 MAE: 0.07499999999999993
3Pred3 MAE: 0.10000000000000009
4Weighted_pre MAE: 0.05750000000000027

可以发现加权结果相对于之前的结果是有提升的，这种我们称其为简单的加权平均。
还有一些特殊的形式，比如mean平均，median平均

1## 定义结果的加权平均函数
2def Mean_method(test_pre1,test_pre2,test_pre3):
3    Mean_result = pd.concat([pd.Series(test_pre1),pd.Series(test_pre2),pd.Series(test_pre3)],axis=1).mean(axis=1)
4    return Mean_result
5
6Mean_pre = Mean_method(test_pre1,test_pre2,test_pre3)
7print('Mean_pre MAE:',metrics.mean_absolute_error(y_test_true, Mean_pre))
8
9## 定义结果的加权平均函数
10def Median_method(test_pre1,test_pre2,test_pre3):
11    Median_result = pd.concat([pd.Series(test_pre1),pd.Series(test_pre2),pd.Series(test_pre3)],axis=1).median(axis=1)
12    return Median_result
13
14Median_pre = Median_method(test_pre1,test_pre2,test_pre3)
15print('Median_pre MAE:',metrics.mean_absolute_error(y_test_true, Median_pre))

1Mean_pre MAE: 0.06666666666666693
2Median_pre MAE: 0.07500000000000007

(2) Stacking融合(回归)

1from sklearn import linear_model
2
3def Stacking_method(train_reg1,train_reg2,train_reg3,y_train_true,test_pre1,test_pre2,test_pre3,model_L2= linear_model.LinearRegression()):
4    model_L2.fit(pd.concat([pd.Series(train_reg1),pd.Series(train_reg2),pd.Series(train_reg3)],axis=1).values,y_train_true)
5    Stacking_result = model_L2.predict(pd.concat([pd.Series(test_pre1),pd.Series(test_pre2),pd.Series(test_pre3)],axis=1).values)
6    return Stacking_result
7
8## 生成一些简单的样本数据，test_prei 代表第i个模型的预测值
9train_reg1 = [3.2, 8.2, 9.1, 5.2]
10train_reg2 = [2.9, 8.1, 9.0, 4.9]
11train_reg3 = [3.1, 7.9, 9.2, 5.0]
12# y_test_true 代表第模型的真实值
13y_train_true = [3, 8, 9, 5] 
14
15test_pre1 = [1.2, 3.2, 2.1, 6.2]
16test_pre2 = [0.9, 3.1, 2.0, 5.9]
17test_pre3 = [1.1, 2.9, 2.2, 6.0]
18
19# y_test_true 代表第模型的真实值
20y_test_true = [1, 3, 2, 6] 
21
22model_L2= linear_model.LinearRegression()
23Stacking_pre = Stacking_method(train_reg1,train_reg2,train_reg3,y_train_true,
24                               test_pre1,test_pre2,test_pre3,model_L2)
25print('Stacking_pre MAE:',metrics.mean_absolute_error(y_test_true, Stacking_pre))

Stacking_pre MAE: 0.04213483146067404

可以发现模型结果相对于之前有进一步的提升，这是我们需要注意的一点是，对于第二层Stacking的模型不宜选取的过于复杂，这样会导致模型在训练集上过拟合，从而使得在测试集上并不能达到很好的效果。

5.4.2 分类模型融合

1import numpy as np
2import lightgbm as lgb
3from sklearn.datasets import make_blobs
4from sklearn import datasets
5from sklearn.tree import DecisionTreeClassifier
6from sklearn.ensemble import RandomForestClassifier
7from sklearn.ensemble import VotingClassifier
8from sklearn.linear_model import LogisticRegression
9from sklearn.svm import SVC
10from sklearn.model_selection import train_test_split
11from sklearn.datasets import make_moons
12from sklearn.metrics import accuracy_score,roc_auc_score
13from sklearn.model_selection import cross_val_score
14from sklearn.model_selection import StratifiedKFold

(1) Voting投票机制

Voting即投票机制，分为软投票和硬投票两种，其原理采用少数服从多数的思想。

1'''
2硬投票：对多个模型直接进行投票，不区分模型结果的相对重要度，最终投票数最多的类为最终被预测的类。
3'''
4iris = datasets.load_iris()
5
6x=iris.data
7y=iris.target
8x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.3)
9
10clf1 = lgb.LGBMClassifier(learning_rate=0.1, n_estimators=150, max_depth=3, min_child_weight=2, subsample=0.7,
11                     colsample_bytree=0.6, objective='binary:logistic')
12clf2 = RandomForestClassifier(n_estimators=200, max_depth=10, min_samples_split=10,
13                              min_samples_leaf=63,oob_score=True)
14clf3 = SVC(C=0.1)
15
16# 硬投票
17eclf = VotingClassifier(estimators=[('lgb', clf1), ('rf', clf2), ('svc', clf3)], voting='hard')
18for clf, label in zip([clf1, clf2, clf3, eclf], ['LGB', 'Random Forest', 'SVM', 'Ensemble']):
19    scores = cross_val_score(clf, x, y, cv=5, scoring='accuracy')
20    print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))

1Accuracy: 0.95 (+/- 0.05) [LGB]
2Accuracy: 0.33 (+/- 0.00) [Random Forest]
3Accuracy: 0.92 (+/- 0.03) [SVM]
4Accuracy: 0.95 (+/- 0.05) [Ensemble]

(2) 分类的Stacking\Blending融合：

stacking是一种分层模型集成框架。

以两层为例，第一层由多个基学习器组成，其输入为原始训练集，第二层的模型则是以第一层基学习器的输出作为训练集进行再训练，从而得到完整的stacking模型, stacking两层模型都使用了全部的训练数据。

1'''
25-Fold Stacking
3'''
4from sklearn.ensemble import RandomForestClassifier
5from sklearn.ensemble import ExtraTreesClassifier,GradientBoostingClassifier
6import pandas as pd
7#创建训练的数据集
8data_0 = iris.data
9data = data_0[:100,:]
10
11target_0 = iris.target
12target = target_0[:100]
13
14#模型融合中使用到的各个单模型
15clfs = [LogisticRegression(solver='lbfgs'),
16        RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),
17        ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),
18        ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),
19        GradientBoostingClassifier(learning_rate=0.05, subsample=0.5, max_depth=6, n_estimators=5)]
20 
21#切分一部分数据作为测试集
22X, X_predict, y, y_predict = train_test_split(data, target, test_size=0.3, random_state=2020)
23
24dataset_blend_train = np.zeros((X.shape[0], len(clfs)))
25dataset_blend_test = np.zeros((X_predict.shape[0], len(clfs)))
26
27#5折stacking
28n_splits = 5
29skf = StratifiedKFold(n_splits)
30skf = skf.split(X, y)
31
32for j, clf in enumerate(clfs):
33    #依次训练各个单模型
34    dataset_blend_test_j = np.zeros((X_predict.shape[0], 5))
35    for i, (train, test) in enumerate(skf):
36        #5-Fold交叉训练，使用第i个部分作为预测，剩余的部分来训练模型，获得其预测的输出作为第i部分的新特征。
37        X_train, y_train, X_test, y_test = X[train], y[train], X[test], y[test]
38        clf.fit(X_train, y_train)
39        y_submission = clf.predict_proba(X_test)[:, 1]
40        dataset_blend_train[test, j] = y_submission
41        dataset_blend_test_j[:, i] = clf.predict_proba(X_predict)[:, 1]
42    #对于测试集，直接用这k个模型的预测值均值作为新的特征。
43    dataset_blend_test[:, j] = dataset_blend_test_j.mean(1)
44    print("val auc Score: %f" % roc_auc_score(y_predict, dataset_blend_test[:, j]))
45
46clf = LogisticRegression(solver='lbfgs')
47clf.fit(dataset_blend_train, y)
48y_submission = clf.predict_proba(dataset_blend_test)[:, 1]
49
50print("Val auc Score of Stacking: %f" % (roc_auc_score(y_predict, y_submission)))

1val auc Score: 1.000000
2val auc Score: 0.500000
3val auc Score: 0.500000
4val auc Score: 0.500000
5val auc Score: 0.500000
6Val auc Score of Stacking: 1.000000

Blending，其实和Stacking是一种类似的多层模型融合的形式

其主要思路是把原始的训练集先分成两部分，比如70%的数据作为新的训练集，剩下30%的数据作为测试集。
在第一层，我们在这70%的数据上训练多个模型，然后去预测那30%数据的label，同时也预测test集的label。
在第二层，我们就直接用这30%数据在第一层预测的结果做为新特征继续训练，然后用test集第一层预测的label做特征，用第二层训练的模型做进一步预测

其优点在于

比stacking简单（因为不用进行k次的交叉验证来获得stacker feature）
避开了一个信息泄露问题：generlizers和stacker使用了不一样的数据集

缺点在于：

使用了很少的数据（第二阶段的blender只使用training set10%的量）
blender可能会过拟合
stacking使用多次的交叉验证会比较稳健 '''

1'''
2Blending
3'''
4 
5#创建训练的数据集
6#创建训练的数据集
7data_0 = iris.data
8data = data_0[:100,:]
9
10target_0 = iris.target
11target = target_0[:100]
12 
13#模型融合中使用到的各个单模型
14clfs = [LogisticRegression(solver='lbfgs'),
15        RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),
16        RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),
17        ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),
18        #ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),
19        GradientBoostingClassifier(learning_rate=0.05, subsample=0.5, max_depth=6, n_estimators=5)]
20
21#切分一部分数据作为测试集
22X, X_predict, y, y_predict = train_test_split(data, target, test_size=0.3, random_state=2020)
23
24#切分训练数据集为d1,d2两部分
25X_d1, X_d2, y_d1, y_d2 = train_test_split(X, y, test_size=0.5, random_state=2020)
26dataset_d1 = np.zeros((X_d2.shape[0], len(clfs)))
27dataset_d2 = np.zeros((X_predict.shape[0], len(clfs)))
28 
29for j, clf in enumerate(clfs):
30    #依次训练各个单模型
31    clf.fit(X_d1, y_d1)
32    y_submission = clf.predict_proba(X_d2)[:, 1]
33    dataset_d1[:, j] = y_submission
34    #对于测试集，直接用这k个模型的预测值作为新的特征。
35    dataset_d2[:, j] = clf.predict_proba(X_predict)[:, 1]
36    print("val auc Score: %f" % roc_auc_score(y_predict, dataset_d2[:, j]))
37
38#融合使用的模型
39clf = GradientBoostingClassifier(learning_rate=0.02, subsample=0.5, max_depth=6, n_estimators=30)
40clf.fit(dataset_d1, y_d2)
41y_submission = clf.predict_proba(dataset_d2)[:, 1]
42print("Val auc Score of Blending: %f" % (roc_auc_score(y_predict, y_submission)))

1val auc Score: 1.000000
2val auc Score: 1.000000
3val auc Score: 1.000000
4val auc Score: 1.000000
5val auc Score: 1.000000
6Val auc Score of Blending: 1.000000

5.4.3 一些其它方法

将特征放进模型中预测，并将预测结果变换并作为新的特征加入原有特征中再经过模型预测结果（Stacking变化）
（可以反复预测多次将结果加入最后的特征中）

1def Ensemble_add_feature(train,test,target,clfs):
2    
3    # n_flods = 5
4    # skf = list(StratifiedKFold(y, n_folds=n_flods))
5
6    train_ = np.zeros((train.shape[0],len(clfs*2)))
7    test_ = np.zeros((test.shape[0],len(clfs*2)))
8
9    for j,clf in enumerate(clfs):
10        '''依次训练各个单模型'''
11        # print(j, clf)
12        '''使用第1个部分作为预测，第2部分来训练模型，获得其预测的输出作为第2部分的新特征。'''
13        # X_train, y_train, X_test, y_test = X[train], y[train], X[test], y[test]
14
15        clf.fit(train,target)
16        y_train = clf.predict(train)
17        y_test = clf.predict(test)
18
19        ## 新特征生成
20        train_[:,j*2] = y_train**2
21        test_[:,j*2] = y_test**2
22        train_[:, j+1] = np.exp(y_train)
23        test_[:, j+1] = np.exp(y_test)
24        # print("val auc Score: %f" % r2_score(y_predict, dataset_d2[:, j]))
25        print('Method ',j)
26
27    train_ = pd.DataFrame(train_)
28    test_ = pd.DataFrame(test_)
29    return train_,test_
30

1from sklearn.model_selection import cross_val_score, train_test_split
2from sklearn.linear_model import LogisticRegression
3clf = LogisticRegression()
4
5data_0 = iris.data
6data = data_0[:100,:]
7
8target_0 = iris.target
9target = target_0[:100]
10
11x_train,x_test,y_train,y_test=train_test_split(data,target,test_size=0.3)
12x_train = pd.DataFrame(x_train) ; x_test = pd.DataFrame(x_test)
13
14#模型融合中使用到的各个单模型
15clfs = [LogisticRegression(),
16        RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),
17        ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),
18        ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),
19        GradientBoostingClassifier(learning_rate=0.05, subsample=0.5, max_depth=6, n_estimators=5)]
20
21New_train,New_test = Ensemble_add_feature(x_train,x_test,y_train,clfs)
22
23clf = LogisticRegression()
24# clf = GradientBoostingClassifier(learning_rate=0.02, subsample=0.5, max_depth=6, n_estimators=30)
25clf.fit(New_train, y_train)
26y_emb = clf.predict_proba(New_test)[:, 1]
27
28print("Val auc Score of stacking: %f" % (roc_auc_score(y_test, y_emb)))

1Method  0
2Method  1
3Method  2
4Method  3
5Method  4
6Val auc Score of stacking: 1.000000

5.5 本赛题示例

5.5.1 准备工作

准备工作进行内容有：

导入数据集并进行简单的预处理
将数据集划分成训练集和验证集
构建单模：Random Forest，LGB，NN
读取并演示如何利用融合模型生成可提交预测数据

1import pandas as pd
2import numpy as np
3import warnings
4import matplotlib
5import matplotlib.pyplot as plt
6import seaborn as sns
7
8warnings.filterwarnings('ignore')
9%matplotlib inline
10
11import itertools
12import matplotlib.gridspec as gridspec
13from sklearn import datasets
14from sklearn.linear_model import LogisticRegression
15from sklearn.neighbors import KNeighborsClassifier
16from sklearn.naive_bayes import GaussianNB 
17from sklearn.ensemble import RandomForestClassifier,RandomForestRegressor
18# from mlxtend.classifier import StackingClassifier
19from sklearn.model_selection import cross_val_score, train_test_split
20# from mlxtend.plotting import plot_learning_curves
21# from mlxtend.plotting import plot_decision_regions
22
23from sklearn.model_selection import StratifiedKFold
24from sklearn.model_selection import train_test_split
25from sklearn.model_selection import StratifiedKFold
26from sklearn.model_selection import train_test_split
27import lightgbm as lgb
28from sklearn.neural_network import MLPClassifier,MLPRegressor
29from sklearn.metrics import mean_squared_error, mean_absolute_error

这里引入一个降内存的函数。

1def reduce_mem_usage(df):
2    start_mem = df.memory_usage().sum() / 1024**2 
3    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))
4    
5    for col in df.columns:
6        col_type = df[col].dtype
7        
8        if col_type != object:
9            c_min = df[col].min()
10            c_max = df[col].max()
11            if str(col_type)[:3] == 'int':
12                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
13                    df[col] = df[col].astype(np.int8)
14                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
15                    df[col] = df[col].astype(np.int16)
16                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
17                    df[col] = df[col].astype(np.int32)
18                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
19                    df[col] = df[col].astype(np.int64)  
20            else:
21                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
22                    df[col] = df[col].astype(np.float16)
23                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
24                    df[col] = df[col].astype(np.float32)
25                else:
26                    df[col] = df[col].astype(np.float64)
27        else:
28            df[col] = df[col].astype('category')
29
30    end_mem = df.memory_usage().sum() / 1024**2 
31    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))
32    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))
33    
34    return df

1train = pd.read_csv('./data/train.csv')
2test = pd.read_csv('./data/testA.csv')
3
4# 简单预处理
5train_list = []
6for items in train.values:
7    train_list.append([items[0]] + [float(i) for i in items[1].split(',')] + [items[2]])
8    
9test_list = []
10for items in test.values:
11    test_list.append([items[0]] + [float(i) for i in items[1].split(',')])
12
13train = pd.DataFrame(np.array(train_list))
14test = pd.DataFrame(np.array(test_list))
15
16# id列不算入特征
17features = ['s_'+str(i) for i in range(len(train_list[0])-2)] 
18train.columns = ['id'] + features + ['label']
19test.columns = ['id'] + features
20
21train = reduce_mem_usage(train)
22test = reduce_mem_usage(test)

1Memory usage of dataframe is 157.93 MB
2Memory usage after optimization is: 39.67 MB
3Decreased by 74.9%
4Memory usage of dataframe is 31.43 MB
5Memory usage after optimization is: 7.90 MB
6Decreased by 74.9%

1# 根据8：2划分训练集和校验集
2X_train = train.drop(['id','label'], axis=1)
3y_train = train['label']
4
5# 测试集
6X_test = test.drop(['id'], axis=1)
7
8# 第一次运行可以先用一个subdata，这样速度会快些
9X_train = X_train.iloc[:50000,:20]
10y_train = y_train.iloc[:50000]
11X_test = X_test.iloc[:,:20]
12
13# 划分训练集和测试集
14X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2)

1# 单模函数
2def build_model_rf(X_train,y_train):
3    model = RandomForestRegressor(n_estimators = 100)
4    model.fit(X_train, y_train)
5    return model
6
7
8def build_model_lgb(X_train,y_train):
9    model = lgb.LGBMRegressor(num_leaves=63,learning_rate = 0.1,n_estimators = 100)
10    model.fit(X_train, y_train)
11    return model
12
13
14def build_model_nn(X_train,y_train):
15    model = MLPRegressor(alpha=1e-05, hidden_layer_sizes=(5, 2), random_state=1,solver='lbfgs')
16    model.fit(X_train, y_train)
17    return model

1# 这里针对三个单模进行训练，其中subA_rf/lgb/nn都是可以提交的模型
2# 单模没有进行调参，因此是弱分类器，效果可能不是很好。
3
4print('predict rf...')
5model_rf = build_model_rf(X_train,y_train)
6val_rf = model_rf.predict(X_val)
7subA_rf = model_rf.predict(X_test)
8
9
10print('predict lgb...')
11model_lgb = build_model_lgb(X_train,y_train)
12val_lgb = model_lgb.predict(X_val)
13subA_lgb = model_rf.predict(X_test)
14
15
16print('predict NN...')
17model_nn = build_model_nn(X_train,y_train)
18val_nn = model_nn.predict(X_val)
19subA_nn = model_rf.predict(X_test)

1predict rf...
2predict lgb...
3predict NN...

5.5.2 加权融合

首先我们尝试加权融合模型：

如果没有给权重矩阵，就是均值融合模型
权重矩阵可以进行自定义，这里我们是用三个单模进行融合。如果有更多需要更改矩阵size

1# 加权融合模型，如果w没有变，就是均值融合
2def Weighted_method(test_pre1,test_pre2,test_pre3,w=[1/3,1/3,1/3]):
3    Weighted_result = w[0]*pd.Series(test_pre1)+w[1]*pd.Series(test_pre2)+w[2]*pd.Series(test_pre3)
4    return Weighted_result
5
6# 初始权重，可以进行自定义，这里我们随便设置一个权重
7w = [0.2, 0.3, 0.5]
8
9val_pre = Weighted_method(val_rf,val_lgb,val_nn,w)
10MAE_Weighted = mean_absolute_error(y_val,val_pre)
11print('MAE of Weighted of val:',MAE_Weighted)

MAE of Weighted of val: 0.09326

这里单独展示一下将多个单模预测结果融合成融和模型结果

1## 预测数据部分
2subA = Weighted_method(subA_rf,subA_lgb,subA_nn,w)
3
4## 生成提交文件
5sub = pd.DataFrame()
6sub['SaleID'] = X_test.index
7sub['price'] = subA
8sub.to_csv('./sub_Weighted.csv',index=False)

5.5.3 Stacking融合

1## Stacking
2
3## 第一层
4train_rf_pred = model_rf.predict(X_train)
5train_lgb_pred = model_lgb.predict(X_train)
6train_nn_pred = model_nn.predict(X_train)
7
8stacking_X_train = pd.DataFrame()
9stacking_X_train['Method_1'] = train_rf_pred
10stacking_X_train['Method_2'] = train_lgb_pred
11stacking_X_train['Method_3'] = train_nn_pred
12
13stacking_X_val = pd.DataFrame()
14stacking_X_val['Method_1'] = val_rf
15stacking_X_val['Method_2'] = val_lgb
16stacking_X_val['Method_3'] = val_nn
17
18stacking_X_test = pd.DataFrame()
19stacking_X_test['Method_1'] = subA_rf
20stacking_X_test['Method_2'] = subA_lgb
21stacking_X_test['Method_3'] = subA_nn

1stacking_X_test.head()

<div> <style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

1.dataframe tbody tr th {
2    vertical-align: top;
3}
4
5.dataframe thead th {
6    text-align: right;
7}

</style> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>Method_1</th> <th>Method_2</th> <th>Method_3</th> </tr> </thead> <tbody> <tr> <th>0</th> <td>0.0</td> <td>0.0</td> <td>0.0</td> </tr> <tr> <th>1</th> <td>2.0</td> <td>2.0</td> <td>2.0</td> </tr> <tr> <th>2</th> <td>3.0</td> <td>3.0</td> <td>3.0</td> </tr> <tr> <th>3</th> <td>0.0</td> <td>0.0</td> <td>0.0</td> </tr> <tr> <th>4</th> <td>0.0</td> <td>0.0</td> <td>0.0</td> </tr> </tbody> </table> </div>

1# 第二层是用random forest
2model_lr_stacking = build_model_rf(stacking_X_train,y_train)
3
4## 训练集
5train_pre_Stacking = model_lr_stacking.predict(stacking_X_train)
6print('MAE of stacking:',mean_absolute_error(y_train,train_pre_Stacking))
7
8## 验证集
9val_pre_Stacking = model_lr_stacking.predict(stacking_X_val)
10print('MAE of stacking:',mean_absolute_error(y_val,val_pre_Stacking))
11
12## 预测集
13print('Predict stacking...')
14subA_Stacking = model_lr_stacking.predict(stacking_X_test)
15

1MAE of stacking: 0.0
2MAE of stacking: 0.03384
3Predict stacking...

5.6 经验总结

模型融合是数据挖掘比赛后期上分的主要方式，尤其是进行队伍合并后，模型融合有很多优势。总结一下三个方面：

结果层面的融合，这种是最常见的融合方法，其可行的融合方法也有很多，比如根据结果的得分进行加权融合，还可以做Log，exp处理等。在做结果融合的时候。有一个很重要的条件是模型结果的得分要比较近似但结果的差异要比较大，这样的结果融合往往有比较好的效果提升。如果不满足这个条件带来的效果很低，甚至是负效果。
特征层面的融合，这个层面叫融合融合并不准确，主要是队伍合并后大家可以相互学习特征工程。如果我们用同种模型训练，可以把特征进行切分给不同的模型，然后在后面进行模型或者结果融合有时也能产生比较好的效果。
模型层面的融合，模型层面的融合可能就涉及模型的堆叠和设计，比如加stacking，部分模型的结果作为特征输入等，这些就需要多实验和思考了，基于模型层面的融合最好不同模型类型要有一定的差异，用同种模型不同的参数的收益一般是比较小的。

天池比赛数据挖掘心电图模型融合