Task 2 数据分析

2.1 EDA 目标

EDA的价值主要在于熟悉数据集，了解数据集，对数据集进行验证来确定所获得数据集可以用于接下来的机器学习或者深度学习使用。
当了解了数据集之后我们下一步就是要去了解变量间的相互关系以及变量与预测值之间的存在关系。
引导数据科学从业者进行数据处理以及特征工程的步骤,使数据集的结构和特征集让接下来的预测问题更加可靠。
完成对于数据的探索性分析，并对于数据进行一些图表或者文字总结并打卡。

2.2 内容介绍

载入各种数据科学以及可视化库:
- 数据科学库 pandas、numpy、scipy；
- 可视化库 matplotlib、seabon；
载入数据：
- 载入训练集和测试集；
- 简略观察数据(head()+shape)；
数据总览:
- 通过describe()来熟悉数据的相关统计量
- 通过info()来熟悉数据类型
判断数据缺失和异常
- 查看每列的存在nan情况
- 异常值检测
了解预测值的分布
- 总体分布概况
- 查看skewness and kurtosis
- 查看预测值的具体频数

2.3 代码示例

2.3.1 载入各种数据科学与可视化库

1#coding:utf-8
2#导入warnings包，利用过滤器来实现忽略警告语句。
3import warnings
4warnings.filterwarnings('ignore')
5import missingno as msno
6import pandas as pd
7from pandas import DataFrame
8import matplotlib.pyplot as plt 
9import seaborn as sns
10import numpy as np

2.3.2 载入训练集和测试集

导入训练集train.csv

1import pandas as pd
2from pandas import DataFrame, Series
3import matplotlib.pyplot as plt
4Train_data = pd.read_csv('./train.csv')

导入测试集testA.csv

1import pandas as pd
2from pandas import DataFrame, Series
3import matplotlib.pyplot as plt 
4Test_data = pd.read_csv('./testA.csv')

所有特征集均脱敏处理(方便大家观看)

id - 心跳信号分配的唯一标识
heartbeat_signals - 心跳信号序列
label - 心跳信号类别（0、1、2、3）

data.head().append(data.tail())——观察首尾数据

data.shape——观察数据集的行列信息

观察train首尾数据

1Train_data.head().append(Train_data.tail())

1<bound method DataFrame.info of           id                                  heartbeat_signals  label
20          0  0.9912297987616655,0.9435330436439665,0.764677...    0.0
31          1  0.9714822034884503,0.9289687459588268,0.572932...    0.0
42          2  1.0,0.9591487564065292,0.7013782792997189,0.23...    2.0
53          3  0.9757952826275774,0.9340884687738161,0.659636...    0.0
64          4  0.0,0.055816398940721094,0.26129357194994196,0...    2.0
7...      ...                                                ...    ...
899995  99995  1.0,0.677705342021188,0.22239242747868546,0.25...    0.0
999996  99996  0.9268571578157265,0.9063471198026871,0.636993...    2.0
1099997  99997  0.9258351628306013,0.5873839035878395,0.633226...    3.0
1199998  99998  1.0,0.9947621698382489,0.8297017704865509,0.45...    2.0
1299999  99999  0.9259994004527861,0.916476635326053,0.4042900...    0.0
13
14[100000 rows x 3 columns]>

观察train数据集的行列信息

1Train_data.shape

1(100000, 3)

观察testA首尾数据

1Test_data.head().append(Test_data.tail())

1id	heartbeat_signals
20	100000	0.9915713654170097,1.0,0.6318163407681274,0.13...
31	100001	0.6075533139615096,0.5417083883163654,0.340694...
42	100002	0.9752726292239277,0.6710965234906665,0.686758...
53	100003	0.9956348033996116,0.9170249621481004,0.521096...
64	100004	1.0,0.8879490481178918,0.745564725322326,0.531...
719995	119995	1.0,0.8330283177934747,0.6340472606311671,0.63...
819996	119996	1.0,0.8259705825857048,0.4521053488322387,0.08...
919997	119997	0.951744840752379,0.9162611283848351,0.6675251...
1019998	119998	0.9276692903808186,0.6771898159607004,0.242906...
1119999	119999	0.6653212231837624,0.527064114047737,0.5166625...

观察testA数据集的行列信

1Test_data.shape

1(20000, 2)

要养成看数据集的head()以及shape的习惯，这会让你每一步更放心，导致接下里的连串的错误, 如果对自己的pandas等操作不放心，建议执行一步看一下，这样会有效的方便你进行理解函数并进行操作

2.3.3 总览数据概况

describe种有每列的统计量，个数count、平均值mean、方差std、最小值min、中位数25% 50% 75% 、以及最大值看这个信息主要是瞬间掌握数据的大概的范围以及每个值的异常值的判断，比如有的时候会发现999 9999 -1 等值这些其实都是nan的另外一种表达方式，有的时候需要注意下
info 通过info来了解数据每列的type，有助于了解是否存在除了nan以外的特殊符号异常

data.describe()——获取数据的相关统计量

data.info()——获取数据类型

获取train数据的相关统计量

1Train_data.describe()

1id	label
2count	100000.000000	100000.000000
3mean	49999.500000	0.856960
4std	28867.657797	1.217084
5min	0.000000	0.000000
625%	24999.750000	0.000000
750%	49999.500000	0.000000
875%	74999.250000	2.000000
9max	99999.000000	3.000000

获取train数据类型

1Train_data.info

1<bound method DataFrame.info of           id                              heartbeat_signals  label
20          0  0.9912297987616655,0.9435330436439665,0.764677...    0.0
31          1  0.9714822034884503,0.9289687459588268,0.572932...    0.0
42          2  1.0,0.9591487564065292,0.7013782792997189,0.23...    2.0
53          3  0.9757952826275774,0.9340884687738161,0.659636...    0.0
64          4  0.0,0.055816398940721094,0.26129357194994196,0...    2.0
7...      ...                                                ...    ...
899995  99995  1.0,0.677705342021188,0.22239242747868546,0.25...    0.0
999996  99996  0.9268571578157265,0.9063471198026871,0.636993...    2.0
1099997  99997  0.9258351628306013,0.5873839035878395,0.633226...    3.0
1199998  99998  1.0,0.9947621698382489,0.8297017704865509,0.45...    2.0
1299999  99999  0.9259994004527861,0.916476635326053,0.4042900...    0.0
13
14[100000 rows x 3 columns]>

获取testA数据的相关统计量

1Test_data.describe()

1 					id
2count	20000.000000
3mean	109999.500000
4std	5773.647028
5min	100000.000000
625%	104999.750000
750%	109999.500000
875%	114999.250000
9max	119999.000000

获取testA数据类型

1Test_data.info()

1<class 'pandas.core.frame.DataFrame'>
2RangeIndex: 20000 entries, 0 to 19999
3Data columns (total 2 columns):
4 #   Column             Non-Null Count  Dtype 
5---  ------             --------------  ----- 
6 0   id                 20000 non-null  int64 
7 1   heartbeat_signals  20000 non-null  object
8dtypes: int64(1), object(1)
9memory usage: 312.6+ KB

2.3.4 判断数据缺失和异常

data.isnull().sum()——查看每列的存在nan情况

查看trian每列的存在nan情况

1Train_data.isnull().sum()

1id                   0
2heartbeat_signals    0
3label                0
4dtype: int64

查看testA每列的存在nan情况

1Test_data.isnull().sum()

1id                   0
2heartbeat_signals    0
3dtype: int64

2.3.5 了解预测值的分布

1Train_data['label']

10        0.0
21        0.0
32        4.0
43        0.0
54        0.0
6        ... 
799995    4.0
899996    0.0
999997    0.0
1099998    0.0
1199999    1.0
12Name: label, Length: 100000, dtype: float64

1Train_data['label'].value_counts()

10.0    58883
24.0    19660
32.0    12994
41.0     6522
53.0     1941
6Name: label, dtype: int64

1## 1) 总体分布概况（无界约翰逊分布等）
2import scipy.stats as st
3y = Train_data['label']
4plt.figure(1); plt.title('Default')
5sns.distplot(y, rug=True, bins=20)
6plt.figure(2); plt.title('Normal')
7sns.distplot(y, kde=False, fit=st.norm)
8plt.figure(3); plt.title('Log Normal')
9sns.distplot(y, kde=False, fit=st.lognorm)

![image-20210131211440263](/Users/haoyue/Library/Application Support/typora-user-images/image-20210131211440263.png)

![image-20210131211452271](/Users/haoyue/Library/Application Support/typora-user-images/image-20210131211452271.png)

1# 2）查看skewness and kurtosis
2sns.distplot(Train_data['label']);
3print("Skewness: %f" % Train_data['label'].skew())
4print("Kurtosis: %f" % Train_data['label'].kurt())

1Skewness: 0.917596
2Kurtosis: -0.825276

![image-20210131211600245](/Users/haoyue/Library/Application Support/typora-user-images/image-20210131211600245.png)

1Train_data.skew(), Train_data.kurt()

1(id       0.000000
2 label    0.917596
3 dtype: float64, id      -1.200000
4 label   -0.825276
5 dtype: float64)

1sns.distplot(Train_data.kurt(),color='orange',axlabel ='Kurtness')

![image-20210131211722579](/Users/haoyue/Library/Application Support/typora-user-images/image-20210131211722579.png)

1## 3) 查看预测值的具体频数
2plt.hist(Train_data['label'], orientation = 'vertical',histtype = 'bar', color ='red')
3plt.show()

![image-20210131211751810](/Users/haoyue/Library/Application Support/typora-user-images/image-20210131211751810.png)

天池比赛数据挖掘心电图数据分析