HelloWorld
登录

Tensorflow2.0

Easter79

2021-10-12

1595 0 0

Tensorflow2.0

  1. Tensorflow 简介

    1. Tensorflow是什么

      1. Google开源软件库

        1. 采用数据流图,用于数值计算
        2. 支持多平台 GPU CPU 移动设备
        3. 最初用于深度学习,变得通用

      2. 数据流图

        1. 节点---处理数据
        2. 线---节点之间的输入输出关系
        3. 线上运输张量
        4. 节点被分配到各种计算设备上运行

      3. 特性

        1. 高度的灵活性
        2. 真正的可移植性
        3. 产品与科研结合
        4. 自动求微分
        5. 多语言支持
        6. 性能最优化

    2. 历史

      1. 历史版本

        1. 2015年11月:首次发布
        2. 2015年12月:支持GPU python3.3(v0.6)
        3. 2016年4月:分布式tensorflow(v0.8)
        4. 2016年11月:支持windows(v0.11)
        5. 2017年2月:性能改进,API稳定性(v1.0)
        6. 2017年4月:Keras集成(v1.1)
        7. 2017年8月:高级API,预算估计器,更多模型,初始TPU支持(v1.3)
        8. 2017年11月:Eager execution和Tensorflow Lite(v1.5)
        9. 2018年3月:TF hub(与训练的库),Tensorflow.js Tensorflow Extended(TFX)
        10. 2018年5月:新入门内容 Cloud TPU模块与管道(v1.6)
        11. 2018年6月:新的分布式策略API:概率编程工具,Tensorflow Probability(v1.8)
        12. 2018年8月:Cloud Big Table 集成(v1.10)
        13. 2018年10月:侧重可用性的API改进(v1.12)
        14. 2019年:tensorflow2.0

      2. Tensorflow1.0--------------主要特性

        1. XLA:Accelerate Linear Algebra
          1. 提升速度58倍
          2. 可以在移动设备运行
        2. 引入更高级的API-------tf.layers/tf.metrics/tf.losses/tf.keras
        3. Tensorflow 调试器
        4. 支持docker,引入tensorflow serving服务

      3. Tensorflow1.0---------------------架构

        Kears,Estimator,Datasets,Layers,Distribution engine

      4. Tensorflow2.0----------------------主要特性

        1. 使用tf.keras和eager mode进行更加简单的模型构建
        2. 鲁棒的跨平台部署
        3. 强大的研究实验
        4. 清除了不推荐使用的API和减少了重复来简化PI

      5. Tensorflow2.0------------------------架构

      6. Tensorflow2.0--------简化的模型开发流程

        1. 使用tf.data加载数据
        2. 使用tf.keras构建模型,也可以使用premade estimator来验证模型
          1. 使用tensorflow hub 进行迁移学习
        3. 使用eager mode 进行运行和调试
        4. 使用分发策略进行分布式训练
        5. 导出到SavedModel
        6. 使用Tensorflow Serve,Tensorflow Lite,Tensorflow.js部署模型

      7. Tensorflow2.0-----------强大的跨平台能力

        1. Tensorflow服务
          1. 直接通过Http/REST或GRPC/协议缓冲区
        2. Tensorflow Lite----------可部署在Android ios和其他嵌入式设备
        3. Tensorflow.js------可在javascript中部署模型
        4. 其他语言
          1. c,java,go,c#等

      8. Tensorflow2.0------------强大的研究实现

        1. keras功能和子类API,允许创建一些复杂的拓扑结构
        2. 自定义训练逻辑,使用tf.GradientTape和tf.custom_gradient进行更细力度的控制
        3. 低层API自始至终可以与高层结合使用,完全的可定制
        4. 高级扩展:Ragged Tensors,Tensor2Tensor等

    3. Tensorflow vs Pytorch

      1. 入门时间

        1. Tensorflow 1.*

          1. 静态图
          2. 学习额外概念
            1. 图,会话,变量,占位符等
          3. 写样板代码

        2. Tensorflow 2.*

          1. 动态图
          2. Eager mode避免1.0缺点,直接集成在python中

        3. Pytorch

          1. 动态图
          2. Numpy的扩展,直接集成在python中

        4. 静态图效率高,动态图容易调试

        5. 代码示例 1+$\frac{1}{2}$+ $\frac{1}{2^2}$+.....+$\frac{1}{2^{50}}$

          1. 1 # python
2 x = 0
3 y = 1
4 for iteration in range(50):
5     x = x + y
6     y = y / 2
7 print(x)
8 
          2. 1 # Pytorch
2 import torch
3 x = tf.constant(0.)
4 y = tf.constant(1.)
5 for iteration in range(50):
6     x = x + y
7     y = y / 2
8 print(x)
9 
          3. 1 # tensorflow1.*
2 import tensorflow as tf
3 x = tf.constant(0.)
4 y = tf.constant(1.)
5 add_op = x.assgin(x+y)
6 div_op = y.assgin(y/2)
7 with tf.Session() as sess:
8     sess.run(tf.golbal_variables_initializers())
9     for iteration in range(50):
10         sess.run(add_op)
11         sess.run(dic_op)
12      print(x.eval()) # sess.eval(x)
13 
          4. 1 # tensorflow2.*
2 import tensorflow as tf
3 x = tf.constant(0.)
4 y = tf.constant(1.)
5 for iteration in range(50)
6 	x = x + y
7     y = y / 2
8 print(x.numpy())
9

      2. 图创建和调试

        1. Tensorflow 1.*
          1. 静态图,难以调试,学习tfdbg调试
        2. Tensorflow 2.*与pytorch
          1. 动态图,python自带的调试工具

      3. 全面性

        1. Pythorch缺少
          1. 沿维翻转张量(np.flip,np.flipud,np.fliplr)
          2. 检查无穷与非数值张量(np.is_nan,np.is_inf)
          3. 快速傅里叶变换(np.fft)
        2. 随着时间变化,越来越接近

      4. 序列化和部署

        1. Tensorflow支持更加广泛
        2. 图保存为protocol buffer
        3. 跨语言
        4. 跨平台
        5. Pytorch支持比较简单

    4. 环境配置

      1. 本地配置

        1. Virtualenv 安装或anaconda3
        2. GPU环境配置

      2. 云端配置

        1. 为什么在云端配置

          1. 规格统一,节省自己的机器
          2. 有直接配置好的镜像

        2. 云环境

          1. Google Cloud,Amazon

          2. 实战

            1. 从0配置

              1# ubuntu 18.04
2sudo apt-get install python3 # 安装python3
3sudo apt-get install python # 安装python3
4sudo apt-get install software-properties-common
5sudo apt-add-repository universe  # 奖pip所在源加入ubuntu
6sudo apt-get update
7sudo apt-get install python-pip
8sudo apt-get install python3-pip
9sudo pip3 install -U virtualenv # 或使用anaconda3
10# 创建虚拟环境
11mkdir environment && cd environment
12virtualenv --system-site-packages -p python3 ./tf_py3
13source tf_py3/bin/activate # 激活环境
14pip install tensorflow
15# pip install tensorflow==2.0.0
16pip install numpy pandas matplotlib sklearn jupyter
17# 配置jupyter
18# 修改为静态ip配置,端口
19jupyter notebook --generate-config # 家目录生成配置文件 .jupyter
20c = get_config()
21c.NotebookApp.ip = "*"
22c.NotebookAPP.open_borwser = False
23c.NotebookAPP.post = 6006
24c.NotebookApp.allow_remote_access = True
25# 可使用token设置密码 供以后使用
26deactivate # 退出环境
27
28
29# 配置gpu环境
30# https://tensorflow.google.cn/install/gpu
31# 找到ubuntu18.04 复制为安装脚本执行即可
32# 可能会出现版本号不对应错误 libnvinfer-dev 将脚本中最后一条命令修改后运行libnvinfer-dev-5.1.5-1+cuda10.1 即可成功
33nvidia-smi # 查看GPU信息
34# tensorflow-gpu安装与cpu版类似

      tf.test.is_gpu_available() # 判断gpu可用与否 ```

      1     2. 从镜像配置
2
3        ```shell
4        # 云端的系统镜像直接有开发环境
5        # 升级tensorflow 版本
6        pip install --upgrade tensorflow-gpu==2.0.0
7        pip3 install --upgrade tensorflow-gpu==2.0.0
8        ```

  2. Tensorflow keras 实战

    1. keras是什么

      1. 基于python的高级神经网络API
      2. Francois Chollet2014-2015编写
      3. 以Tensorflow CNTK,或Theano为后端运行,keras必须有后端才可以
        1. 后端可以切换,现在多用tensorflow
      4. 及方便用于快速试验

    2. tf.keras是什么

      1. Tensorflow 对kerasAPI规范的实现
      2. 相对于tensorflow为后端的keras,Tensorflow-keras与tensorflow结合更紧密
      3. 实现在tf.keras下

    3. Tf-keras与keras的

      1. 联系

        1. 基于同一套API
          1. keras程序可转化为tf.keras
          2. 反之可能不成立,tf.keras有其它特性
        2. 相同的JSON与hdf5模型序列化格式和语义

      2. 区别

        1. Tf.keras全面支持eager mode
          1. 只使用keras.Sequential和keras.Model时没影响
          2. 自定义Model内部运算逻辑时会有影响
            1. T低层API可以使用keras的model.fit等抽象法
            2. 适用于研究人员
        2. Tf.keras支持基于tf.data的模型训练
        3. Tf.keras支持TPU训练
        4. Tf.keras支持tf.distribution中的分布式策略
        5. 其他特性
          1. Tf.keras可以与Tensorflow中的estimator集成
          2. Tf.keras可以保存为SavedModel

    4. 知识点

      1. 分类问题与回归问题

        1. 分类问题:输出类型是概率分布
        2. 回归问题:输出是一个是数值

      2. 目标函数

        1. 参数逐步调整

        2. 目标函数帮助衡量模型好坏

        3. 分类问题

          1. 要衡量目标类别与当前预测的差距

            1. 三分类问题输出:[0.2,0.7,0.1]
            2. 真是类别:2->ont_hot->[0,0,1]

          2. One_hot编码,把正整数变为向量表达

            1. 生成一个长度不小于正整数的向量,只有正整数的位置处为1,其余位置都为0

          3. 平方差损失 $$ \frac{1}{n} \sum_{x,y}\frac{1}{2}(y-Model(x))^2 $$

          4. 交叉熵损失 $$ \frac{1}{n}\sum_{x,y}yln(Model(x)) $$

        4. 回归问题

          1. 预测值与真实值的差距
          2. 平方差损失
          3. 绝对值损失

    5. Tf框架:keras,回调函数

      1. keras搭建模型

        1. 1 # coding:utf-8
2 # file: tf_keras_classification_model.py
3 # author: Dean
4 # contact: 1028968939@qq.com
5 # time: 2019/12/17 11:47
6 # desc: keras模型搭建
7 
8 import matplotlib as mpl
9 import matplotlib.pyplot as plt
10 import numpy as np
11 import sklearn
12 import pandas as pd
13 import os,sys,time
14 import tensorflow as tf
15 from tensorflow import keras
16 # import keras
17 def showVersion():
18     print((tf.__version__))
19     print(sys.version_info)
20     for module in mpl, np, pd, sklearn, tf, keras:
21         print(module.__name__,module.__version__)
22 
23 class_names = ['T-shirt','Trouser','Pullover','Dress',
24                'Coat', 'Sandal', 'Shirt', 'Sneaker',
25                'Bag', 'Ankle boot'
26                ]
27 
28 fashion_mnist = keras.datasets.fashion_mnist
29 (x_train_all,y_train_all),(x_test,y_test) = fashion_mnist.load_data()
30 x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
31 y_valid, y_train = y_train_all[:5000], y_train_all[5000:]
32 def showDataShape():
33     print(x_valid.shape, y_valid.shape)
34     print(x_train.shape, y_train.shape)
35     print(x_test.shape, y_test.shape)
36 def show_single_image(img_arr):
37      plt.imshow(img_arr,cmap="binary")
38      plt.show()
39 def show_images(n_rows, n_cols, x_data, y_data, class_names):
40     assert len(x_data) == len(y_data)
41     assert n_rows * n_cols < len(x_data)
42     plt.figure(figsize = (n_cols * 1.4, n_rows * 1.6))
43     for row in range(n_rows):
44         for col in range(n_cols):
45             index = n_cols * row + col
46             plt.subplot(n_rows, n_cols, index+1)
47             plt.imshow(x_data[index], cmap="binary",interpolation="nearest")
48             plt.axis("off")
49             plt.title(class_names[y_data[index]])
50     plt.show()
51 def nn():
52     model = keras.models.Sequential()   #  keras.Sequential() 好像与此一样
53     model.add(keras.layers.Flatten(input_shape=[28,28]))
54     model.add(keras.layers.Dense(300,activation="relu"))
55     model.add(keras.layers.Dense(100,activation="relu"))  # relu : y = max(0,x)
56     model.add(keras.layers.Dense(10,activation="softmax"))
57     """
58     或
59     model = keras.models.Sequential([
60         keras.layers.Flatten(input_shape=[28,28]),
61         keras.layers.Dense(300,activation="relu"),
62         keras.layers.Dense(100,activation="relu"),
63         keras.layers.Dense(10,activation="softmax")
64     ])
65     """
66     # softmax : x = [x1, x2, x3],
67     #   y = [e^x1/sum, e^x2/sum,e^x3/sum]   sum = e^x1+e^x2+e^x3
68     model.compile(loss="sparse_categorical_crossentropy",
69                   optimizer = "adam",  # 若loss太低,可能是算法的问题,换用优化过的梯度下降算法
70                   metrics = ['accuracy'])
71     # model.summary()  # 显示模型信息
72     history = model.fit(x_train,y_train,epochs=10,validation_data=(x_valid,y_valid))
73     # history.history  # 中间结果 json
74     return history
75 def plot_learning_curves(history):
76     pd.DataFrame(history.history).plot(figsize=(8,5))
77     plt.grid(True)
78     plt.gca().set_ylim(0,1)
79     plt.show()
80 if __name__ =="__main__":
81     pass
82     # show_single_image(x_train[0])
83     # showVersion()
84     # show_images(3,5,x_train,y_train,class_names)
85     history = nn()
86     plot_learning_curves(history)
87 
        2. 1 # coding:utf-8
2 # file: tf_keras_classification_model_normalizer.py
3 # author: Dean
4 # contact: 1028968939@qq.com
5 # time: 2019/12/17 11:47
6 # desc: 归一化提高准确率
7 
8 import matplotlib as mpl
9 import matplotlib.pyplot as plt
10 import numpy as np
11 import sklearn
12 import pandas as pd
13 import os,sys,time
14 import tensorflow as tf
15 from tensorflow import keras
16 def showVersion():
17     print((tf.__version__))
18     print(sys.version_info)
19     for module in mpl, np, pd, sklearn, tf, keras:
20         print(module.__name__,module.__version__)
21 
22 class_names = ['T-shirt','Trouser','Pullover','Dress',
23                'Coat', 'Sandal', 'Shirt', 'Sneaker',
24                'Bag', 'Ankle boot'
25                ]
26 
27 fashion_mnist = keras.datasets.fashion_mnist
28 (x_train_all,y_train_all),(x_test,y_test) = fashion_mnist.load_data()
29 x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
30 y_valid, y_train = y_train_all[:5000], y_train_all[5000:]
31 # 归一化处理 x = (x-u) / std   x-减去均值/方差  : 均值是0方差是1的正态分布
32 from sklearn.preprocessing import StandardScaler
33 scaler = StandardScaler()
34 # x_train:[None,28,28] ---->[None, 784] ------>[None, 28, 28]
35 x_train_scaled = scaler.fit_transform(x_train.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)
36 x_valid_scaled = scaler.transform(x_valid.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)  # 使用训练集的均值,方差
37 x_test_scaled = scaler.transform(x_test.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)
38 
39 def nn():
40     model = keras.models.Sequential()   #  keras.Sequential() 好像与此一样
41     model.add(keras.layers.Flatten(input_shape=[28,28]))
42     model.add(keras.layers.Dense(300,activation="relu"))
43     model.add(keras.layers.Dense(100,activation="relu"))  # relu : y = max(0,x)
44     model.add(keras.layers.Dense(10,activation="softmax"))
45 
46     model.compile(loss="sparse_categorical_crossentropy",
47                   optimizer = "adam",  # 若loss太低,可能是算法的问题,换用优化过的梯度下降算法
48                   metrics = ['accuracy'])
49     history = model.fit(x_train_scaled,y_train,epochs=10,validation_data=(x_valid_scaled,y_valid))
50     model.evaluate(x_test_scaled,y_test)  # 验证集验证
51     return history
52 
53 def plot_learning_curves(history):
54     pd.DataFrame(history.history).plot(figsize=(8,5))
55     plt.grid(True)
56     plt.gca().set_ylim(0,1)
57     plt.show()
58 if __name__ =="__main__":
59     pass
60     history = nn()
61     plot_learning_curves(history)
62

      2. 回调函数

        1. module: tf.keras.callbacks(本文中只有重要的几个)

          • EarlyStopping:提起终止训练

          • ModelCheckpoint:每隔一段时间保存模型

          • TensorBoard:可在训练过程中图形化显示

            1# coding:utf-8
2# file: tf_keras_classification_model_callbacks.py
3# author: Dean
4# contact: 1028968939@qq.com
5# time: 2019/12/17 11:47
6# desc: 回调函数
7
8import matplotlib as mpl
9import matplotlib.pyplot as plt
10import numpy as np
11import sklearn
12import pandas as pd
13import os,sys,time
14import tensorflow as tf
15from tensorflow import keras
16def showVersion():
17    print((tf.__version__))
18    print(sys.version_info)
19    for module in mpl, np, pd, sklearn, tf, keras:
20        print(module.__name__,module.__version__)
21
22class_names = ['T-shirt','Trouser','Pullover','Dress',
23               'Coat', 'Sandal', 'Shirt', 'Sneaker',
24               'Bag', 'Ankle boot'
25               ]
26
27fashion_mnist = keras.datasets.fashion_mnist
28(x_train_all,y_train_all),(x_test,y_test) = fashion_mnist.load_data()
29x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
30y_valid, y_train = y_train_all[:5000], y_train_all[5000:]
31
32from sklearn.preprocessing import StandardScaler
33scaler = StandardScaler()
34
35x_train_scaled = scaler.fit_transform(x_train.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)
36x_valid_scaled = scaler.transform(x_valid.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)  # 使用训练集的均值,方差
37x_test_scaled = scaler.transform(x_test.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)
38
39def nn():
40    model = keras.models.Sequential()   #  keras.Sequential() 好像与此一样
41    model.add(keras.layers.Flatten(input_shape=[28,28]))
42    model.add(keras.layers.Dense(300,activation="relu"))
43    model.add(keras.layers.Dense(100,activation="relu"))  # relu : y = max(0,x)
44    model.add(keras.layers.Dense(10,activation="softmax"))
45
46    model.compile(loss="sparse_categorical_crossentropy",
47                  optimizer = "adam",  # 若loss太低,可能是算法的问题,换用优化过的梯度下降算法
48                  metrics = ['accuracy'])
49
50    # Tensorboard,EarlyStopping,ModelCheckpoint
51    # 相对路径可能会报错，另外注意tf2.0以上版本，使用tensorboard时命令行路径不能有中文，最好是进入事件文件所在文件夹 --logdir=. 报错可能小，一般报错都是因为中文路径问题，还有就是--logdir的问题，相对，绝对路径问题(好像是这样，出错时多试试即可，宗旨，少使用中文，logdir=.)
52    logdir = r"D:\desktop\Workspace\PythonWorkSpace\Tensorflow2.0\Tensorflow2.0_谷歌\callbacks"
53    if not os.path.exists(logdir):
54        os.mkdir(logdir)
55    output_model_file = os.path.join(logdir,"fashion_mnist_model.h5")
56    callbacks = [
57        keras.callbacks.TensorBoard(logdir),
58        keras.callbacks.ModelCheckpoint(output_model_file,save_best_only = True),  # 默认保存最近一次训练,True表示保存效果最好的
59        keras.callbacks.EarlyStopping(patience = 5, min_delta = 1e-3)  # 提前结束 当阈值低于1e-3时记录一次,5次后停止
60    ]
61    history = model.fit(x_train_scaled,y_train,epochs=10,
62                        validation_data=(x_valid_scaled,y_valid),
63                        callbacks = callbacks)
64    model.evaluate(x_test_scaled,y_test)  # 验证集验证
65    return history
66
67def plot_learning_curves(history):
68    pd.DataFrame(history.history).plot(figsize=(8,5))
69    plt.grid(True)
70    plt.gca().set_ylim(0,1)
71    plt.show()
72if __name__ =="__main__":
73    history = nn()
74    plot_learning_curves(history)

    6. 图像分类,房价预测

    7. 知识点总结

      1. 分类问题,回归问题,损失函数

      2. 神经网络,激活函数,批量归一化,Dropout

        1. Min-max 归一化: $x*=\frac{x-min}{max-min}$

        2. Z-score归一化: $x*= \frac{x-\mu}{\sigma}$

        3. 批归一化:将输入的归一化扩展到每层激活值上,每层的输出是下一层的输入,都做归一化

        4. 归一化可以加速训练,<font color="red">一定程度上缓解梯度消失</font>

          1# coding:utf-8
2# file: tf_keras_classification_model_dnn.py
3# author: Dean
4# contact: 1028968939@qq.com
5# time: 2019/12/17 11:47
6# desc: 深度神经网络
7
8import matplotlib as mpl
9import matplotlib.pyplot as plt
10import numpy as np
11import sklearn
12import pandas as pd
13import os,sys,time
14import tensorflow as tf
15from tensorflow import keras
16def showVersion():
17    print((tf.__version__))
18    print(sys.version_info)
19    for module in mpl, np, pd, sklearn, tf, keras:
20        print(module.__name__,module.__version__)
21
22class_names = ['T-shirt','Trouser','Pullover','Dress',
23               'Coat', 'Sandal', 'Shirt', 'Sneaker',
24               'Bag', 'Ankle boot'
25               ]
26
27fashion_mnist = keras.datasets.fashion_mnist
28(x_train_all,y_train_all),(x_test,y_test) = fashion_mnist.load_data()
29x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
30y_valid, y_train = y_train_all[:5000], y_train_all[5000:]
31
32from sklearn.preprocessing import StandardScaler
33scaler = StandardScaler()
34
35x_train_scaled = scaler.fit_transform(x_train.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)
36x_valid_scaled = scaler.transform(x_valid.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)  # 使用训练集的均值,方差
37x_test_scaled = scaler.transform(x_test.astype(np.float32).reshape(-1,1)).reshape(-1,28,28)
38
39def nn():
40    model = keras.models.Sequential()
41    model.add(keras.layers.Flatten(input_shape=[28,28]))
42    for _ in range(20):  # 循环生成网络
43        model.add(keras.layers.Dense(100,activation="relu"))  # 若使用"selu" 是一个自带归一化的激活函数,使用它不用自己归一化
44        model.add(keras.layers.BatchNormalization())
45        """
46        对于先激活还是先归一化,不确定,好像都可以,因此下边写法也对
47        model.add(keras.layers.Dense(100))
48        model.add(keras.layers.BatchNormalization())
49        model.add(keras.layers.Activation('relu'))
50        """
51    model.add(keras.layers.AlphaDropout(rate=0.5))  # dropout层,更强大的dropout,drop后均值,方差不变,归一化性质不变  因此可与selu一起使用
52    # model.add(keras.layers.Dropout(rate=0.5))
53    model.add(keras.layers.Dense(10,activation="softmax"))
54
55    model.compile(loss="sparse_categorical_crossentropy",
56                  optimizer = "adam",
57                  metrics = ['accuracy'])
58    logdir = r"D:\desktop\Workspace\PythonWorkSpace\Tensorflow2.0\Tensorflow2.0_谷歌\dnn-callbacks"
59    if not os.path.exists(logdir):
60        os.mkdir(logdir)
61    output_model_file = os.path.join(logdir,"fashion_mnist_model.h5")
62    callbacks = [
63        keras.callbacks.TensorBoard(logdir),
64        keras.callbacks.ModelCheckpoint(output_model_file,save_best_only = True),
65        keras.callbacks.EarlyStopping(patience = 5, min_delta = 1e-3)
66    ]
67    history = model.fit(x_train_scaled,y_train,epochs=10,
68                        validation_data=(x_valid_scaled,y_valid),
69                        callbacks = callbacks)
70    model.evaluate(x_test_scaled,y_test)  # 验证集验证
71    return history
72
73def plot_learning_curves(history):
74    pd.DataFrame(history.history).plot(figsize=(8,5))
75    plt.grid(True)
76    plt.gca().set_ylim(0,3)
77    plt.show()
78if __name__ =="__main__":
79    history = nn()
80    plot_learning_curves(history)

      3. Wide,deep模型 超参数搜索

        1. 稀疏特征

          1. 离散值特征
          2. One-hot表示,则是稀疏特征
          3. Eg: 词表={'你好','你',....} 你={0,1,0,0,0,0...}
          4. 可以叉乘
          5. 优点:有效,应用于工业界
          6. 缺点:
            1. 需要人工设计
            2. 可能过拟合

        2. 密集特征

          1. 向量表达
            1. Eg: 词表={人工智能,他,你,慕课网}
            2. 他=[0.3,0.5,0.3,(维向量)]
          2. Word2vec工具
          3. 优点:
            1. 带有语义信息,不同向量之间有相关性
            2. 兼容没有出现过的特征组合
            3. 更少的人工参与
          4. 绝点:过度泛化

        4. 子类API

        5. 功能API(函数式API)

        6. 多输入与多输出

          1# coding:utf-8
2# file: tf_keras_regression_wide_deep.py
3# author: Dean
4# contact: 1028968939@qq.com
5# time: 2019/12/17 11:47
6# desc: wide && deep
7
8import matplotlib as mpl
9import matplotlib.pyplot as plt
10import numpy as np
11import sklearn
12import pandas as pd
13import os,sys,time
14import tensorflow as tf
15from tensorflow import keras
16def showVersion():
17    print((tf.__version__))
18    print(sys.version_info)
19    for module in mpl, np, pd, sklearn, tf, keras:
20        print(module.__name__,module.__version__)
21from sklearn.datasets import fetch_california_housing
22housing = fetch_california_housing()
23
24from sklearn.model_selection import train_test_split
25(x_train_all,x_test,y_train_all,y_test) = train_test_split(
26    housing.data,housing.target,random_state=7
27)
28x_train, x_valid,y_train, y_valid = train_test_split(
29    x_train_all,y_train_all,random_state=11
30)
31
32from sklearn.preprocessing import StandardScaler
33scaler = StandardScaler()
34
35x_train_scaled = scaler.fit_transform(x_train)
36x_valid_scaled = scaler.transform(x_valid)
37x_test_scaled = scaler.transform(x_test)
38
39def nn():
40    pass
41    """
42    # 函数式API
43    input = keras.layers.Input(shape=x_train.shape[1:])
44    hidden1 = keras.layers.Dense(30,activation="relu")(input)
45    hidden2 = keras.layers.Dense(30,activation="relu")(hidden1)
46
47    concat = keras.layers.concatenate([input,hidden2])  # 拼接   input 直接到输出,input经过hidden后输出   此时input 是相同的
48    output = keras.layers.Dense(1)(concat)
49    model = keras.models.Model(inputs=[input],outputs=[output])
50    """
51    """
52    # 子类API
53    class WideDeepModel(keras.models.Model):
54        def __init__(self):
55            super(WideDeepModel,self).__init__()
56            # 定义模型的层次
57            self.hidden1_layer = keras.layers.Dense(30,activation="relu")
58            self.hidden2_layer = keras.layers.Dense(30,activation="relu")
59            self.output_layer = keras.layers.Dense(1)
60        def call(self,input):
61            # 完成模型的正向计算
62            hidden1 = self.hidden1_layer(input)
63            hidden2 = self.hidden2_layer(hidden1)
64            concat = keras.layers.concatenate([input,hidden2])
65            output = self.output_layer(concat)
66            return output
67    model = WideDeepModel()  # model = keras.models.Sequential([WideDeepModel(),])
68    model.build(input_shape=(None,8))
69    """
70    """
71    # 多输入
72    input_wide = keras.layers.Input(shape=[5])
73    input_deep = keras.layers.Input(shape=[6])
74    hidden1 = keras.layers.Dense(30,activation="relu")(input_deep)
75    hidden2 = keras.layers.Dense(30,activation="relu")(hidden1)
76    concat = keras.layers.concatenate([input_wide,hidden2])   # 拼接input是不同的
77    output = keras.layers.Dense(1)(concat)
78    model = keras.models.Model(inputs=[input_deep,input_wide],
79                              outputs=[output])
80    x_train_scaled_wide = x_train_scaled[:, :5]
81    x_train_scaled_deep = x_train_scaled[:, 2:]
82    x_valid_scaled_wide = x_valid_scaled[:, :5]
83    x_valid_scaled_deep = x_valid_scaled[:, 2:]
84    x_test_scaled_wide = x_test_scaled[:, :5]
85    x_test_scaled_deep = x_test_scaled[:, 2:]
86    # fit ,evaluate 中换对应数据即可 实现多输入x_train_scaled --->[x_train_scaled_wide,x_train_scaled_deep] 还要验证输入,测试输入
87    """
88    # 多输出
89    input_wide = keras.layers.Input(shape=[5])
90    input_deep = keras.layers.Input(shape=[6])
91    hidden1 = keras.layers.Dense(30, activation="relu")(input_deep)
92    hidden2 = keras.layers.Dense(30, activation="relu")(hidden1)
93    concat = keras.layers.concatenate([input_wide, hidden2])  # 拼接input是不同的
94    output = keras.layers.Dense(1)(concat)   # 此时有两个输出
95    output2 = keras.layers.Dense(1)(hidden2)
96    model = keras.models.Model(inputs=[input_deep, input_wide],
97                               outputs=[output,output2])
98    # deep,wide 同上进行分割,注意此时fit时 y_train----->[y_train,y_train]  还有验证y_valid y_test
99    # ===================================================
100    model.summary()
101    model.compile(loss="mean_squared_error", optimizer="adam")  # 若出现梯度爆炸 可换用梯度下降算法
102    callbacks = [keras.callbacks.EarlyStopping(patience=5, min_delta=1e-4)]
103
104    history = model.fit(x_train_scaled,y_train,epochs=100,
105                        validation_data=(x_valid_scaled,y_valid),
106                        callbacks=callbacks)
107    model.evaluate(x_test_scaled,y_test)  # 验证集验证
108    return history
109
110def plot_learning_curves(history):
111    pd.DataFrame(history.history).plot(figsize=(8,5))
112    plt.grid(True)
113    plt.gca().set_ylim(0,1)
114    plt.show()
115if __name__ =="__main__":
116    history = nn()
117    plot_learning_curves(history)

        7. 超参数搜索

          1. 神经网络训练过程不变的参数

            1. 网络结构参数:几层,每层宽度,每层激活函数等

          2. 训练参数:batch_size,学习率,学习衰减算法等

          3. 手工试耗费人力

          4. 搜索策略

            1. 网格搜索
              • 定义n维方格
              • 每个方格对应一组超参数
              • 一组一组尝试
              • 例如,学习率定义m个,dropout n个,交叉m*n 次运算
            2. 随机搜索
              • 在网格搜索的网格中,随机搜索,次数变多,
            3. 遗传算法搜索
              1. 对自然界模拟
              2. A,初始化参数集合->训练->得到模型指标作为生存概率
              3. B,选择->交叉->变异->产生下一代集合
              4. C,重新到A
            4. 启发式搜索
              1. 研究热点-AutoML
              2. 使用循环神经网络生成参数
              3. 使用强化学习来进行反馈,使用模型来训练生成参数

            coding:utf-8

            file: tf_keras_regression_hp_search.py

            author: Dean

            contact: 1028968939@qq.com

            time: 2019/12/17 11:47

            desc: 超参数搜索

            import matplotlib as mpl import matplotlib.pyplot as plt import numpy as np import sklearn import pandas as pd import os,sys,time import tensorflow as tf from tensorflow import keras def showVersion(): print((tf.version)) print(sys.version_info) for module in mpl, np, pd, sklearn, tf, keras: print(module.name,module.version) from sklearn.datasets import fetch_california_housing housing = fetch_california_housing()

            from sklearn.model_selection import train_test_split (x_train_all,x_test,y_train_all,y_test) = train_test_split( housing.data,housing.target,random_state=7 ) x_train, x_valid,y_train, y_valid = train_test_split( x_train_all,y_train_all,random_state=11 )

            from sklearn.preprocessing import StandardScaler scaler = StandardScaler()

            x_train_scaled = scaler.fit_transform(x_train) x_valid_scaled = scaler.transform(x_valid) x_test_scaled = scaler.transform(x_test)

            def nn(): # 超参数搜索 """ # 自实现超参数搜索,本例简单,顺序运行,参数单一 # learning_rate:[1e-3,3e-4,1e-4,3e-3,1e-2,3e-2] learning_rate = [1e-3,3e-4,1e-4,3e-3,1e-2,3e-2] # w = w + learning_rate * grad historys =[] for lr in learning_rate: model = keras.models.Sequential([ keras.layers.Dense(30,activation="relu",input_shape=x_train.shape[1:]), keras.layers.Dense(1) ]) optimizer = keras.optimizers.Adam(lr) # lr应该是根据不同的策略会逐渐衰减的 model.compile(loss="mean_squared_error", optimizer=optimizer) callbacks = [keras.callbacks.EarlyStopping(patience=5, min_delta=1e-2)] history = model.fit(x_train_scaled,y_train,epochs=100, validation_data=(x_valid_scaled,y_valid), callbacks=callbacks) historys.append(history) """ # RandomizedSearchCV # 1,转化为sklear的model # tf.keras.wrappers.scikit_learn.KerasRegressor # tf.keras.wrappers.scikit_learn.KerasClassifier # 2,定义参数集合 # 3,搜索参数 def build_model(hidden_layers = 1,layer_size = 30, learning_rate = 3e-3): model = keras.models.Sequential() model.add(keras.layers.Dense(layer_size,activation="relu",input_shape=x_train.shape[1:])) for _ in range(hidden_layers - 1): model.add(keras.layers.Dense(layer_size,activation="relu")) model.add(keras.layers.Dense(1)) optimizer = keras.optimizers.Adam(learning_rate) model.compile(loss="mean_squared_error", optimizer=optimizer) return model sklearn_model = keras.wrappers.scikit_learn.KerasRegressor(build_model) # 传入函数名

            1from scipy.stats import reciprocal
2# f(x) = 1/(x*log(b/a))  a<=x<=b
3param_distribution = {
4    "hidden_layers": [1, 2, 3, 4],
5    "layer_size": np.arange(1, 100),  # [1,2,3....100]
6    "learning_rate": reciprocal(1e-4, 1e-2)  # 按照某种分布生成
7}
8
9from sklearn.model_selection import RandomizedSearchCV
10random_search_cv = RandomizedSearchCV(sklearn_model,
11                                      param_distribution,
12                                      cv = 5,  # 交叉验证 份数
13                                      n_iter=10,  # 随机寻找参数组合的数量，默认值为10。
14                                      n_jobs=1)  # 并行计算时使用的计算机核心数量，默认值为1。当n_jobs的值设为-1时，则使用所有的处理器。
15callbacks = [keras.callbacks.EarlyStopping(patience=5, min_delta=1e-2)]
16history = random_search_cv.fit(x_train_scaled, y_train, epochs=100,   # 还使用fit
17                        validation_data=(x_valid_scaled, y_valid),
18                        callbacks=callbacks)
19# cross_validation:交叉验证 ,训练集分为n份 n-1份训练,1份测试
20random_search_cv.best_params_  # 最好的参数
21random_search_cv.best_score_  # 最好的参数对应的分数
22model = random_search_cv.best_estimator_.model  # 最好的模型
23model.evaluate(x_test_scaled,y_test)  # 测试
24
25return historys

            def plot_learning_curves(history): pd.DataFrame(history.history).plot(figsize=(8,5)) plt.grid(True) plt.gca().set_ylim(0,1) plt.show() if name =="main": historys = nn() for history in historys: print() plot_learning_curves(history)

  3. Tensorflow基础API使用

    1. 概要

      1. Tf框架:基础数据类型,自定义模型与损失函数,自定义求导,tf.function,图结构\
      2. 项目:图像分类,房价预测

    2. 知识点

      1. 基础API
      2. 基础API与keras的集成
        1. 自定义损失函数
        2. 自定义层次
      3. @tf.function的使用(2.0专有:将python转化为图结构)
      4. 自定义求导

    3. @tf.function

      1. 将python函数编译为图
      2. 易于将模型导出为GraphDef+checkpoint 或者SavedModel
      3. 使得eager execution可以默认打开
      4. 1.0的代码可以通过tf.function在2.0继续使用
        1. 代替session

    4. API

      1. 基础数据类型

        1. Tf.constant,tf.string

          1. 1 # 0维 一个数 shape=()
2 # 1维 列表 shape=(n)
3 # 2维 二维数组 shape=(m,n)
4 t = tf.constant([
5     [1,2,3],
6     [4,5,6]
7 ])  # 2.0中可以直接获取值
8 print(t)
9 print(t[:,1:])
10 print(t[...,2]) # 一维向量 tf.Tensor([3 6], shape=(2,), dtype=int32)
11 print(t+10)  # 都加上10
12 print(tf.square(t))  # 每个数平方
13 print(t @ tf.transpose(t))  # 返回t与它转置的乘积 2*2
14 print(t.numpy())  # 直接转化为numpy对象 2*3矩阵
15 
          2. 1 t = tf.constant("tensorflow")
2 print(t)  # tf.Tensor(b'tensorflow', shape=(), dtype=string)
3 print(tf.strings.length(t))  # tf.Tensor(10, shape=(), dtype=int32)
4 print(tf.strings.length(t,unit="UTF8_CHAR"))  # tf.Tensor(10, shape=(), dtype=int32)
5 print(tf.strings.unicode_decode(t,"utf8"))  # tf.Tensor([116 101 110 115 111 114 102 108 111 119], shape=(10,), dtype=int32)
6 t = tf.constant(['cafe','coffee','咖啡'])  #
7 print(tf.strings.length(t,unit="UTF8_CHAR"))  # tf.Tensor([4 6 2], shape=(3,), dtype=int32)
8 print(tf.strings.unicode_decode(t,"utf8"))  # <tf.RaggedTensor [[99, 97, 102, 101], [99, 111, 102, 102, 101, 101], [21654, 21857]]>
9

        2. tf.ragged.constant,tfSpareTensor

          1. 1 v = tf.Variable([[1,2,3,4],[5,6,7,8]])
2 print(v)  # Variable 对象
3 print(v.value())  # tensor
4 print(v.numpy())  # numpy矩阵
5 # 重新赋值,不能用等于号
6 v.assign(2*v)
7 v[0,1].assign(42)
8 v[1].assign([9,10,11,12])
9 
          2. 1 t = tf.ragged.constant([[11,12],[13,14,15]])
2 print(t)  # <tf.RaggedTensor [[11, 12], [13, 14, 15]]>
3 print(t[1])  # tf.Tensor([13 14 15], shape=(3,), dtype=int32)
4 print(t[1:2])  # <tf.RaggedTensor [[13, 14, 15]]>
5 print(tf.concat([t,t],axis=0))  # 按照行拼接变为4行
6 print(tf.concat([t,t],axis=1))  # 若要按照列拼接,首先行数要相同
7 print(t.to_tensor())  # 变为普通tensor 空的位置补为0,0都在正常值后边
8 
          3. 1 t = tf.SparseTensor(indices=[[0,1],[1,0],[2,3]],  # 注意必须先[0,1] 再[0,2] 否则to_dense 会报错,若必须不按顺序 tf.sparse.reorder(t) 即可使用
2                     values=[1,2,3],
3                     dense_shape=[3,4])
4 print(t)  # 存储稀疏矩阵,指定值位置,数值,shape即可
5 print(tf.sparse.to_dense(t))  # 转换为普通tensor
6 # 不能加法
7 t2 = tf.constant("..")  # 普通的4*3 Tensor
8 tf.sparse.sparse_dense_matmul(t,t2)  # 得到的是3*3普通Tensor
9

      2. 自定义损失函数------Tf.reduce_mean

        1 def customized_mse(y_true, y_pred):
2        return tf.reduce_mean(tf.square(y_pred - y_true))
3 model.compile(loss=customized_mse, optimizer="adam",metrics=["mean_squared_error"])

      3. 自定义层次-----Keras.layers.Lambda和继承法

        1. 1 layer = tf.keras.layers.Dense(100,input_shape=[None,5])
2 layer(tf.zero([10,5]))   # 输出[10,100] 二维
3 # [10,5] * w  +b = [10,100]
4 # w  [5:100]
5 layer.trainable_variables # 获得kernel 与bias 可以查看
6 # ---------------------------------------------
7  class CustomizedDenseLayer(keras.layers.Layer):
8         def __init__(self,units,activation=None,**kwargs):
9             self.units = units
10             self.activation = keras.layers.Activation(activation)
11             super(CustomizedDenseLayer, self).__init__(**kwargs)
12         def build(self,input_shape):
13             """构建需要的参数"""
14             # x * w +b [None,a] w[a,b] [None,b]
15             self.kernel = self.add_weight(name="kernel",
16                                           shape=(input_shape[1],self.units),
17                                           initilizer="uniform", # 定义随机初始化kernel的方法:此处使用均匀分布
18                                           trainable=True)
19             self.bias = self.add_weight(name="bias",
20                                         shape=(self.units,),
21                                         initilizer="zeros",
22                                         trainable=True)
23             super(CustomizedDenseLayer,self).build(input_shape)
24         def call(self,x):
25             """完整的正向计算"""
26             return self.activation(x @ self.kernel + self.bias)  # @ 表示矩阵乘法
27 
28     model = keras.models.Sequential([
29         CustomizedDenseLayer(30, activation="relu", input_shape=x_train.shape[1:]),
30         CustomizedDenseLayer(1)
31     ])
32 
        2. 1  # 定义简单的层次,激活函数层,dropout层等
2     #   eg:tf.nn.softplus: log(1+e^x)
3     # customized_softplus = keras.layers.Dense(1,activation="softplus") = keras.layers.Dense(1),keras.layers.Activation("softplus")
4     customized_softplus = keras.layers.Lambda(lambda x:tf.nn.softplus(x))
5

      4. Tf.function

        1. Tf.fucntion,tf.autograph.to_code,

          1. 1 # python ----->图
2 #   法一
3 def scaled_elu(z,scale=1.0,alpha=1.0):   # python函数
4     # z>=0?scale *z :scale * alpha * tf.nn.elu(z)
5     is_positive = tf.greater_equal(z,0.0)
6     return scale * tf.where(is_positive,z,alpha * tf.nn.elu(z))
7 print(scaled_elu(tf.constant(-3)))   #   常量
8 print(scaled_elu(tf.constant([-3,-2.5])))  # 列表向量  都何以全部接受并处理
9 scaled_elu_tf = tf.function(scaled_elu)
10 scaled_elu_tf.python_function  # 返回原来的python函数
11 print(scaled_elu(tf.constant(-3)))
12 print(scaled_elu(tf.constant([-3,-2.5]))) # 与上边结果相同,转换的作用是速度加快
13 #   法二
14 #   1+1/2+....1/2^n
15 @tf.function
16 def converge_to(n_iters):
17     total = tf.constant(0.)
18     increment = tf.constant(1.)
19     for _ in range(n_iters):
20         total += increment
21         increment /= 2.0
22     return total
23 print(converge_to(20))
24 
25 
          2. 1 def display_tf_code(func):
2     # 中间代码
3     # python代码转为tf代码, 图就是通过此转换的
4     code = tf.autograph.to_code(func)   # 还有to_graph 是将代码转为图的
5     from IPython.display import display,Markdown
6     display(Markdown('```python\n{}\n```'.format(code)))
7 display_tf_code(scaled_elu)
8 
          3. 1 var = tf.Variable(0.)
2 @tf.function
3 def add_21():
4     return var.assign_add(21)
5 print(add_21())  # 结果返回21的tensor   ,若var在内部会报错, 神经网络中大多是变量,需要在外边初始化
6 
7 @tf.function(input_signature=[tf.TensorSpec([None],tf.int32,name='x')])
8 def cube(z):  # 可接收浮点数,整数  使用输入签名后会限制只能输入int32
9     return tf.pow(z,3)
10 print(cube(tf.constant([1.2,2.6])))
11 print(cube(tf.constant([1,2])))
12 
          4. 1 # 只有经过输入签名的才能保存为Saved_Model,在这个过程中使用get_concrete_function,把tf.function标注的转换为有图建议的函数
2 cube_func_int32 = cube.get_concrete_function(tf.TensorSpec([None],tf.int32))
3 
4 print(cube_func_int32 is cube.get_concerte_function(tf.TensorSpec([2],tf.int32)))
5

      5. GraphDef

        1. get_operations,get_operation_by_name

        2. get_tensorf_by_name,as_graph_def

          1cube_func_int32.graph  # 图
2cube_func_int32.graph.get_operations()  # 获取操作
3cube_func_int32.graph.get_operations()[2]  # 获取某一个操作
4cube_func_int32.graph.get_operations()[2].xxx  # 获取某一个操作中的某个属性
5
6cube_func_int32.graph.get_operation_by_name("operationName")  # 通过operation名字获取
7cube_func_int32.graph.get_tensor_by_name("x:0")  # 通过tensor名字获取
8
9cube_func_int32.graph.as_graph_def()  # 显示图的结构信息
10
11# 主要用来保存图结构,模型,恢复图结构,模型

      6. 自动求导

        1. 普通求导方法

          1def f(x):
2    return 3. * x ** 2 + 2. * x -1
3def approximae_derivative(f,x,eps= 1e-3):  # 求函数 f 在 x 点的导数
4    # x点向右eps, 向左eps 中间直线的斜率近似导数, eps足够小,导数足够接近
5    return (f(x+eps)-f(x-eps))/(2*eps)
6# print(approximae_derivative(f,1))
7
8
9def g(x1,x2):
10    return (x1 + 5)*(x2 ** 2)
11def approximae_gredient(g,x1,x2,eps=1e-3):
12    dg_x1 = approximae_derivative(lambda x:g(x,x2),x1,eps)
13    dg_x2 = approximae_derivative(lambda x:g(x1,x),x2,eps)
14    return dg_x1,dg_x2
15# print(approximae_gredient(g,2,3))

        2. Tf.GrandientTape

          1. 1 x1 = tf.Variable(2.0)
2 x2 = tf.Variable(3.0)
3 with tf.GradientTape(persistent = True) as tape:
4     z = g(x1,x2)
5 
6 dz_x1 = tape.gradient(z,x1)  # 9.0
7 dz_x2 = tape.gradient(z,x2)   # tape 只能使用一次,本次会报错 GradientTape(persistent=Ture) 表示不释放tape就可以多次使用,但使用完毕后需要手动释放那个资源
8 print(dz_x1,dz_x2)
9 del tape
10 
          2. 1 with tf.GradientTape(persistent = True) as tape: # 不用persistent
2     z = g(x1,x2)
3 dz_x1,dz_x2 = tape.gradient(z,[x1,x2])   # 也可以这样一次求出两个
4 
          3. 1 # 当x1,x2 为tf.constant()时,返回值为None
2 with tf.GradientTape(persistent = True) as tape:  # 这样修改即可
3     tape.watch(x1)
4     tape.watch(x2)
5     z = g(x1,x2)
6 
          4. 1 x = tf.Variable(5.0)
2 with tf.GradientTape() as tape:
3     z1 = 3 * x
4     z2 = x **2
5 tape.gradient([z1,z2],x)  # z1对于x的导数加上z2对于x的导数
6 
          5. 1 # 二阶导数
2 x1 = tf.Variable(2.0)
3 x2 = tf.Variable(3.0)
4 with tf.GradientTape(persistent=True) as outer_tape:
5     with tf.GradientTape(persistent=True) as inner_tape:
6         z = g(x1,x2)
7     inner_grads = inner_tape.gradient(z,[x1,x2])
8 outer_grads = [outer_tape.gradient(inner_grad,[x1,x2])
9                for inner_grad in inner_grads]
10 print(outer_grads)
11 del inner_tape
12 del outer_tape
13 
          6. 1 # 简单梯度下降
2 learning_rate = 0.1
3 x = tf.Variable(0.0)
4 for _ in range(100):
5     with tf.GradientTape() as tape:
6         z = 3. * x ** 2 + 2. * x -1
7     dz_dx = tape.gradient(z,x)
8     x.assign_sub(learning_rate*dz_dx)
9 print(x) # -0.333333
10

        3. Optimizer.apply_gradients

          1. 1 # 优化梯度下降
2 learning_rate = 0.1
3 x = tf.Variable(0.0)
4 optimizer = keras.optimizers.SGD(lr = learning_rate)
5 for _ in range(100):
6     with tf.GradientTape() as tape:
7         z = 3. * x ** 2 + 2. * x -1
8     dz_dx = tape.gradient(z,x)
9     optimizer.apply_gradients([(dz_dx,x)])  # 与上边不同
10 print(x) # -0.333333
11 
          2. 1 def test():
2     metric = keras.metrics.MeanSquaredError()  # 第一个形参是真实值,第二个是预测值  返回的是平均 均方误差
3     print(metric([5],[2]))  # 9
4     print(metric([0],[1]))  # 5
5     print(metric.result())  # 5
6     metric.reset_statues()  # 不再累加
7     print(metric([1],[3]))  # 4
8 
          3. 1 def nn(): 
2     # fit 的内容
3     #   1,batch 遍历训练集 metric
4     #       1.1,自动求导
5     #   2,epoch结束 验证集 metric
6     epochs = 100
7     batch_size = 32
8     steps_per_epoch = len(x_train_scaled) // batch_size
9     optimizer = keras.optimizers.SGD()
10     metric = keras.metrics.MeanSquaredError()
11 
12     def random_batch(x,y,batch_size=32):
13         idx = np.random.randint(0,len(x),size=batch_size)  # 在0到 len(x) 中随机取batch_size个数
14         return x[idx],y[idx]
15     model = keras.models.Sequential([
16         keras.layers.Dense(30,activation="relu",input_shape=x_train.shape[1:]),
17         keras.layers.Dense(1),
18     ])
19     for epoch in range(epochs):
20         metric.reset_states()
21         for step in range(steps_per_epoch):
22             x_batch,y_batch = random_batch(x_train_scaled,y_train,batch_size)  # 获取数据
23             with tf.GradientTape() as  tape:
24                 y_pred = model(x_batch)  # 计算预测
25                 loss = tf.reduce_mean(keras.losses.mean_squared_error(y_batch,y_pred))  # 定义损失函数
26                 metric(y_batch,y_pred)
27             grads = tape.gradient(loss,model.variables)
28             grads_and_vars = zip(grads,model.variables)   # 每个参数对应他的梯度
29             optimizer.apply_gradients(grads_and_vars)  # 将梯度的变化应用到变量上
30             print("\rEpoch",epoch,"train mse:",metric.result().numpy(),end="")
31         y_valid_pred = model(x_valid_scaled)
32         valid_loss = tf.reduce_mean(
33             keras.losses.mean_squared_error(y_valid_pred,y_valid)
34         )
35         print("\t","valid mse",valid_loss.numpy())
36

  4. Tensorflow dataset使用

    1. 基础知识

      1. Tf框架:

        1. 基础api

          1. tf.data.Dateset.from_tensor_slics

            1dataset = tf.data.Dataset.from_tensor_slices((np.arange(10)))
2print(dataset)  # <TensorSliceDataset shapes: (), types: tf.int32>    # 里边的元素每一个为一组

          2. repeat,batch,interleave,map,shuffle,list_files

            1for item in dataset.batch(2):  # 还是10打个数,但此时每组2个,5组
2     print(item)
3
4
5dataset = dataset.repeat(3)  # dataset 变为30个数,30组
6dataset =dataset.batch(7)   # 每组7个数 5组
7for item in dataset:
8    print(item)
9
10
11# interleave: 将dataset中的每一个数据处理后,合并返回
12# case:dataset 中是一系列文件的名字,通过他,遍历名字读取内容,使用它合并
13dataset2 = dataset.interleave(
14    lambda v:tf.data.Dataset.from_tensor_slices(v),  # map_fn:处理函数
15    cycle_length=5, # cycle_length:并行处理个数
16    block_length=5 # block_length: 从处理的元素中每次取多少个出来 [0,1,2,3,4,5,6]  只取[0,1,2,3,4]  最后不够的时候 从第一个中未取到的取
17)
18for item in dataset2:
19    print(item)
20
21
22x = np.array([[1,2],[3,4],[5,6]])
23y = np.array(['cat','dog','fox'])
24
25dataset3 = tf.data.Dataset.from_tensor_slices((x,y))
26for item_x,item_y in dataset3:
27    print(item_x.numpy(),item_y.numpy())
28"""
29[1 2] b'cat'
30[3 4] b'dog'
31[5 6] b'fox'
32"""
33dataset4 = tf.data.Dataset.from_tensor_slices({'feature':x,"label":y})
34for item in dataset4:
35    print(item['feature'].numpy(),item['label'].numpy())  # 同上

        2. csv文件,

          1. 生成csv

            1import matplotlib as mpl
2import matplotlib.pyplot as plt
3import numpy as np
4import sklearn
5import pandas as pd
6import os,sys,time
7import tensorflow as tf
8from tensorflow import keras
9def showVersion():
10    print((tf.__version__))
11    print(sys.version_info)
12    for module in mpl, np, pd, sklearn, tf, keras:
13        print(module.__name__,module.__version__)
14from sklearn.datasets import fetch_california_housing
15housing = fetch_california_housing()
16
17from sklearn.model_selection import train_test_split
18(x_train_all,x_test,y_train_all,y_test) = train_test_split(
19    housing.data,housing.target,random_state=7
20)
21x_train, x_valid,y_train, y_valid = train_test_split(
22    x_train_all,y_train_all,random_state=11
23)
24
25from sklearn.preprocessing import StandardScaler
26scaler = StandardScaler()
27
28x_train_scaled = scaler.fit_transform(x_train)
29x_valid_scaled = scaler.transform(x_valid)
30x_test_scaled = scaler.transform(x_test)
31output_dir = r"generate_csv"
32if not os.path.exists(output_dir):
33    os.mkdir(output_dir)
34def save_to_csv(output_dir,data,name_prefix,header=None,n_parts=10):
35    path_format = os.path.join(output_dir,"{}_{:02d}.csv")  # 文件名格式:第一个表示是train还是test 第二个表示为2位的整数
36    filenames= []
37    for file_idx,row_indics in enumerate(  # 将数据分为索引加数据 通过idx indics获取
38            np.array_split( # 将索引分为n_parts部分  [array(1,2,3,4,), array(5,6,7,8..)]
39                np.arange(len(data)),  # 生成和data一样长的数组,当索引
40                         n_parts)   # 将索引分为n_patrs
41            ):
42        """
43        file_idx row_indics
44        0       [0,1,2,3,4..]
45        1       [5,6,7,8,9..]
46        2       ...
47        """
48        print(name_prefix,file_idx)
49        part_csv = path_format.format(name_prefix,file_idx)
50        filenames.append(part_csv)
51        with open(part_csv,"wt",encoding="utf-8") as f:
52            if header is not None:
53                f.write(header+"\n")  # 写入header
54            for row_index in row_indics:
55                f.write(",".join([repr(col) for col in data[row_index]]))
56                f.write('\n')
57    return filenames
58train_data = np.c_[x_train_scaled,y_train]  # 将数据和标签合并
59valid_data = np.c_[x_valid_scaled,y_valid]
60test_data = np.c_[x_test_scaled,y_test]
61header_cols = housing.feature_names + ['MidianHouseValue']
62header_str = ','.join(header_cols)
63train_filenames = save_to_csv(output_dir,train_data,"train",header_str,n_parts=10)

          2. tf.data.TextLineDataset

          3. tf.io.decode_csv

            1# 1. filename ---> dataset
2# 2. read file --> dataset -> datasets ->merge
3# 3, parse csv
4filename_dataset = tf.data.Dataset.list_files(train_filenames)
5n_readers =5
6dataset = filename_dataset.interleave(
7    lambda filename:tf.data.TextLineDataset(filename).skip(1),  # 根据文件名按照行读取文件内容, 并跳过header
8    cycle_length=n_readers
9)
10for line in dataset.take(15):  # 读取数据的前15行
11    print(line.numpy())  #  是一个整字符串
12# tf.io.decode_csv(str,record_defaults)
13sample_str = '1,2,3,4,5'
14records_defaults = [tf.constant(0,dtype=tf.int32)]*5 # 要解析的类型
15records_defaults = [tf.constant(0,dtype=tf.int32),0,np.nan,"hello",tf.constant([])] # 要解析的类型
16parsed_fields = tf.io_csv(sample_str,records_defaults)
17print(parsed_fields)
18
19def parse_csv_line(line,n_fileds=9):
20    defs = [tf.constant(np.nan)] * n_fileds
21    parsed_fields = tf.io.decode_csv(line,record_defaults=defs)
22    x = tf.stack(parsed_fields[0:-1])
23    y = tf.stack(parsed_fields[-1:])
24    return x,y
25
26
27def csv_reader_dataset(filenames,n_readers=5,
28                       batch_size=32,n_parse_threads=5,  # 解析时并行数
29                       shuffle_buffer_size=10000):  # buffer大小
30    dataset = tf.data.Dataset.list_files(filenames)
31    dataset = dataset.repeat()  # 重复无限次
32    dataset = dataset.interleave(
33        lambda filename: tf.data.TextLineDataset(filename).skip(1),
34        cycle_length= n_readers,
35    )
36    dataset.shuffle(shuffle_buffer_size)  # 混排
37    dataset = dataset.map(parse_csv_line,num_parallel_calls=n_parse_threads)  # 将数据经过处理后返回 与interleave类似
38    dataset = dataset.batch(batch_size)
39    return dataset
40train_set = csv_reader_dataset(train_filenames,batch_size=32)
41valid_set = csv_reader_dataset(train_filenames,batch_size=32)  # 换位valid_filenames
42model = keras.models.Sequential([
43        keras.layers.Dense(30, activation="relu", input_shape=x_train.shape[8]),
44        keras.layers.Dense(1)
45    ])
46model.compile(loss='mse', optimizer="adam")
47callbacks = [keras.callbacks.EarlyStopping(patience=5, min_delta=1e-4)]
48history = model.fit(train_set, epochs=100,
49                    steps_per_epoch = 1160//32,  # 指定每个epoch 的次数
50                    validation_steps =3870//32,
51                    validation_data=valid_set,
52                    callbacks=callbacks)
53# model.evaluate(test_set,steps = 5160//32)

        3. tfrecord文件

          1. tf.train.FloatList, tf.train.Int64List, tf.train.bytesList

            1# tfrecord 文件格式
2# -> tf.train.Example
3#   -> tf.train.Features ->{'key':tf.train.Feature}

          ->tf.train.Feature ->{tf.train.ByteList/FloatList/Int64List}

          favorite_books = [name.encode("utf-8") for name in ["machine learning","cc150"]] favorite_books_bytelist = tf.train.BytesList(value = favorite_books) """ value: "machine learning" value: "cc150" """ hours_floatlist = tf.train.FloatList(value=[15.5,9.0,7.0,8.0]) age = tf.train.Int64List(value=[64])

          12. tf.train.Feature,    tf.train.Features,    tf.train.Example
2
3```python
4features = tf.train.Features(
5    feature = {
6        "favorite_books":tf.train.Feature(bytes_list = favorite_books_bytelist),
7        "hours":tf.train.Feature(float_list = hours_floatlist),
8        "age":tf.train.Feature(int64_list = age)
9    }
10)
11print(features)  # json格式,显示每个feature
12"""
13feature {
14  key: "age"
15  value {
16    int64_list {
17      value: 64
18    }
19  }
20}
21faeture{....}
22"""
23example = tf.train.Example(features=features)  # 与features 类似   featrues{feature{}...}

          1. exapmle.SerializeToString

            serizlized_example = example.SerializeToString()  # 压缩,

          2. tf.io.ParseSingleExample

            1output_dir = "tf_tfrecord_basic"
2if not os.path.exists(output_dir):
3    os.mkdir(output_dir)
4filename = "test.tfrecords"
5filename_fullpath = os.path.join(output_dir,filename)
6# 写入文件
7with tf.io.TFRecordWriter(filename_fullpath) as writer:  # 打开文件
8    for i in range(3):  # 写进去3次
9        writer.write(serizlized_example)
10# 读取
11dataset = tf.data.TFRecordDataset([filename_fullpath])
12for seralized_example in dataset:
13    print(serizlized_example)   # 与上边的serialized_example 类似  压缩过的
14

          3. tf.io.VarLenFeature,tf.io.FixedLenFeature

            1expected_features = {
2    "favorite_books": tf.io.VarLenFeature(dtype=tf.string),   # 变长  一会解析后是sparsetensor
3    "hours":tf.io.VarLenFeature(dtype=tf.float32),  # 变长
4    "age":tf.io.FixedLenFeature([],dtype=tf.int64),  # 定长,普通的tensor,[] 表示是0维的数,[8] 就表示8特特征
5}

          4. tf.data.TFRecordDataset,tf.io.TFRecordOptions

            1dataset = tf.data.TFRecordDataset([filename_fullpath])
2for serialized_example_tensor in dataset:
3    example = tf.io.parse_single_example(
4        serialized_example_tensor,
5        expected_features
6    )
7    books = tf.sparse.to_dense(example["favorite_books"],default_value=b"")  # 解析sparsetensor 为tensor  sparsetensor为0的地方 不能转为字符串 要制定default_value
8
9    print(example)
10
11#  存为压缩格式
12filename_fullpath_zip = filename_fullpath+ ".zip"
13options = tf.io.TFRecordOptions(compress_type = "GZIP")
14with tf.io.TFRecordWriter(filename_fullpath_zip,options) as writer:
15    pass
16
17# 读取压缩格式
18dataset = tf.data.TFRecordDataset([filename_fullpath_zip],compression_type="GZIP")  # 其余不变即可
19

      2. 房价csv转record

        1# coding:utf-8
2# file: tf_data_generate_tfrecord.py
3# author: Dean
4# contact: 1028968939@qq.com
5# time: 2019/12/22 11:29
6# desc:
7# coding:utf-8
8# file: tf_keras_classification_model_dnn.py
9# author: Dean
10# contact: 1028968939@qq.com
11# time: 2019/12/17 11:47
12# desc: csv 文件转换为record
13
14import matplotlib as mpl
15import matplotlib.pyplot as plt
16import numpy as np
17import sklearn
18import pandas as pd
19import os,sys,time
20import tensorflow as tf
21from tensorflow import keras
22source_dir = "generate_csv"
23def get_filenames_by_prefix(source_dir,prefix_name):  # 按照文件名分类
24    all_files = os.listdir(source_dir)
25    results=[]
26    for filename in all_files:
27        if filename.startswith(prefix_name):
28            results.append(os.path.join(source_dir,filename))
29    return results
30train_filenames = get_filenames_by_prefix(source_dir,"train")
31import pprint
32# pprint.pprint(train_filenames)  # 换行打印
33
34def parse_csv_line(line,n_fileds=9):
35    defs = [tf.constant(np.nan)] * n_fileds
36    parsed_fields = tf.io.decode_csv(line,record_defaults=defs)
37    x = tf.stack(parsed_fields[0:-1])
38    y = tf.stack(parsed_fields[-1:])
39    return x,y
40def csv_reader_dataset(filenames,n_readers=5,
41                       batch_size=32,n_parse_threads=5,  # 解析时并行数
42                       shuffle_buffer_size=10000):  # buffer大小
43    dataset = tf.data.Dataset.list_files(filenames)
44    dataset = dataset.repeat()  # 重复无限次
45    dataset = dataset.interleave(
46        lambda filename: tf.data.TextLineDataset(filename).skip(1),
47        cycle_length= n_readers,
48    )
49    dataset.shuffle(shuffle_buffer_size)  # 混排
50    dataset = dataset.map(parse_csv_line,num_parallel_calls=n_parse_threads)  # 将数据经过处理后返回 与interleave类似
51    dataset = dataset.batch(batch_size)
52    return dataset
53train_set = csv_reader_dataset(train_filenames,batch_size=32)
54
55# 遍历数据写入文件
56def serialize_example(x,y):
57    """Converts x ,y to tf.train.FloatList and serialize"""
58    input_features = tf.train.FloatList(value=x)
59    label = tf.train.FloatList(value=y)
60    features = tf.train.Features(
61        feature={
62            "input_features":tf.train.Feature(float_list = input_features),
63            "label":tf.train.Feature(float_list = label)
64        }
65    )
66    example = tf.train.Example(features = features)
67    return example.SerializeToString()
68def csv_dataset_to_tfrecords(base_filename,dataset,n_shards,steps_per_shard,compression_type = None):
69    """
70    n_shard:文件个数
71    steps_per_shard:每个小文件,走多少步
72    """
73    options = tf.io.TFRecordOptions(compression_type=compression_type)
74    all_filenames=[]
75    for shard_id in range(n_shards):
76        filename_fullpath = '{}_{:05d}-of-{:05d}'.format(
77            base_filename,shard_id,n_shards
78        )
79        with tf.io.TFRecordWriter(filename_fullpath,options) as writer:
80            for x_batch,y_batch in dataset.take(steps_per_shard):  #  每个batch都是32个的
81                for x_example,y_example in zip(x_batch,y_batch):
82                    writer.write(serialize_example(x_example,y_example))
83        all_filenames.append(filename_fullpath)
84    return all_filenames
85n_shards = 20
86train_step_per_shard = 11610//32//n_shards
87output_dir = "generate_tfreocrds"  # 先创建
88train_basename = os.path.join(output_dir,"train")
89
90train_tfrecord_filename = csv_dataset_to_tfrecords(
91    train_basename,train_set,n_shards,train_step_per_shard,None
92)
93
94
95# 读取文件
96expected_feature = {
97    "input_ferature":tf.io.FixedLenFeature([8],dtype=tf.float32),
98    "label":tf.io.FixedLenFeature([1],dtype=tf.float32)
99}
100def parse_example(serialized_example):
101    example = tf.io.parse_single_example(serialize_example,expected_feature)
102    return expected_feature["input_ferature"],expected_feature['label']
103def tfrecord_reader_dataset(filenames,n_readers=5,
104                       batch_size=32,n_parse_threads=5,  # 解析时并行数
105                       shuffle_buffer_size=10000):  # buffer大小
106    dataset = tf.data.Dataset.list_files(filenames)
107    dataset = dataset.repeat()  # 重复无限次
108    dataset = dataset.interleave(
109        lambda filename: tf.data.TFRecordDataset(filename,compression_type=nONE),
110        cycle_length= n_readers,
111    )
112    dataset.shuffle(shuffle_buffer_size)  # 混排
113    dataset = dataset.map(parse_example,num_parallel_calls=n_parse_threads)  # 将数据经过处理后返回 与interleave类似
114    dataset = dataset.batch(batch_size)
115    return dataset
116tfrecords_train = tfrecord_reader_dataset(train_tfrecord_filename,
117                                          batch_size=3)
118for  x_batch,y_batch in tfrecords_train:
119    print(x_batch,y_batch)

  5. Tensorflow Estimator使用与tf1.0

    1. 知识点

      Tf框架:estimator使用,特征列使用,tf1.0基本使用

      项目:泰坦尼克号生存

    2. API

      1. Tf.keras.estimator.to_estimator

        1. Train,evaluate

          1# dataset 泰坦尼克号
2# https://storage.googleapis.com/tf-datasets/titanic/train.csv
3# https://storage.googleapis.com/tf-datasets/titanic/eval.csv
4
5train_file = "csv/train.csv"
6eval_file = "csv/eval.csv"
7
8train_df = pd.read_csv(train_file)  # (627,9)
9eval_df = pd.read_csv(eval_file)  # (264.9)
10
11y_train = train_df.pop("survived")
12y_eval = eval_df.pop("survived")
13# print(train_df)
14# print(y_train.head())
15# print(train_df.describe()) 非离散值的信息
16# train_df.age.hist(bins = 20)   画出age的分布直方图 bin表示分为多少份
17# train_df.sex.value_counts().plot(kind="barh")  # 横向柱状图 "barv"纵向: 统计男性个数,女性个数
18# train_df['class'].value_counts().plot(kind="barh") 不同舱位人数
19# 男性,女性中获救比例,
20# pd.concat([train_df,y_train],axis=1).groupby('sex').survived.mean()
21
22# 离散值
23categorical_columns = ["sex",'n_siblings_spouses','parch','class','deck','embark_town','alone']
24# 连续纸
25numeric_columns = ['age','fare']
26feature_columns =[]
27for categorical_column in categorical_columns:
28    vocab = train_df[categorical_column].unique()
29    feature_columns.append(
30        tf.feature_column.indicator_column(
31            tf.feature_column.categorical_column_with_vocabulary_list(
32                categorical_column,vocab)))
33for categorical_column in numeric_columns:
34    feature_columns.append(
35        tf.feature_column.numeric_column(
36            categorical_column,dtype=tf.float32
37        )
38    )
39# 构件dataset的函数
40def make_dataset(data_df,label_df,epochs=10,shuffle=True,batch_size=32):
41    dataset = tf.data.Dataset.from_tensor_slices(
42        (dict(data_df),label_df)
43    )
44    if shuffle:
45        dataset =dataset.shuffle(10000)
46    dataset = dataset.repeat(epochs).batch(batch_size)
47    return dataset
48train_dataset = make_dataset(train_df,y_train,batch_size=5)
49
50# keras.layers.DenseFeature
51tf.keras.backend.set_floatx('float64')
52# for x,y in train_dataset.take(1):
53#     age_column = feature_columns[7]
54#     gender_column = feature_columns[0]
55#     print(keras.layers.DenseFeatures(age_column)(x).numpy())  # 连续值不变
56#     print(keras.layers.DenseFeatures(gender_column)(x).numpy())  # 离散值变为one-hot
57# for x,y in train_dataset.take(1):
58#     print(keras.layers.DenseFeatures(feature_columns)(x).numpy())
59
60model = keras.models.Sequential([
61    keras.layers.DenseFeatures(feature_columns),
62    keras.layers.Dense(100,activation="relu"),
63    keras.layers.Dense(100,activation="relu"),
64    keras.layers.Dense(2,activation="softmax")
65])
66model.compile(loss="sparse_categorical_crossentropy",
67              optimizer=keras.optimizers.SGD(lr=0.01),
68              metrics = ['accuracy'])
69# 1,model.fit
70# 2.model-->estimator ->train
71# 第一种
72# train_dataset = make_dataset(train_df,y_train,epochs=100)
73# eval_dataset = make_dataset(eval_df,y_eval,epochs=1,shuffle=False)
74# model.fit(train_dataset,
75#           validation_data = eval_dataset,
76#           steps_per_epoch = 20,
77#           validation_steps=8,
78#           epochs=100)
79# 第二种
80estimator = keras.estimator.model_to_estimator(model)
81# input_fn:函数或lambda  return  (feature,label)    或dataset--->(feature,label)
82estimator.train(input_fn = lambda :make_dataset(train_df,y_train,epochs=100))    # 会出现bug

      2. Tf.estimator.BaselineClassifier

      3. Tf.estimator.LinerClassifier

      4. Tf.estimator.DNNClassifier

      5. Tf.feature_column

        1. categorical_column_with_vocabulary_list
        2. numeric_column
        3. indicator_column
        4. cross_column

      6. keras.layers.DenseFeatures

        1train_file = "csv/train.csv"
2eval_file = "csv/eval.csv"
3train_df = pd.read_csv(train_file)  # (627,9)
4eval_df = pd.read_csv(eval_file)  # (264.9)
5y_train = train_df.pop("survived")
6y_eval = eval_df.pop("survived")
7
8
9# 离散值
10categorical_columns = ["sex",'n_siblings_spouses','parch','class','deck','embark_town','alone']
11# 连续纸
12numeric_columns = ['age','fare']
13feature_columns =[]
14for categorical_column in categorical_columns:
15    vocab = train_df[categorical_column].unique()
16    feature_columns.append(
17        tf.feature_column.indicator_column(
18            tf.feature_column.categorical_column_with_vocabulary_list(
19                categorical_column,vocab)))
20for categorical_column in numeric_columns:
21    feature_columns.append(
22        tf.feature_column.numeric_column(
23            categorical_column,dtype=tf.float32
24        )
25    )
26# 构件dataset的函数
27def make_dataset(data_df,label_df,epochs=10,shuffle=True,batch_size=32):
28    dataset = tf.data.Dataset.from_tensor_slices(
29        (dict(data_df),label_df)
30    )
31    if shuffle:
32        dataset =dataset.shuffle(10000)
33    dataset = dataset.repeat(epochs).batch(batch_size)
34    return dataset
35# =================
36# output_dir = "baseline_model"
37# if not os.path.exists(output_dir):
38#     os.mkdir(output_dir)
39# baseline_estimator = tf.estimator.BaselineClassifier(model_dir=output_dir,
40#                                                      n_classes=2)
41# baseline_estimator.train(input_fn=lambda :make_dataset(train_df,y_train,epochs=100))
42# baseline_estimator.evaluate(input_fn=lambda :make_dataset(eval_df,y_eval,epochs=1,shuffle=False,batch_size=20))
43# =================
44# linear_output_dir = "liner_model"
45# if not os.path.exists(linear_output_dir):
46#     os.mkdir(linear_output_dir)
47# linear_estimator = tf.estimator.LinearClassifier(
48#     model_dir=linear_output_dir,
49#     n_classes=2,
50#     feature_columns=feature_columns
51# )
52# linear_estimator.train(input_fn=lambda :make_dataset(
53#     train_df,y_train,epochs=100))
54# =====================================
55dnn_output_dir = "dnn_model"
56if not os.path.exists(dnn_output_dir):
57    os.mkdir(dnn_output_dir)
58dnn_estimator = tf.estimator.DNNClass ifier(
59    model_dir=dnn_output_dir,
60    n_classes=2,
61    feature_columns=feature_columns,
62    hidden_units=[128,128], # 两层,都是128
63    activation_fn=tf.nn.relu,
64    optimizer="Adam"
65)
66dnn_estimator.train(input_fn= lambda :make_dataset(train_df,y_train,epochs=100))
67dnn_estimator.evaluate(input_fn= lambda :make_dataset(
68    eval_df,y_eval,epochs=1,shuffle=False))

  6. 卷积神经网络

    1. 知识点

      1. Tf框架:卷积实现

        卷积网络:卷积+池化---全连接 ------>分类任务

        全卷积网络:去掉了最后的全连接层,变为反卷积层(是尺寸变大) ----->物体分割

      2. 项目:图像分类,kaggle 10monkeys,kaggle cifar10

      3. 理论:卷积,数据增强,迁移学习

      4. 卷积网问题:

        1. 参数太多
          1. 局部连接, 图像区域性
          2. 参数共享 图像特征与位置无关
        2. 池化时,剩余的数据会丢弃

    2. keras实现卷积神经网络

      1. 1 # coding:utf-8
2 # file: tf_keras_classification_model_cnn.py
3 # author: Dean
4 # contact: 1028968939@qq.com
5 # time: 2019/12/17 11:47
6 # desc: 深度神经网络
7 
8 import matplotlib as mpl
9 import matplotlib.pyplot as plt
10 import numpy as np
11 import sklearn
12 import pandas as pd
13 import os,sys,time
14 import tensorflow as tf
15 from tensorflow import keras
16 def showVersion():
17     print((tf.__version__))
18     print(sys.version_info)
19     for module in mpl, np, pd, sklearn, tf, keras:
20         print(module.__name__,module.__version__)
21 
22 # class_names = ['T-shirt','Trouser','Pullover','Dress',
23 #                'Coat', 'Sandal', 'Shirt', 'Sneaker',
24 #                'Bag', 'Ankle boot'
25 #                ]
26 
27 fashion_mnist = keras.datasets.fashion_mnist
28 (x_train_all,y_train_all),(x_test,y_test) = fashion_mnist.load_data()
29 x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
30 y_valid, y_train = y_train_all[:5000], y_train_all[5000:]
31 
32 from sklearn.preprocessing import StandardScaler
33 scaler = StandardScaler()
34 
35 x_train_scaled = scaler.fit_transform(x_train.astype(np.float32).reshape(-1,1)).reshape(-1,28,28,1)
36 x_valid_scaled = scaler.transform(x_valid.astype(np.float32).reshape(-1,1)).reshape(-1,28,28,1)  # 使用训练集的均值,方差
37 x_test_scaled = scaler.transform(x_test.astype(np.float32).reshape(-1,1)).reshape(-1,28,28,1)
38 
39 def nn():
40     model = keras.models.Sequential()
41     model.add(keras.layers.Conv2D(filters=32,  # 卷积核数量,
42                                   kernel_size=3,  # 大小
43                                   padding="same",  # 是否填充是的输入输出大小一样
44                                   activation="relu",  # 使用selu效果会更好
45                                   input_shape=(28,28,1)
46                                 ))
47     model.add(keras.layers.Conv2D(filters=32,kernel_size=3,
48                                   padding="same",
49                                   activation="relu"))
50     model.add(keras.layers.MaxPool2D(pool_size=2))   # 一般步长与大小相同  pool 后的卷积层filter一般会翻倍
51     model.add(keras.layers.Conv2D(filters=64, kernel_size=3,
52                                   padding="same",
53                                   activation="relu"))
54     model.add(keras.layers.Conv2D(filters=64, kernel_size=3,
55                                   padding="same",
56                                   activation="relu"))
57     model.add(keras.layers.MaxPool2D(pool_size=2))
58     model.add(keras.layers.Conv2D(filters=128,kernel_size=3,
59                                   padding="same",
60                                   activation="relu"))
61     model.add(keras.layers.Conv2D(filters=128, kernel_size=3,
62                                   padding="same",
63                                   activation="relu"))
64     model.add(keras.layers.MaxPool2D(pool_size=2))
65     model.add(keras.layers.Flatten())  # 将输入一维化
66     model.add(keras.layers.Dense(128,activation="relu"))
67     model.add(keras.layers.Dense(10,activation="softmax"))
68     model.summary()
69     model.compile(loss="sparse_categorical_crossentropy",
70                   optimizer = "adam",
71                   metrics = ['accuracy'])
72     logdir = "cnn-callbacks"
73     if not os.path.exists(logdir):
74         os.mkdir(logdir)
75     output_model_file = os.path.join(logdir,"fashion_mnist_model.h5")
76     callbacks = [
77         keras.callbacks.TensorBoard(logdir),
78         keras.callbacks.ModelCheckpoint(output_model_file,save_best_only = True),
79         keras.callbacks.EarlyStopping(patience = 5, min_delta = 1e-3)
80     ]
81     history = model.fit(x_train_scaled,y_train,epochs=10,
82                         validation_data=(x_valid_scaled,y_valid),
83                         callbacks = callbacks)
84     model.evaluate(x_test_scaled,y_test)  # 验证集验证
85     return history
86 
87 def plot_learning_curves(history):
88     pd.DataFrame(history.history).plot(figsize=(8,5))
89     plt.grid(True)
90     plt.gca().set_ylim(0,3)
91     plt.show()
92 if __name__ =="__main__":
93     history = nn()
94     plot_learning_curves(history)
95

    3. keras实现深度可分离卷积

      1. 标准卷积

      2. 卷积核大小($D_k$)

      3. 卷积核输入通道数M

      4. 卷积核个数N,输出通道N

      5. 图像大小$(D_F)$

        $D_k_D_k_M_N_D_F*D_F$

      6. 深度可分离卷积

        (此时的卷积核输入时原图n*n*3,卷积核为k*k*1,为了计算原图所有通道,该卷积核个数就是输入通道数M) $D_k_D_k_1_M_D_F_D_F+1_1_M_N_D_F_D_F$

        参考:

        https://blog.csdn.net/xiewenbo/article/details/84315560

        https://blog.csdn.net/qq_21997625/article/details/87106152

      7. 1  # 代码同上
2  # 第一个Conv2D不变
3  # 其余的Conv2D修改为SeparableConv2D即可
4  #  计算量少
5 
6 # 两层 3*3 视野域就是 一个5*5
7 
8 # 计算量几乎缩小n倍
9 
10 # 深度可分离卷积
11 
12

    4. kera实战kaggle

      1. 10 monkey,cifar10

        1# coding:utf-8
2# file: 1_10_monkey_model.py
3# author: Dean
4# contact: 1028968939@qq.com
5# time: 2019/12/23 11:30
6# desc: keras_generator 搭建模型
7#    model.fit  将数据放入内存训练
8#    model.fit_generator   处理大数据,  使用生成器的数据集
9
10import matplotlib as mpl
11import matplotlib.pyplot as plt
12import numpy as np
13import sklearn
14import pandas as pd
15import os,sys,time
16import tensorflow as tf
17from tensorflow import keras
18
19
20train_dir=r"I:\人工智能数据\10_monkey\training\training"
21valid_dir=r"I:\人工智能数据\10_monkey\validation\validation"
22label_txt=r"I:\人工智能数据\10_monkey\monkey_labels.txt"
23
24labels = pd.read_csv(label_txt,header=0)
25
26height = 128  # 缩放大小
27width = 128
28channels = 3
29batch_size = 64
30num_classes = 10
31
32# keras中的数据集:读取数据,数据分享
33
34train_datagen = keras.preprocessing.image.ImageDataGenerator(
35    rescale = 1./255,  # 像素点缩放到0-1
36    rotation_range = 40,  # 图片增强方式,随机旋转40
37    width_shift_range = 0.2,  # 偏移
38    height_shift_range = 0.2,  #
39    shear_range = 0.2,  # 剪切强度
40    zoom_range = 0.2,  # 缩放程度
41    horizontal_flip = True,  # 水平翻转
42    fill_mode = "nearest",  # 放大时填充像素,选择最近的填充
43)
44
45# 读取图片,按照上边方法进行处理
46train_generator = train_datagen.flow_from_directory(train_dir,
47                                                    target_size=(height,width),
48                                                    batch_size=batch_size,
49                                                    seed=7,  # 随机数
50                                                    shuffle=True,
51                                                    class_mode="categorical")  # label格式 one-hot
52valid_datagen = keras.preprocessing.image.ImageDataGenerator(rescale=1./255)
53valid_generator = valid_datagen.flow_from_directory(valid_dir,
54                                                    target_size=(height,width),
55                                                    batch_size=batch_size,
56                                                    seed=7,
57                                                    shuffle=False,  # 不用训练不用打乱
58                                                    class_mode="categorical")
59train_num = train_generator.samples
60valid_num = valid_generator.samples
61# print(train_num,valid_num) #  1098 272
62# x,y = train_generator.next()  # (64,128,128,3) (64,10)
63
64model = keras.models.Sequential([
65    keras.layers.Conv2D(filters=32,kernel_size=3,padding="same",
66                        activation="relu",input_shape=(width,height,channels)),
67    keras.layers.Conv2D(filters=32,kernel_size=3,padding="same",
68                        activation="relu"),
69    keras.layers.MaxPool2D(pool_size=2),
70    keras.layers.Conv2D(filters=64,kernel_size=3,padding="same",
71                        activation="relu"),
72    keras.layers.Conv2D(filters=64,kernel_size=3,padding="same",
73                        activation="relu"),
74    keras.layers.MaxPool2D(pool_size=2),
75    keras.layers.Conv2D(filters=128,kernel_size=3,padding="same",
76                        activation="relu"),
77    keras.layers.Conv2D(filters=128,kernel_size=3,padding="same",
78                        activation="relu"),
79    keras.layers.MaxPool2D(pool_size=2),
80    keras.layers.Flatten(),
81    keras.layers.Dense(128,activation="relu"),
82    keras.layers.Dense(num_classes,activation="softmax")
83])
84model.compile(loss="categorical_crossentropy",
85              optimizer="adam",metrics=['accuracy'])
86model.summary()
87
88epochs = 300
89# 最后acc 97% 98%
90history = model.fit_generator(train_generator,
91                              steps_per_epoch= train_num//batch_size,
92                              epochs = epochs,
93                              validation_data=valid_generator,
94                              validation_steps= valid_num//batch_size)
95# history.history.keys()  # loss,acc, val_loss,val_acc
96
97def plot_learning_curves(history,label,epochs,min_value,max_value):
98    data={}
99    data[label] = history.history[label]
100    data["val_"+label]=history.history["val_"+label]
101    pd.DataFrame(data).plot(figsize=(8.5))
102    plt.grid(True)
103    plt.axis([0,epochs,min_value,max_value])
104    plt.show()
105plot_learning_curves(history,"acc",epochs,0,1)
106plot_learning_curves(history,"loss",epochs,1.5,2.5)

      2. 数据增强(调整数据 提高准确率)与迁移学习

        1. 1 # model 部分修改即可
2 resnet50_fine_tune = keras.models.Sequential();
3 resnet50_fine_tune.add(keras.applications.ResNet50(include_top=False,   # 不包含最后一层
4                                                    pooling = 'avg',
5                                                    weights="imagenet"))  # None 从头开始,imagenet接着
6 resnet50_fine_tune.add(keras.layers.Dense(num_classes,activation="softmax"))
7 resnet50_fine_tune.layers[0].trainable= False  # 设置第一层的参数不调整
8 """
9 resnet50 = keras.applications.ResNet50(include_top=False,
10                                        pooling="avg",
11                                        weights="imagenet")
12 resnet50.summary()
13 for layer in resnet50.layers[0:-5]:  # 设置resnet50中后0--5层也不可训练,其余可训练
14     layer.trainable=False
15 resnet50_new = keras.models.Sequential([
16     resnet50,
17     keras.layers.Dense(num_classes,activation="softmax"),
18 ])
19 """
20 
21
22 # coding:utf-8
23 # file: 1_cifar10_model.py
24 # author: Dean
25 # contact: 1028968939@qq.com
26 # time: 2019/12/23 11:30
27 # desc:
28 import matplotlib as mpl
29 import matplotlib.pyplot as plt
30 import numpy as np
31 import sklearn
32 import pandas as pd
33 import os,sys,time
34 import tensorflow as tf
35 from tensorflow import keras
36 # data https://www.kaggle.com/c/cifar-10/data
37 class_names = [
38     'airplane',
39     'automobile',
40     'bird',
41     'cat',
42     'deer',
43     'dog',
44     'frog',
45     'horse',
46     'ship',
47     'truck'
48 ]
49 train_lables_file = r'I:\人工智能数据\cifar10\trainLabels.csv'
50 test_csv_file = r'I:\人工智能数据\cifar10\sampleSubmission.csv'
51 train_folder = r'I:\人工智能数据\cifar10\train'
52 test_floder = r'I:\人工智能数据\cifar10\test'
53 def parse_csv_file(filepath,folder):
54     """Paeses csv files into (filename(path),label) format  (xxx.png,'cat')"""
55     results =[]
56     with open(filepath,"r") as f:
57         lines = f.readlines()[1:]  # 去掉header
58         for line in lines:
59             image_id,label_str = line.strip("\n").split(',')
60             image_full_path = os.path.join(folder,image_id+".png")
61             results.append((image_full_path,label_str))
62     return results
63 train_lables_info = parse_csv_file(train_lables_file,train_folder)  # 50000
64 test_csv_info = parse_csv_file(test_csv_file,test_floder)  # 300000
65 
66 # 制作dataframe
67 # train_df = pd.DataFrame(train_lables_info)
68 train_df = pd.DataFrame(train_lables_info[0:45000])
69 valid_df = pd.DataFrame(train_lables_info[45000:])
70 test_df = pd.DataFrame(test_csv_info)
71 
72 # 设置dataframe的列名
73 train_df.columns = ["filepath",'class']
74 valid_df.columns = ["filepath",'class']
75 test_df.columns = ["filepath",'class']
76 
77 
78 height = 32  # 缩放大小
79 width = 32
80 channels = 3
81 batch_size = 32
82 num_classes = 10
83 
84 # keras中的数据集:读取数据,数据分享
85 
86 train_datagen = keras.preprocessing.image.ImageDataGenerator(
87     rescale = 1./255,  # 像素点缩放到0-1
88     rotation_range = 40,  # 图片增强方式,随机旋转40
89     width_shift_range = 0.2,  # 偏移
90     height_shift_range = 0.2,  #
91     shear_range = 0.2,  # 剪切强度
92     zoom_range = 0.2,  # 缩放程度
93     horizontal_flip = True,  # 水平翻转
94     fill_mode = "nearest",  # 放大时填充像素,选择最近的填充
95 )
96 
97 # 读取图片,按照上边方法进行处理
98 train_generator = train_datagen.flow_from_dataframe(
99     train_df,
100     directory="./",
101     x_col="filepath",
102     y_col="class",
103     classes=class_names,
104     target_size=(height, width),
105     batch_size=batch_size,
106     seed=7,
107     shuffle=True,
108     class_mode="sparse"
109 )
110 valid_datagen = keras.preprocessing.image.ImageDataGenerator(rescale=1./255)
111 valid_generator = valid_datagen.flow_from_dataframe(valid_df,
112                                                     directory="./",
113                                                     x_col="filepath",
114                                                     y_col="class",
115                                                     classes=class_names,
116                                                     target_size=(height,width),
117                                                     batch_size=batch_size,
118                                                     seed=7,
119                                                     shuffle=False,
120                                                     class_mode="sparse")
121 train_num = train_generator.samples
122 valid_num = valid_generator.samples
123 # print(train_num,valid_num) #  1098 272
124 # x,y = train_generator.next()  # (64,128,128,3) (64,10)
125 
126 model = keras.models.Sequential([
127     keras.layers.Conv2D(filters=128,kernel_size=3,padding="same",
128                         activation="relu",input_shape=(width,height,channels)),
129     keras.layers.BatchNormalization(),
130     keras.layers.Conv2D(filters=128,kernel_size=3,padding="same",
131                         activation="relu"),
132     keras.layers.BatchNormalization(),
133     keras.layers.MaxPool2D(pool_size=2),
134     keras.layers.Conv2D(filters=256,kernel_size=3,padding="same",
135                         activation="relu"),
136     keras.layers.BatchNormalization(),
137     keras.layers.Conv2D(filters=256,kernel_size=3,padding="same",
138                         activation="relu"),
139     keras.layers.BatchNormalization(),
140     keras.layers.MaxPool2D(pool_size=2),
141     keras.layers.Conv2D(filters=512,kernel_size=3,padding="same",
142                         activation="relu"),
143     keras.layers.BatchNormalization(),
144     keras.layers.Conv2D(filters=512,kernel_size=3,padding="same",
145                         activation="relu"),
146     keras.layers.BatchNormalization(),
147     keras.layers.MaxPool2D(pool_size=2),
148     keras.layers.Flatten(),
149     keras.layers.Dense(512,activation="relu"),
150     keras.layers.Dense(num_classes,activation="softmax")
151 ])
152 model.compile(loss="sparse_categorical_crossentropy",
153               optimizer="adam",metrics=['accuracy'])
154 model.summary()
155 
156 epochs = 1
157 history = model.fit_generator(train_generator,
158                               steps_per_epoch= train_num//batch_size,
159                               epochs = epochs,
160                               validation_data=valid_generator,
161                               validation_steps= valid_num//batch_size)
162 
163 def plot_learning_curves(history,label,epochs,min_value,max_value):
164     data={}
165     data[label] = history.history[label]
166     data["val_"+label]=history.history["val_"+label]
167     pd.DataFrame(data).plot(figsize=(8.5))
168     plt.grid(True)
169     plt.axis([0,epochs,min_value,max_value])
170     plt.show()
171 plot_learning_curves(history,"acc",epochs,0,1)
172 plot_learning_curves(history,"loss",epochs,0,2)
173

  7. 循环神经网络

    1. Tf框架:LSTM实现
    2. 项目:文本分类,文本生成,Kaggle文本分类
    3. 理论:序列式问题,循环网络,LSTM,双向LSTM

  8. Tensorflow分布式

    1. 理论部分

      1. GPU设置
        1. 默认用全部GPU并占满内存
        2. 如何不浪费内存和计算资源
          1. 内存增长
          2. 虚拟设备机制
        3. 多GPU
          1. 虚拟GPU&实际GPU
          2. 手工设置&分布式机制
        4. API
          1. tfdebugging.set_log_device_placement
          2. tf.config.experimental.set_visible_devices 设置可见设备
          3. tf.config.experimental.list_logical_devices 获取逻辑设备
          4. tf.config.experimental.list_physical_devices 获取物理设备
          5. tf.config.experimental.set_memory——growth 设置内存自增
          6. tf.config.experimental.VirtualDeviceConfiguration 建立逻辑设备
          7. tf.config.set_soft_device_placement 自动分配任务到设备
      2. 分布式策略

    2. 实战部分

      1. GPU设置部分
      2. 分布式训练

  9. Tensorflow模型保存于部署

    1. Tf框架:模型保存,导出tflite,部署

    2. 项目:图像分类

    3. 模型保存

      1. 文件格式

        1. Ckeckpoint与graphdef (tf1.0)
        2. keras(hdf5),SavedModel(tf2.0)(参数+网络结构)

      2. 保存的是什么

        1. 参数
        2. 参数+网络结构

      3. TFLite

        1. TFLite Converter
          1. 模型转化 将上边的三种模型转化为tflite
        2. TFLite Interpreter
          1. 模型加载
          2. 支持android与ios
          3. 支持多种语言
        3. TFLite-FlatBuffer
          1. google开源的跨平台数据序列化库
          2. 优点
            1. 直接读取序列化数据
            2. 高效内存使用和速度
            3. 灵活,数据前后向兼容,灵活控制数据结构
            4. 代码少
            5. 强类型数据
        4. TFlite-量化
          1. 参数从float变为8bit整数
            1. 准确率损失
            2. 模型大小变为1/4
          2. 量化方法
            1. float=(int-归零点)*因子

      4. 案例

        1. keras-保存参数与保存模型+参数

          1keras.callbacks.ModelCheckpoint(out_put_dir,
2                               save_best_only=True,
3                               save_weights_only=False) # 为false表示只保存参数,True表示都保存
4keras.models.load_model(out_put_dir) # 若都保存了,从新加载后和可直接使用
5
6
7# 定义模型之后,导入weights 可使用
8
9
10# 也可保存权重
11model.save_weights("xxxx/xxx.h5") 
12
13
14tf.saved_model.save(model,".../xxx_garph")
15# assets
16# variables checkpoint 信息
17# saved_model.pb 
18
19
20# 命令行工具查看model信息
21!saved_model_cli show --dir "path" --all
22# 载入model
23model = tf.saved_model.load("path")
24inference = model.signatures["serving_default"]
25results = inference(tf.constant(x_test_scaled[0:1]))

        2. keras签名函数保存到SavedModel

          1tf_export =tf.Model()
2tf_export.cube = 签名函数
3tf.saved_model.save(tf._export,dir)

        3. keras,SavedModel,签名函数到具体函数

          1load_saved_model=keras.models.load_model(".../...h5")
2load_saved_model(np.zeros([1,28,28]))
3
4run_model = tf.function(lambda x:load_saved_model(x))
5keras_concrete_function = run_model.get_concrete_function(
6    tf.TensorSpec(
7    	load_keras_model.inputs[0].shape,
8        load_keras_model.inputs[0].dtype
9    ))
10# 测试
11keras_concrete_function(tf.constant(np.zeros(1,28,28,dtype=np.float32)))

        4. keras,SavedModel,具体函数到tflite

        5. tflite量化

        6. tensorflow js.android部署

  10. 机器翻译于tensorflow2tensor使用 ========================

1.  Tf框架:transformer实现,tensor2tensor使用

12.  项目:机器翻译
23.  理论:序列道序列模型.注意力机制,可缩放点积注意力,多头注意力
点赞
收藏

评论区

加载中...

相关推荐

MySQL:[Err] 1292 - Incorrect datetime value: ‘0000-00-00 00:00:00‘ for column ‘CREATE_TIME‘ at row 1

文章目录问题用navicat导入数据时，报错：原因这是因为当前的MySQL不支持datetime为0的情况。解决修改sql\mode:sql\mode:SQLMode定义了MySQL应支持的SQL语法、数据校验等，这样可以更容易地在不同的环境中使用MySQL。全局s

Oracle 分组与拼接字符串同时使用

SELECTT.,ROWNUMIDFROM(SELECTT.EMPLID,T.NAME,T.BU,T.REALDEPART,T.FORMATDATE,SUM(T.S0)S0,MAX(UPDATETIME)CREATETIME,LISTAGG(TOCHAR(

MySQL部分从库上面因为大量的临时表tmp_table造成慢查询

背景描述Time:20190124T00:08:14.70572408:00User@Host:@Id:Schema:sentrymetaLast_errno:0Killed:0Query_time:0.315758Lock_

皕杰报表之UUID

​在我们用皕杰报表工具设计填报报表时，如何在新增行里自动增加id呢？能新增整数排序id吗？目前可以在新增行里自动增加id，但只能用uuid函数增加UUID编码，不能新增整数排序id。uuid函数说明：获取一个UUID，可以在填报表中用来创建数据ID语法：uuid()或uuid(sep)参数说明：sep布尔值，生成的uuid中是否包含分隔符''，缺省为

手写Java HashMap源码

HashMap的使用教程HashMap的使用教程HashMap的使用教程HashMap的使用教程HashMap的使用教程22

2020年前端实用代码段，为你的工作保驾护航

有空的时候，自己总结了几个代码段，在开发中也经常使用，谢谢。1、使用解构获取json数据let jsonData  id: 1,status: "OK",data: 'a', 'b';let  id, status, data: number   jsonData;console.log(id, status, number )