环境：ubuntu18 + nvidia 430 + cuda 10.0 + cudnn7.6.0 + tensorflow-gpu 2.0.0
调用 layers.Conv2D() 就报错，报错信息：
1Epoch 1/5 22019-10-11 21:50:00.814925: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 32019-10-11 21:50:01.026836: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 42019-10-11 21:50:01.472727: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 52019-10-11 21:50:01.476545: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 62019-10-11 21:50:01.476593: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. 7[[{{node sequential/conv2d/Conv2D}}]] 864/60000 [..............................] - ETA: 16:19Traceback (most recent call last): 9File "/media/dxs/E/Project/AI/PycharmProject/deepLearningProject-201903/tf2_demo/mnist_cnn.py", line 53, in <module> 10model.fit(train_images, train_labels, epochs=5,batch_size=64) 11File "/home/dxs/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 728, in fit 12use_multiprocessing=use_multiprocessing) 13File "/home/dxs/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 324, in fit 14total_epochs=epochs) 15File "/home/dxs/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 123, in run_one_epoch 16batch_outs = execution_function(iterator) 17File "/home/dxs/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 86, in execution_function 18distributed_function(input_fn)) 19File "/home/dxs/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__ 20result = self._call(*args, **kwds) 21File "/home/dxs/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 520, in _call 22return self._stateless_fn(*args, **kwds) 23File "/home/dxs/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1823, in __call__ 24return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access 25File "/home/dxs/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1141, in _filtered_call 26self.captured_inputs) 27File "/home/dxs/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1224, in _call_flat 28ctx, args, cancellation_manager=cancellation_manager) 29File "/home/dxs/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 511, in call 30ctx=ctx) 31File "/home/dxs/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute 32six.raise_from(core._status_to_exception(e.code, message), None) 33File "<string>", line 3, in raise_from 34tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. 35[[node sequential/conv2d/Conv2D (defined at home/dxs/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1751) ]] [Op:__inference_distributed_function_1000] 36 37Function call stack: 38distributed_function
尝试过 升级cuda到10.1 还是报错，经过一番查找，发现只要在开头设置下 就可以了， 在开头加上以下语句：
1# # ===================================================================== 2# # 不加这几句，则CONV 报错 3# physical_devices = tf.config.experimental.list_physical_devices('GPU') 4# assert len(physical_devices) > 0, "Not enough GPU hardware devices available" 5# tf.config.experimental.set_memory_growth(physical_devices[0], True) 6# # =========================================================================
发个完整代码：
1# -*- coding: utf-8 -*- 2# @Time : 2019/10/10 下午10:39 3# @Author : dxs 4# @Email : dangxusheng163163.com 5# @File : mnist_cnn.py 6# @Project : PycharmProject 7 8from __future__ import absolute_import, division, print_function, unicode_literals 9 10import os 11import os.path as osp 12import tensorflow as tf 13from tensorflow import keras 14from tensorflow.keras import datasets, layers, models 15 16import cv2 17import numpy as np 18import matplotlib.pyplot as plt 19 20# 21# ===================================================================== 22# 不加这几句，则CONV 报错 23physical_devices = tf.config.experimental.list_physical_devices('GPU') 24assert len(physical_devices) > 0, "Not enough GPU hardware devices available" 25tf.config.experimental.set_memory_growth(physical_devices[0], True) 26# ========================================================================= 27 28 29 30(train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data() 31 32train_images = train_images.reshape((60000, 28, 28, 1)) 33test_images = test_images.reshape((10000, 28, 28, 1)) 34 35# 特征缩放[0, 1]区间 36train_images, test_images = train_images / 255.0, test_images / 255.0 37 38model = models.Sequential() 39model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1))) 40model.add(layers.MaxPooling2D((2, 2))) 41model.add(layers.Conv2D(64, (3, 3), activation='relu')) 42model.add(layers.MaxPooling2D((2, 2))) 43model.add(layers.Conv2D(64, (3, 3), activation='relu')) 44model.add(layers.Flatten()) 45model.add(layers.Dense(64, activation='relu')) 46model.add(layers.Dense(10, activation='softmax')) 47model.summary() # 显示模型的架构 48 49model.compile(optimizer='adam', 50 loss='sparse_categorical_crossentropy', 51 metrics=['accuracy']) 52 53model.fit(train_images, train_labels, epochs=5,batch_size=64)
输出结果如下：
12019-10-11 21:54:15.109707: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 22019-10-11 21:54:15.109714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 32019-10-11 21:54:15.109717: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 42019-10-11 21:54:15.109766: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 52019-10-11 21:54:15.109991: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 62019-10-11 21:54:15.110211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5562 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1660, pci bus id: 0000:01:00.0, compute capability: 7.5) 7Model: "sequential" 8_________________________________________________________________ 9Layer (type) Output Shape Param # 10================================================================= 11conv2d (Conv2D) (None, 26, 26, 32) 320 12_________________________________________________________________ 13max_pooling2d (MaxPooling2D) (None, 13, 13, 32) 0 14_________________________________________________________________ 15conv2d_1 (Conv2D) (None, 11, 11, 64) 18496 16_________________________________________________________________ 17max_pooling2d_1 (MaxPooling2 (None, 5, 5, 64) 0 18_________________________________________________________________ 19conv2d_2 (Conv2D) (None, 3, 3, 64) 36928 20_________________________________________________________________ 21flatten (Flatten) (None, 576) 0 22_________________________________________________________________ 23dense (Dense) (None, 64) 36928 24_________________________________________________________________ 25dense_1 (Dense) (None, 10) 650 26================================================================= 27Total params: 93,322 28Trainable params: 93,322 29Non-trainable params: 0 30_________________________________________________________________ 31Train on 60000 samples 32Epoch 1/5 332019-10-11 21:54:15.965416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 342019-10-11 21:54:16.172605: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 352019-10-11 21:54:17.364824: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Not found: ./bin/ptxas not found 36Relying on driver to perform ptx compilation. This message will be only logged once. 3760000/60000 [==============================] - 9s 145us/sample - loss: 0.1734 - accuracy: 0.9475 38Epoch 2/5 3960000/60000 [==============================] - 8s 127us/sample - loss: 0.0504 - accuracy: 0.9844 40Epoch 3/5 4160000/60000 [==============================] - 7s 124us/sample - loss: 0.0365 - accuracy: 0.9886 42Epoch 4/5 4360000/60000 [==============================] - 7s 113us/sample - loss: 0.0290 - accuracy: 0.9902 44Epoch 5/5 4560000/60000 [==============================] - 7s 110us/sample - loss: 0.0229 - accuracy: 0.9929 46 47Process finished with exit code 0