Есть ли в Ubuntu 20.04 взлом, позволяющий избавиться от следующей ошибки CUDA нехватки памяти без перезапуска машины?
RuntimeError: CUDA out of memory. Пытался выделить 40.00 MiB (GPU 0; Общая емкость 7,80 ГиБ; 6,34 ГиБ уже выделено; 32,44 МиБ бесплатно; 6.54 ГиБ зарезервировано PyTorch)
Я понимаю, что следующее работает, но также убивает мою записную книжку Jupyter. Есть ли способ освободить память в GPU, не убивая ноутбук Jupyter?
(base) mona@mona:~/research/facial_landmark$ nvidia-smi
Tue Oct 6 20:28:05 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 2070 Off | 00000000:01:00.0 Off | N/A |
| N/A 47C P8 9W / N/A | 7883MiB / 7982MiB | 2% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1306 G /usr/lib/xorg/Xorg 255MiB |
| 0 N/A N/A 1743 G /usr/bin/gnome-shell 151MiB |
| 0 N/A N/A 3273 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 3359 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 3844 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 4222 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 4587 C ...mona/anaconda3/bin/python 7459MiB |
+-----------------------------------------------------------------------------+
(base) mona@mona:~/research/facial_landmark$ kill -9 4587
(base) mona@mona:~/research/facial_landmark$ nvidia-smi
Tue Oct 6 20:28:24 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 2070 Off | 00000000:01:00.0 Off | N/A |
| N/A 47C P8 9W / N/A | 433MiB / 7982MiB | 4% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1306 G /usr/lib/xorg/Xorg 255MiB |
| 0 N/A N/A 1743 G /usr/bin/gnome-shell 152MiB |
| 0 N/A N/A 3273 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 3359 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 3844 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 4222 G /usr/lib/firefox/firefox 2MiB |
+-----------------------------------------------------------------------------+
(base) mona@mona:~/research/facial_landmark$