Why does spacy keeps gpu memory in use when the language object is deleted?

[samehraban]

How to reproduce the behaviour

import spacy
import gc
import cupy as cp

print(cp.get_default_memory_pool().used_bytes())
# 0
spacy.require_gpu()
spacy.load('/tmp/nlp')
print(cp.get_default_memory_pool().used_bytes())
# 2700994560
gc.collect()
print(cp.get_default_memory_pool().used_bytes())
# 2700994560

Your Environment

• spaCy version: 2.3.2
• Platform: Linux-4.4.0-112-generic-x86_64-with-Ubuntu-16.04-xenial
• Python version: 3.6.9

[honnibal]

GPU memory management is a bit more complicated, and I'm not 100% sure of all the details. I think you might need to tell cupy to clear its memory pools as well?

The test we care most about is whether, if you do the above allocation and destruction in a loop, you actually run out of memory. That would definitely indicate a problem.

[samehraban]

@honnibal No, it won't run out of memory. It goes about 5.5GB in second iteration and if I tell cupy to clean memory pool, it gets back to the 2.7GB again.