Dependency Matcher freezes when large files are loaded in memory

[probavee]

How to reproduce the behaviour

I have a spacy pipeline with many components. Many of those components use data from dict. (this is important)
Every now and then the component which handle dependency matches freezes and instead of taking 1 second takes 5, and then back to 1 second.
So I was able to reproduce with this minimal code:
(the "large-file.json" I used to test is here: https://github.com/json-iterator/test-data/blob/master/large-file.json , but you can load your own as long as it takes enough memory)

import json
from copy import deepcopy
from datetime import datetime as dt

import spacy
from spacy.language import Language
from spacy.matcher import DependencyMatcher

with open("large-file.json") as f:
    test_sample = json.load(f)
test = [deepcopy(test_sample) for i in range(30)] # if you comment this, there is no freeze
# test = "a" * 2516582400 * 2 # if you load this in memory instead of the json it does not freeze either

class Matching:
    def __init__(self, nlp):
        self.nlp = nlp
        self.matcher = DependencyMatcher(nlp.vocab, validate=True)
        patterns = {
            0: [
                [
                    {
                        "RIGHT_ID": "head",
                        "RIGHT_ATTRS": {"POS": {"IN": ["NOUN", "PROPN", "ADJ"]}},
                    },
                    {
                        "LEFT_ID": "head",
                        "REL_OP": ".",
                        "RIGHT_ID": "adp",
                        "RIGHT_ATTRS": {"POS": "ADP"},
                    },
                    {
                        "LEFT_ID": "adp",
                        "REL_OP": ".",
                        "RIGHT_ID": "nmod",
                        "RIGHT_ATTRS": {"POS": {"IN": ["NOUN", "PROPN", "ADJ"]}},
                        "DEP": {"IN": ["nmod", "amod"]},
                    },
                ]
            ]
        }

        for name, pattern in patterns.items():
            self.matcher.add(name, pattern)

    def __call__(self, doc):
        t = dt.now()
        matches = self.matcher(doc)
        t2 = dt.now()
        print(f"DONE MATCH -> {(t2-t).total_seconds()}")
        return doc

@Language.factory("matching")
def create_matching(nlp, name: str):
    """create component for matching"""
    return Matching(nlp)

nlp = spacy.load("fr_core_news_lg")
nlp.add_pipe("matching", before="ner")

text = "Retrouvez dans cet article différents témoignages de blogueurs voyage qui nous parlent des plus belles randonnées qu’ils ont pu faire en France. "

for i, j in enumerate(nlp.pipe((text * 1000 for i in range(100)), batch_size=1)):
    del j

With this code, the dep matcher takes usualy 0.23s but once in a while (for me at doc 25, 62, 98) it takes ~1.17s .
The bug does not appears if we don't load the big JSON in memory. This is really weird and I don't know how to debug this. It is problematic if you have many patterns (the more you have, the more time it freezes once in a while).
As you probably noticed, the big json data are not even used within the pipeline. Just by loading it in memory causes the freeze.

Thank you for reading !

Environment

• Operating System: Linux-6.1.12-1-MANJARO-x86_64-with-glibc2.37
• Python Version Used: 3.10.9
• spaCy Version Used: 3.5.0 & 3.4.2
• Pipeline: fr_core_news_lg (3.5.0 & 3.4.2)
  &
• Operating System: macOS-13.2.1-arm64-arm-64bit
• Python Version Used: 3.10.6
• spaCy Version Used: 3.5.0
• Pipelines: fr_core_news_lg (3.5.0)

[adrianeboyd]

From the description it sounds like your computer is swapping because RAM is full. Keep an eye on the memory usage (I think you can look at "memory" in the Activity Monitor) and see if the delays start when memory usage gets close to 100%?

[probavee]

Hello !
It's definitely not full RAM, we are able to reproduce this on a 32Go MacBook and on my 16Go linux the total ram used is only 1.8Go and swap doesn't move.
This shows the ram pressure with the sample code above. The dep_matcher usually takes 0.1s but regularly peaks at 1s.

Same with our full pipeline which takes more RAM but not enough to pressure the machine. Still the problem appears and the dep_matcher regularly switch from 0.8 to 5s, again, no swap used.

Finally it happens in kube containers which I believe do not have swap memory, it would just kill the container which does not happen.
Can you think of any tools we can use better identify the problem and help you ?

[adrianeboyd]

My next guess would be that the periodic automatic garbage collection is slower when there are a lot of large objects in memory and you're occasionally catching this in your timing?

I do see the behavior you're describing when running your example, but if I add a manual gc.collect() in the nlp.pipe loop it looks like it goes away.

Possibly you can improve the performance at the points you're concerned about by adjusting the garbage collection settings and garbage collecting manually outside of the critical points?

I think that you'll probably get better overall performance by relying on python's defaults, but if you need to improve the average worst case in a crucial section of your code, this is something you can look into?

[probavee]

You are correct, it was an issue with the garbage collector. I managed to tweak it to allow more memory at each generation, it does not leak at all (for now at least) and the pipeline is always smooth. It might even have sped things up overall. I think my dictionaries prevented to have enough memory for spacy components, so gc cycles were to short, unnecessary and it slowed down everything. I'm not sure though, but it works. Thank you very much !