Too many open files: '/root/.cache/huggingface/token'

[kopyl]

Describe the bug

I ran this code:

from datasets import load_dataset
dataset = load_dataset("common-canvas/commoncatalog-cc-by", cache_dir="/datadrive/datasets/cc", num_proc=1000)

And got this error.
Before it was some other file though (lie something...incomplete)

runnting

ulimit -n 8192

did not help at all.

Steps to reproduce the bug

Run the code i sent

Expected behavior

Should be no errors

Environment info

linux, jupyter lab.