[BUG] Issue with CACHE_DIR Default Value in Accelerate Pipeline

[ryan-minato]

Describe the bug

The default value for CACHE_DIR in the pipeline creates several issues. On many systems, such as Ubuntu distributions, users typically lack permissions for the root directory (/), which prevents the pipeline from running. On Windows, it creates a scratch folder on the C drive, which is not ideal.

lighteval/src/lighteval/main_accelerate.py

Line 34 in 0977fde

CACHE_DIR: str = os.getenv("HF_HOME", "/scratch")

Proposed Solution

Remove the default value for CACHE_DIR and instead return None. This will allow the pipeline to follow the default logic of the transformers library.

CACHE_DIR: str = os.getenv("HF_HOME")

To Reproduce

Run the example provided in the README on any Linux distribution:

HF_HOME= lighteval accelerate \
    "pretrained=gpt2" \
    "leaderboard|truthfulqa:mc|0|0"

Expected behavior

The running should execute without errors.

Version info

The issue was encountered on the main branch installed with pip install -e .[dev].