Support custom fingerprinting with Dataset.from_generator

[bilelomrani1]

Feature request

When using Dataset.from_generator, the generator is hashed when building the fingerprint. Similar to .map, it would be interesting to let the user bypass this hashing by accepting a fingerprint argument to .from_generator.

Motivation

Using the .from_generator constructor with a non-picklable generator fails. By accepting a fingerprint argument to .from_generator, the user would have the opportunity to manually fingerprint the dataset and thus bypass the crash.

Your contribution

If validated, I can try to submit a PR for this.

[mariosasko]

The fingerprint parameter serves a slightly different purpose - we use it to inject a new fingerprint after transforming a Dataset (computed from the previous fingerprint + transform + transform args), e.g., to be able to compute the cache file for a transform. There is no concept of fingerprint before a Dataset is fully initialized, but we still need to hash the args (e.g., generator func) of the "dataset creation methods" (from_generator, from_csv, etc.) to compute the cache directory (to store the initial version and transformed dataset versions)

I agree it should be easier to bypass the hashing mechanism in this instance, too. However, we should probably first address #5080 before solving this (e.g., maybe exposing hash in load_dataset/load_dataset_builder.

[mlin]

Adding +1 here:

If the generator needs to access some external resources or state, then it's not always straightforward to make it pickle-able. So I'd like to be able to override how the default cache key derivation needs to pickle the generator (and of course, I'd accept responsibility for that part of cache consistency).

Appears to be a recurrent roadbump: #6118 #5963 #5819 #5750 #4983

[mlin]

Silly hack incoming:

import uuid

class _DatasetGeneratorPickleHack:
    def __init__(self, generator, generator_id=None):
        self.generator = generator
        self.generator_id = (
            generator_id if generator_id is not None else str(uuid.uuid4())
        )

    def __call__(self, *args, **kwargs):
        return self.generator(*kwargs, **kwargs)

    def __reduce__(self):
        return (_DatasetGeneratorPickleHack_raise, (self.generator_id,))


def _DatasetGeneratorPickleHack_raise(*args, **kwargs):
    raise AssertionError("cannot actually unpickle _DatasetGeneratorPickleHack!")

Now Dataset.from_generator(_DatasetGeneratorPickleHack(gen)) works even if gen is unpicklable, because Dataset just pickles the shim object that avoids actually traversing gen. Then, one can work out how to set generator_id meaningfully to allow cache reuse.

[hartmans]

I'd like some way to do this too. I find that sometimes the hash doesn't cover enough, and that the dataset is not regenerated even when underlying data has changed, and by supplying a custom fingerprint I could do a better job of controlling when my dataset is regenerated.

[stevemadere]

This is what I did and it works:

https://github.com/stevemadere/s3-datasets/blob/e475a566a16d3051656a66f8ff4d3baa4c55a66c/src/tokengenerators/text_ds_2_tokens_generator.py#L200

[urjitbhatia]

I ran into the same thing - my actual generator reads from a disk source that might have new data (images) available at some point and it ends up ignoring calling the generator. Thanks for the hack @mlin 👋

[vttrifonov]

just wanted to pitch my support for an easy control over the generator id. requiring that generators are pickleable just to get a unique id is limiting: plenty of classes (maybe even hf.datasets own) are written with no pickle support in mind. also as mentioned above the state of a generator might extend beyond its pickle.