Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support pickling sources. #1071

Merged
merged 1 commit into from
Feb 28, 2023
Merged

Support pickling sources. #1071

merged 1 commit into from
Feb 28, 2023

Conversation

manthey
Copy link
Member

@manthey manthey commented Feb 28, 2023

This also supports pickling tiles from the tile iterator.

Closes #1069.

This also supports pickling tiles from the tile iterator.

Closes #1069.
@banesullivan
Copy link
Contributor

I'm curious what's driving the need for this. Multiprocessing?

@manthey
Copy link
Member Author

manthey commented Feb 28, 2023

I'm curious what's driving the need for this. Multiprocessing?

Yes -- @annehaley tried to do some multiprocessing and the inability to pickle sources made the book keeping more involved. This would also make the multiprocessing cleaner in the HistomicsTK examples.

@manthey
Copy link
Member Author

manthey commented Feb 28, 2023

For example, this:

   for tile in ts.tileIterator(**it_kwargs):
        tile_position = tile['tile_position']['position']
        cur_nuclei_list = dask.delayed(detect_tile_nuclei)(
            args.inputImageFile,
            inputImageArgs,
            tile_position,
            it_kwargs,
            args, src_mu_lab, src_sigma_lab
        )
        tile_nuclei_list.append(cur_nuclei_list)

becomes:

   for tile in ts.tileIterator(**it_kwargs):
        cur_nuclei_list = dask.delayed(detect_tile_nuclei)(
            tile,
            args, src_mu_lab, src_sigma_lab
        )
        tile_nuclei_list.append(cur_nuclei_list)

where, since tile is pickleable (since the source is pickleable), it can be passed by itself rather than as a file name, tile position, and arguments for the tile source and tile iterator.

And the delayed function, instead of doing

ts = large_image.getTileSource(slide_path, **slide_args)
tile = ts.getSingleTile(
        tile_position=tile_position,
        format=large_image.tilesource.TILE_FORMAT_NUMPY,
        **it_kwargs)

just uses a reference to tile

@banesullivan
Copy link
Contributor

Thanks for clarifying -- good to capture these examples too

@manthey manthey merged commit 34118ce into master Feb 28, 2023
@manthey manthey deleted the pickle-sources branch February 28, 2023 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make tile sources pickleable
2 participants