Long cache retrieval when resolving dependencies

[Ernest0-Production]

Description

SPM dependency resolution takes a very long time to extract. Each package* takes on average ~1 second to extract.

As I understand it, the problem is that the cache is stored in a compressed form and takes time to extract. But it seems to me that this is of less practical use than a faster resolution.

I would like the user cache to store the original artifacts (the same as in the local .build directory)

[dschaefer2]

I assume you mean "package" not "packet". But yes, it's actually doing a git clone and checkout from the cache which is a very expensive operation.

[Ernest0-Production]

I assume you mean "package" not "packet". But yes, it's actually doing a git clone and checkout from the cache which is a very expensive operation.

Yes, I made a typo.
Is it technically possible to change this strategy, storing all repositories in the user cache?

[dschaefer2]

I think we're probably at the point where we're doing the best we can. SwiftPM does a complete git clone into the user cache under .swiftpm/cache/repositories. We need that so we have all the versions available during dependency resolution and allows you to have multiple projects using different versions. We then do a shallow clone of the selected revision over to the package scratch directory. One second to do is about as fast as git can go.

The solution we're hoping for is the previous work with package registries where the registry would have the metadata we need and a source archive for each available version. Then we can cache those which would be much faster. But I'm not sure when a community wide solution for that will happen.

[Ernest0-Production]

@dschaefer2 I spent some more time on the benchmark:
It seems that the real reason for the long resolution is not in the cache, but in the process of generating the workspace-state.json file.

If you delete everything in the .build folder except this file – restoring from the user cache will happen quite quickly.
But if you delete workspace-state.json (even without deleting the local cache) – the generation of this file takes a surprisingly long time.