perf: don't handle SitePage nodes when deleting stale nodes #39161
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gatsbot
bot
added
the
status: triage needed
pieh
added
topic: core
pieh
force-pushed
the
perf/dont-cleanup-sitepage-node-after-source-nodes
branch
2 times, most recently
from
63be5c4 to
370b4b2
Compare
pieh
force-pushed
the
perf/dont-cleanup-sitepage-node-after-source-nodes
branch
from
370b4b2 to
f611677
Compare
pieh
force-pushed
the
perf/dont-cleanup-sitepage-node-after-source-nodes
branch
from
d0b1452 to
c4adf6a
Compare
pieh
commented
Dec 11, 2024
| @@ -445,3 +452,99 @@ export const clearGatsbyImageSourceUrls = | |||
| type: `CLEAR_GATSBY_IMAGE_SOURCE_URL`, | |||
| } | |||
| } | |||
|
|
|||
| // We add a counter to node.internal for fast comparisons/intersections | |||
There was a problem hiding this comment.
the additions here were just moved from public.js file here to make it easier to reuse
…itepage-node-after-source-nodes
…itepage-node-after-source-nodes
mrstork
approved these changes
Dec 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
SitePagenodes because at this point in time site creation didn't run yet. This results in mass nodes deletion and subsequent mass node creation - other than being wasteful this tends to lead to increase in LMDB datastore file size increasing as managing disk size is not priority for it so it tends to allocate more and more space over time with data mutations even if "logical" data remains at similar volume. This PR mainly looks to avoid wasteful LMDB operations and thus trying to slow down LMDB filesize increases over timeonCreatePagehook is used (either directly or by a plugin) it often result in temporarySitePagenodes - initial page object created, then plugins in their hook implementation might change a page path or page context. This result in unnecessary lmdb operations. This PR adds intermediate stage when page is create and there is at least oneonCreatePagehook that doesn't commit db operations immediately and instead waits foronCreatePagehooks to finish and only commit final operation (this final operation might be no-op in case final page is the same as page from previous build which eliminates db mutations in those cases)Tests
Existing tests (especially e2e and integration) still pass
https://linear.app/netlify/issue/FRB-1388/gatsby-mdb-file-size-balloons-from-120mb-to-400mb-when-not-adding