json patch syntax for overlays #313
Replies: 1 comment 6 replies
-
|
I like the idea of using a notation that is likely familiar to the person using the tool. Why Structures? Granted that this almost always a shift in thinking. We're painfully aware of the pain that many folks go through to make that shift. However, we believe that given the time to do so, we can mitigate those pains sufficiently to make the host of benefits on the other side well worth that trade-off. I don't want to distract with elaborating the total scope of those benefits, here, but just consider what can be accomplished when the tool has information about the exact shape (and potentially meaning) of the structure. One relevant benefit to the user can be increasing their confidence while visually reasoning that the path they have described will match the documents they intend. For example, one trick is when writing an overlay, I can copy the relevant snippet from an actual document to seed my overlay and then resolve the ambiguities with annotations. Modulo the sharp edges around overlays (of which many are getting filed down, as we type), this is a great way to "get it right" out the gate. An indirect relevant benefit to the user is that there's one mechanism/symbol-system for talking about documents: YAML. There's one notation to learn (albeit, an expanding notation when doing things like patching): almost all of what you know about the notation works as you take on more capabilities. This also keeps the implementation and interface complexity of the tool itself minimized. Tools that do this tend to mature more quickly and have fewer defects: they are more reliable. ... So, if there seems to be resistance to supporting JSON Path notation, well ... it can be in part because doing so — despite the obvious value of reuse across (some) tools in the ecosystem — it would mitigate these desired qualities of the tool. Now, it's not to say that once the Carvel tools reach a certain level of maturity that they couldn't be in a place to take on convenience features like supporting JSON Path. Looking into my 🔮 , I see that once this tool starts to tackle problems that practically speaking require paths, it would be important to seriously consider this notation. All of this said to expose the structure of some thinking around the topic. Others (even from within the project) have different perspectives. And impact does not necessarily follow intention. |
Beta Was this translation helpful? Give feedback.
-
|
(Apologies for the long delay in responding; I missed your initial reply and just saw this on Friday afternoon.) I am really sorry for making you feel disrespected, Luke. It's not how we do it, here. We want everyone in this community to be able to share their perspective and experience on an equal footing. I absolutely can see how "Granted that this almost always a shift in thinking" can mean, "when folks finally come around to our way of thinking" effectively dismissing all differing views, out of hand. I also see that linking to Eric Raymond's book's definition of interface and implementation complexity can draw the inference that someone needs to be taught… and about some fundamentals — that's pretty condescending. That sucks and I'm sorry. To lay it all bare, I meant no such disrespect. By "Granted that this almost always a shift in thinking" I meant to acknowledge that So, I missed my own mark of "unambiguously inclusive" and I'm sorry you're the recipient of it. Thank you for your candor, Luke. Sharing sentiments in text — a medium devoid of almost all non-verbal elements — is risky. Given that my role in this exchange is to respectfully contextualize the idea you're bringing, I've got some work to do to more skillfully mitigate that risk. |
Beta Was this translation helpful? Give feedback.
-
|
Into the technical topic at hand: introducing RFC 6902-like syntax to The heart of the message I'm reading, here is:
JSON patch: more compact pathsJSON Path is more concise than the corresponding multi-line YAML equivalent. Agreed: I too find it often is. Though, turning this example over in my hand, I see that the But I think what you're pointing at is more about the path itself. That in the JSON path you only need to scan within one (1) line to get a feel for where the change is happening. JSON patch: more explicit intent
Ironically, the implicitness in this Again, though, if I'm understanding what you're saying, it's not about whether the information is present, but that the eye doesn't have to scan over the whole YAML structure to find it. Slight tangent: I notice that the two examples here are subtly different: Overlaying ArraysYeah, I have found overlay'ing arrays to be my own biggest sources of pain while using One insight that helped things make more sense for me is: the array item on the "right" is evaluated many times: once for each item in the array on the "left" (if needed, precise definitions of "left" and "right" are found in the docs: Overlay Module > Overview). Here, In fact, the example illustrates this well: #@overlay/match by=overlay.subset({"kind":"Package"})
---
spec:
template:
spec:
fetch:
#@overlay/match by=overlay.subset({"imgpkgBundle": {}})
- imgpkgBundle:
#@overlay/match missing_ok=True
image: #@ data.values.dev_imageThis shifts the connaissance from position to name (for those who haven't stumbled over the late (and great) Jim Weirick's treatment, I've found the concept useful in wrangling various forms of coupling: https://connascence.io). Practically speaking, this means that there can be multiple JSON paths in
|
Beta Was this translation helpful? Give feedback.
-
Yep, that definitely captured my thoughts.
Yes, I think that's it - thank you for distilling that.
This might be true, but I'm not sure - https://github.com/json-path/JsonPath#filter-operators There seem to be limitations, so I would expect that there are things that you could do in I am conflicted about overlays or patches that get this 'interesting' though.
Wow, thanks for building a proof of concept in gist form, and sharing the implementation challenges. I don't know - I don't have the maintainer's-eye view of the situation. I'm grateful for the concern about cryptic error messages bubbling up from the Starlark layer, and I don't have much of a suggestion there other than to strongly agree that error messages are really important because debugging overlays is hard. I think If I had to guess what made it frustrating for me to debug my own overlays, it was the breadth of the matching operation vocabulary and not knowing which of the different verbs to look for. Maybe that suggests that rather than having multiple idioms for identifying a particular yaml node and editing it (the existing |
Beta Was this translation helpful? Give feedback.
-
|
@LukeWinikates said:
Oh yeah, this sentiment really resonates with me too. Overlays are multiple flavors of indirection happening at once: they are declarative ("did my overlay match on exactly the set of documents I intended?"); and they are expressions ("aaand, what does the resulting YAML look like?!"). Whenever someone can capture customization in templating, many folks on the Carvel team will recommend doing so citing the inherent complexity of overlays. Yeah, the expressive power is there as a means of last resort; for example, you don't own the YAML and you can't (practically) inject your own additional post-processing. @LukeWinikates reflected:
Yes! We've had an initiative to specifically identify ways to improve our Overlay error messaging. This is an effort that — through prioritization — has had to yield to addressing straight-up missing features. That said, your observation and suggestion here can fold into that synthesis: What can be done to:
Earlier, @LukeWinikates pointed out:
Yeah, that seems kinda perfect... just enough of language to articulate a predicate. 👍🏼 FWIW, my research is showing that JSON Path is a different (evolved?!?) thing from JSON Pointer (i.e. RFC 6901). It's the latter that JSON Patch (RFC 6902) is based on — the same flavor of patching that Kustomize is currently using. RFC 6902 § 4 makes that explicit reference:
|
Beta Was this translation helpful? Give feedback.
-
Yes... tangentially, this is coming up a lot with references to K8s namespaces. They seem like a place where overlays are natural, because they are an obvious place where the person apply the yaml will often want to decide where all the resources are deployed. But as we Carvelize yaml fragments from other sources, we often find references to the namespace in spots like DNS names in arbitrary spots, or most recently in a I think to some degree these references to the namespace can be avoided altogether so they don't need to be modified by an overlay. But this the example that I keep coming back to when it comes to questioning the sweet spot for overlays as well as what the ideal templating solution for K8s resource definitions should look like. Appreciated your response about the overlay error messages. I'll have to keep thinking if I have any concrete suggestions here.
Yup, exactly - and thanks for calling out that that's a different RFC and not what Kustomize actually ships 😅 |
Beta Was this translation helpful? Give feedback.
-
I work on some projects that use
ytt, and some that usekustomize.For patching array items, I do like the use of
json pathandjson patchthatkustomizesupports.https://github.com/kubernetes-sigs/kustomize/blob/master/examples/jsonpatch.md
ytt's matcher functions give a nice option for when you have more complex needs, but I wonder if an@overlay/match by=overlay.jsonpath(/foo/bar/2/baz)option would be consise and readable in some cases. It seems to me thatyttoverlays require several nested layers of#@overlay/matchfor arrays within arrays.Beta Was this translation helpful? Give feedback.
All reactions