Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dekaf runtime changes #1854

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Conversation

jshearer
Copy link
Contributor

@jshearer jshearer commented Jan 3, 2025

Description:

Rolls up a couple more changes to the agent and runtime for Dekaf:

  • Detect tasks intended for Dekaf inside Runtime::unary_materialize() and route them to dekaf::connector::unary_materialize(). Doing it here instead of in build allows more of Flow to interact with Dekaf tasks correctly:
    • The field selection UI uses a dry_run publication (i.e a validate call) to fetch which fields to show. This now intercepts that call and correctly passes it to Dekaf because we set connector_type based on the endpoint in walk_materialization():
      models::MaterializationEndpoint::Dekaf(config) => (
      flow::materialization_spec::ConnectorType::Dekaf as i32,
      serde_json::to_string(config).unwrap(),
    • The connector_tags agent task calls a connector's Spec RPC, and we need to get Dekaf's endpoint/resource config schemas somehow so I figured rather than writing a thing to do it manually, all we need to do is send connector_type: ConnectorType::Dekaf when we detect a task with image_name starting in ghcr.io/estuary/dekaf-.
  • I also had to skip image pulling in connector_tags when processing a Dekaf row as obviously there's no image to pull.

This change is Reviewable

We need `runtime` to depend on `dekaf`, and they both need `unseal`, so it needed to be moved somewhere, and there didn't seem to be anywhere good to put it other than its own crate.
In order for field selection to work in the UI, it needs to be able to validate a draft with a Dekaf connector in it. This detects when the request is for a Dekaf materialization and re-routes it to Dekaf's `unary_materialize`. It also means that the `connector_tags` job can work for Dekaf with a small change to detect image names starting in `ghcr.io/estuary/dekaf-*` and mark them as `ConnectorType::Dekaf`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant