Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add clap blocks for configuring DataFusion and IO runtimes #25664

Closed
hiltontj opened this issue Dec 16, 2024 · 0 comments · Fixed by #25665
Closed

Add clap blocks for configuring DataFusion and IO runtimes #25664

hiltontj opened this issue Dec 16, 2024 · 0 comments · Fixed by #25665
Assignees
Labels

Comments

@hiltontj
Copy link
Contributor

Problem statement

Currently, the main Tokio runtime and Data fusion runtime are configured via code imported from influxdb3_core, i.e., IOx, and which are defined here.

The former is not actually being used, i.e., the main Tokio runtime in influxdb3 is currently using default settings, and cannot be configured.

As part of #25646, we should add a set of clap blocks to influxdb3 to enable configuration of these runtimes. This will:

  • Allow us to remove several references to IOX in the CLI docs defined in there (see Remove references to IOx in codebase  #25662).
  • Enable IO on the DataFusion runtime
  • Allow configuration of the main Tokio IO runtime from the CLI

Proposed solution

  • Create a influxdb3_clap_blocks crate
  • Port over the tokio configurations from influxdb3_core, replacing references to IOX with INFLUXDB3
  • Enable IO on the DataFusion runtime, i.e., do not use this macro argument.

Alternatives considered

Only use a single runtime for the entire system. This would have us define a single Tokio runtime that is shared between IO and DataFusion. This does not seem like a good idea, since CPU intensive work during query planning could stall other parts of the system handling IO, e.g., serving API requests.

For now, we can roll with the two runtimes, and experiment with different configurations to see what works best.

Additional context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant