Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TGI hangs when running two extremely long prompts at once #2842

Open
2 of 4 tasks
JohnTheNerd opened this issue Dec 14, 2024 · 0 comments
Open
2 of 4 tasks

TGI hangs when running two extremely long prompts at once #2842

JohnTheNerd opened this issue Dec 14, 2024 · 0 comments

Comments

@JohnTheNerd
Copy link

System Info

exact command used to run TGI: docker run --gpus all --shm-size 1g -p 5000:80 -v /mnt/disk/models/llama-3.3-70b-instruct-awq:/usr/src/llama-3.3-70b -it ghcr.io/huggingface/text-generation-inference:3.0.1 --model-id llama-3.3-70b --quantize awq --cuda-memory-fraction 1 --sharded true --num-shard 2

model used: https://huggingface.co/casperhansen/llama-3.3-70b-instruct-awq

hardware (all local):

  • 2x RTX 3090 with NVLink, power limited to 260W
  • Threadripper 2920x
  • 96GB DDR4 RAM

Docker is running inside a Ubuntu 22.04 LXC container in Proxmox VE. I am using version 3.0.1.

full nvidia-smi output:

Sat Dec 14 00:02:13 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.05 Driver Version: 550.127.05 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 On | 00000000:0A:00.0 Off | N/A |
| 0% 34C P8 29W / 260W | 1MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 On | 00000000:42:00.0 Off | N/A |
| 0% 37C P8 34W / 260W | 1MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

Information

  • Docker
  • The CLI directly

Tasks

  • An officially supported command
  • My own modifications

Reproduction

please note that this script is explicitly written to saturate the VRAM of my dual 3090 setup, if you are reproducing on a system with more or less VRAM, you should modify the script to send a longer or shorter prompt. my system can handle 13792 tokens, hence we send 13714 per execution in the script. you can find the number of tokens your system can handle by simply running TGI - it will be logged.

to reproduce, simply run the command given in the system info with the given model at the expected location, and then on another terminal, run the below Bash script (please note that it often requires running it twice - just run it once, once you see a response CTRL+C out and run killall curl, then run again):

#!/bin/bash

api_endpoint="http://localhost:5000/v1/completions"
api_key="test"
model="llama-3.3-70b"

# modify the prompt to make it about 100 tokens under the max batch total tokens reported by TGI
# the below prompt works fine for 2xRTX3090 with TGI given the entire VRAM
prompt="Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat."
longprompt="$prompt $prompt $prompt $prompt $prompt $prompt $prompt $prompt $prompt $prompt"
longprompt="$longprompt $longprompt $longprompt $longprompt $longprompt $longprompt $longprompt $longprompt $longprompt"

headers=(-H "Authorization: Bearer $api_key" -H "Content-Type: application/json")
data=(-d "{\"model\": \"$model\", \"prompt\": \"$longprompt\", \"stream\": true}")

curl -X POST "${headers[@]}" "${data[@]}" "$api_endpoint" &

# add a letter to the beginning to break prefix caching
longprompt="a $longprompt"
data=(-d "{\"model\": \"$model\", \"prompt\": \"$longprompt\", \"stream\": true}")

sleep 1

# the server should be dead as soon as this second curl runs
curl -X POST "${headers[@]}" "${data[@]}" "$api_endpoint"

Expected behavior

the second request either being delayed or refused. when the reproduction script runs, TGI immediately hangs and never recovers.

nvidia-smi output after reproduction:

Sat Dec 14 00:15:45 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.05 Driver Version: 550.127.05 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 On | 00000000:0A:00.0 Off | N/A |
| 0% 35C P8 29W / 260W | 22227MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 On | 00000000:42:00.0 Off | N/A |
| 0% 39C P8 34W / 260W | 22227MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+

the low power usage is interesting since I find that it is always maxed out during inference. another interesting detail I have noticed is that I get "Prefix 1 - Suffix 13714" output at any request I run afterwards, regardless of the length of the third request I just sent out.

TGI logs:

tgi@llm:/mnt/disk/text-generation-inference$ docker run --gpus all --shm-size 1g -p 5000:80 -v /mnt/disk/models/llama-3.3-70b-instruct-awq:/usr/src/llama-3.3-70b -it ghcr.io/huggingface/text-generation-inference:3.0.1 --model-id llama-3.3-70b --quantize awq --cuda-memory-fraction 1 --sharded true --num-shard 2
2024-12-14T03:35:16.371090Z INFO text_generation_launcher: Args {
model_id: "llama-3.3-70b",
revision: None,
validation_workers: 2,
sharded: Some(
true,
),
num_shard: Some(
2,
),
quantize: Some(
Awq,
),
speculate: None,
dtype: None,
kv_cache_dtype: None,
trust_remote_code: false,
max_concurrent_requests: 128,
max_best_of: 2,
max_stop_sequences: 4,
max_top_n_tokens: 5,
max_input_tokens: None,
max_input_length: None,
max_total_tokens: None,
waiting_served_ratio: 0.3,
max_batch_prefill_tokens: None,
max_batch_total_tokens: None,
max_waiting_tokens: 20,
max_batch_size: None,
cuda_graphs: None,
hostname: "1be39f3da74b",
port: 80,
shard_uds_path: "/tmp/text-generation-server",
master_addr: "localhost",
master_port: 29500,
huggingface_hub_cache: None,
weights_cache_override: None,
disable_custom_kernels: false,
cuda_memory_fraction: 1.0,
rope_scaling: None,
rope_factor: None,
json_output: false,
otlp_endpoint: None,
otlp_service_name: "text-generation-inference.router",
cors_allow_origin: [],
api_key: None,
watermark_gamma: None,
watermark_delta: None,
ngrok: false,
ngrok_authtoken: None,
ngrok_edge: None,
tokenizer_config_path: None,
disable_grammar_support: false,
env: false,
max_client_batch_size: 4,
lora_adapters: None,
usage_stats: On,
payload_limit: 2000000,
enable_prefill_logprobs: false,
}
2024-12-14T03:35:17.822644Z INFO text_generation_launcher: Using attention flashinfer - Prefix caching true
2024-12-14T03:35:17.822673Z INFO text_generation_launcher: Sharding model on 2 processes
2024-12-14T03:35:17.849179Z WARN text_generation_launcher: Unkown compute for card nvidia-geforce-rtx-3090
2024-12-14T03:35:17.875739Z INFO text_generation_launcher: Default max_batch_prefill_tokens to 4096
2024-12-14T03:35:17.875768Z INFO text_generation_launcher: Using default cuda graphs [1, 2, 4, 8, 16, 32]
2024-12-14T03:35:17.875970Z INFO download: text_generation_launcher: Starting check and download process for llama-3.3-70b
2024-12-14T03:35:27.890657Z INFO text_generation_launcher: Files are already present on the host. Skipping download.
2024-12-14T03:35:29.420810Z INFO download: text_generation_launcher: Successfully downloaded weights for llama-3.3-70b
2024-12-14T03:35:29.422781Z INFO shard-manager: text_generation_launcher: Starting shard rank=1
2024-12-14T03:35:29.422787Z INFO shard-manager: text_generation_launcher: Starting shard rank=0
2024-12-14T03:35:32.446250Z INFO text_generation_launcher: Using prefix caching = True
2024-12-14T03:35:32.446287Z INFO text_generation_launcher: Using Attention = flashinfer
2024-12-14T03:35:36.285801Z INFO text_generation_launcher: Converting awq model to Marlin packing format.
2024-12-14T03:35:36.300938Z INFO text_generation_launcher: Using GPTQ-Marlin kernels
2024-12-14T03:35:39.451778Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2024-12-14T03:35:39.460873Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2024-12-14T03:35:49.462783Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2024-12-14T03:35:49.474199Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2024-12-14T03:35:59.474516Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2024-12-14T03:35:59.485531Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2024-12-14T03:36:09.486393Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2024-12-14T03:36:09.497276Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2024-12-14T03:36:19.497630Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2024-12-14T03:36:19.508224Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2024-12-14T03:36:29.509532Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2024-12-14T03:36:29.520053Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2024-12-14T03:36:39.520249Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=1
2024-12-14T03:36:39.532437Z INFO shard-manager: text_generation_launcher: Waiting for shard to be ready... rank=0
2024-12-14T03:36:47.489100Z INFO text_generation_launcher: Using prefill chunking = True
2024-12-14T03:36:48.429140Z INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-0
2024-12-14T03:36:48.443317Z INFO shard-manager: text_generation_launcher: Shard ready in 78.993849822s rank=0
2024-12-14T03:36:48.799833Z INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-1
2024-12-14T03:36:48.830695Z INFO shard-manager: text_generation_launcher: Shard ready in 79.389127013s rank=1
2024-12-14T03:36:48.896933Z INFO text_generation_launcher: Starting Webserver
2024-12-14T03:36:49.064218Z INFO text_generation_router_v3: backends/v3/src/lib.rs:125: Warming up model
2024-12-14T03:36:49.088619Z INFO text_generation_launcher: Using optimized Triton indexing kernels.
2024-12-14T03:36:59.191040Z INFO text_generation_launcher: KV-cache blocks: 13807, size: 1
2024-12-14T03:36:59.250873Z INFO text_generation_launcher: Cuda Graphs are enabled for sizes [32, 16, 8, 4, 2, 1]
2024-12-14T03:37:01.047823Z INFO text_generation_router_v3: backends/v3/src/lib.rs:137: Setting max batch total tokens to 13793
2024-12-14T03:37:01.047846Z WARN text_generation_router_v3::backend: backends/v3/src/backend.rs:39: Model supports prefill chunking. waiting_served_ratio and max_waiting_tokens will be ignored.
2024-12-14T03:37:01.047862Z INFO text_generation_router_v3: backends/v3/src/lib.rs:166: Using backend V3
2024-12-14T03:37:01.047866Z INFO text_generation_router: backends/v3/src/main.rs:162: Maximum input tokens defaulted to 13792
2024-12-14T03:37:01.047870Z INFO text_generation_router: backends/v3/src/main.rs:168: Maximum total tokens defaulted to 13793
2024-12-14T03:37:04.801366Z INFO text_generation_router::server: router/src/server.rs:1873: Using config Some(Llama)
2024-12-14T03:37:04.802186Z WARN text_generation_router::server: router/src/server.rs:1913: no pipeline tag found for model llama-3.3-70b
2024-12-14T03:37:04.802204Z WARN text_generation_router::server: router/src/server.rs:2015: Invalid hostname, defaulting to 0.0.0.0
2024-12-14T03:37:04.875474Z INFO text_generation_router::server: router/src/server.rs:2402: Connected
2024-12-14T03:44:05.438412Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13714
2024-12-14T03:44:23.198661Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:23.901175Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:23.939484Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:23.981145Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.021598Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.061811Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.101752Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.142250Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.182300Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.222122Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.261730Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.301137Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.340566Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.380027Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.419499Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.458739Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.497873Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.537057Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.576367Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.615820Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.655742Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.695567Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.735184Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.774647Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.814580Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.853758Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.893781Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.933428Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:24.973064Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.012966Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.052809Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.092695Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.132528Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.172559Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.212373Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.252061Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.291598Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.331048Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.372819Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.411984Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.451681Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.491101Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.530461Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.569770Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.609149Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.649029Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.688961Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.728464Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.767930Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.807682Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.847183Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.887078Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.926769Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:25.966730Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.006587Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.046628Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.086462Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.126332Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.165896Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.205668Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.245162Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.285393Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.324863Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.364758Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.404100Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.443770Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.483124Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.522206Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.561107Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.600635Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.640073Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.679584Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.719145Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.758621Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.798067Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.837613Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.877143Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.916736Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.956656Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:26.996465Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.036228Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.075826Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.115254Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.154857Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.194731Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.233897Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.273290Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.312622Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.351949Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.391274Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.430432Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.470074Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.509951Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.549798Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.589315Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.628713Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.668425Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.708032Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.747760Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.787644Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.827714Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.867814Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.907231Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.946870Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:27.986416Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.026271Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.066063Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.105710Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.145248Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.184757Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.224380Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.264140Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.303887Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.343503Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.383180Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.422729Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.462167Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.501168Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.540162Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.579933Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.619041Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.658539Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.698107Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.737773Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.777424Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.817273Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.857182Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.896866Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.936655Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:28.975950Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.015532Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.055276Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.094999Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.134881Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.174141Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.214024Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.253884Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.293415Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.332819Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.372334Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.411844Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.451339Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.491086Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.530769Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.570356Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.610142Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.649937Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.689648Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.729515Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.769333Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.808960Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.848801Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.888071Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.927686Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:29.967463Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.007110Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.047037Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.086756Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.126577Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.166320Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.205845Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.245428Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.284985Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.324444Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.363721Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.403588Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.442933Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.482500Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.522227Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.561694Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.600902Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.640275Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.679941Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.719975Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.759737Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.799595Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.839230Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.879226Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.918845Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.958596Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:30.998068Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:31.037846Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:31.077838Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:31.117637Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:31.157287Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:31.196785Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:31.236406Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:31.276089Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:31.315740Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:31.355239Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:31.394833Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:31.434655Z ERROR text_generation_router::server: router/src/server.rs:882: Failed to send event. Receiver dropped.
2024-12-14T03:44:31.434656Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 13715
2024-12-14T03:44:31.480160Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 1 - Suffix 13714
2024-12-14T03:44:31.480189Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 1 - Suffix 13714
2024-12-14T03:44:50.034179Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 1 - Suffix 13714
2024-12-14T03:44:50.035093Z INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 1 - Suffix 13714

curl output:

:

:

:

:

:

:

:

:

:

:

:

:

:

:

:

:

strace output after reproducing:

root@llm:~# strace -p 95189
strace: Process 95189 attached
restart_syscall(<... resuming interrupted read ...>) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, 0x7ffde10c2e50) = 0
wait4(294, 0x7ffde10c2de0, WNOHANG, NULL) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=100000000}, ^Cstrace: Process 95189 detached
<detached ...>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@JohnTheNerd