unable to run external commands in background or from crontab

[ng5]

i have setup broadcast mirror server, there is no problem with that. there is no problem with sender too.

1. sender.sh - OK
   echo X | websocat -k -E -t - wss://127.0.0.1:9000

2. receiver.sh
   websocat -E -k -t wss://127.0.0.1:9000 | xargs -r -n1 print_args.sh

receiver.sh works only from interactive shell.
i) ./receiver.sh - OK
ii) nohup ./receiver.sh & - NOT working
iii) adding receiver.sh to crontab doesn't work either.

i tried to redirect to named pipe in receiver.sh but that didn't work either. similar tools like netcat works without any problem in sender/receiver.sh.

[vi]

What do you mean by "not working"?

Variants:

1. websocat does not start
2. websocat starts, but exits early
3. websocat starts and connects to the socket, but xargs finishes early
4. both websocat and xargs persist, but xarg does not start any new processes
5. xargs starts child processes, but it does not work properly

?

Typically I use strace tool to debug what is happening, including for backgrounded processes.

[ng5]

From Crontab

1. both websocat and xargs immediately exits

From nohup

1. both websocat and xargs shows up (under ps -eaf command), but they don't work as expected. I would like to invoke
   print_args.sh on each message received.

I suspect it is not able to pipe correctly in the background.

Instead if use a similar tool like netcat (for send/receive)
works consistently across interactive shell, nohup or crontab.

1. sender_nc.sh:
   nc 127.0.0.1 9000
   x
   y
   z

2. receiver_nc.sh
   nc -l 9000 | xargs -r -n1 print_args.sh

[vi]

Note that you may want try foreachmsg: overlay if you want something to happen on each message. Something like

websocat -E -k -t wss://127.0.0.1:9000 foreachmsg:cmd:./print_stdin.sh

[ng5]

foreachmsg:cmd immediately exits as reported earlier by other people.
All 3 scenarios are shown in the screenshot attached.

1. without xargs or foreach
2. with xargs - only works in interactive shell
3. foreachmsg - exits immediately.

For now I ended up writing another websocket client to execute the command after receiving the message (in background). this is not ideal but not a deal breaker either. Atleast I was able to run web socket server and sender without writing code. It is only the receiver part which doesn't seem to work in background mode.

[vi]

Note that instead of broadcast:mirror: it may be better to use wsbroad.

[vi]

Tried some command lines: foreachmsg: exists as it tries both directions by default (data transfer to WebSocket and from WebSocket) and -E commands it to exit if any of that direction finishes. But reverse direction (to WebSocket) finishes immediately in some cases, as stdin is not available in those modes.

Maybe you want -un instead of -E, i.e. use only one direction (from left to right) and prevent sending quit message.

Also you need to modify print_args.sh for foreachmsg: - the data will be on stdin, not in argv.

[ng5]

Note that instead of broadcast:mirror: it may be better to use wsbroad.

I'm running this in production from last 3 weeks weeks. On an average it crashes once every 24 hours without providing anything in stdout or stderr. I have written a crontab script that runs every minute and if websocat server is down it simply restarts it. Therefore it is not causing a serious problem. However do you think wsbroad is more stable than running websocat with broadcast:mirror. Is there a known problem?

[vi]

Websocat is not expected to require periodic restarts. Maybe something wrong is happening? What do you mean by "crashes"? Process exits? (with what exit code?). Is it a high-load scenario or Weboscat is mostly waiting?

---

However do you think wsbroad is more stable than running websocat with broadcast:mirror

It is much simpler than Websocat and it uses modern dependencies (tokio-tungstenite).

Note that it differs from broadcast:mirror: - each seen URL gets a separate broadcast domain instead of being ignored like in Websocat. You can also tune some options of how messages should be forwarded.

[ng5]

load is very minimal hardly 100 concurrent subscribers. yes process exits without writing anything to stdout or stderr. i'm redirecting both 1 and 2 to a file. unfortunately core dump is not enabled on production server which could have revealed the actual cause of the problem. it is very hard to reproduce the error. i will replace websocat with wsbroad and see how it goes.

if you want to close the issue please feel free to close it as i'm not able to provide any stack trace because core is not enabled on production server. i use the websocat behind haproxy tunnel not sure if this matters. reason haproxy is used to provide consistent ssl connection on 443 port.

Once again thank you for all the help and great work.

[PanAeon]

Hi, I experienced similar issue and it was very frustrating. I guess websocat uses some heuristic to determine if stdin is open, which works differently in foreground and background.

my logs:

[INFO websocat::ws_client_peer] Connected to ws
[DEBUG websocat::my_copy] copy skipped
[INFO websocat::sessionserve] Reverse finished
[DEBUG websocat::sessionserve] Reverse shutdown finished
[DEBUG websocat::my_copy] zero len
[DEBUG websocat::my_copy] read_done
[DEBUG websocat::my_copy] done
[INFO websocat::sessionserve] Forward finished
[DEBUG websocat::sessionserve] Forward shutdown finished
[DEBUG websocat::ws_peer] drop WsWriteWrapper
[INFO websocat::sessionserve] Both directions finished
[DEBUG websocat::stdio_peer] restore_blocking_status
[DEBUG websocat::stdio_peer] Restoring blocking status for stdin
[DEBUG websocat::stdio_peer] Restoring blocking status for stdout
[DEBUG websocat::stdio_peer] restore_blocking_status
[DEBUG websocat::stdio_peer] Restoring blocking status for stdin
[DEBUG websocat::stdio_peer] Restoring blocking status for stdout

-Un flags have fixed the issue for me.

[vi]

works differently in foreground and background

I don't remember any explicit detection of background vs foreground mode in Websocat1.

Restoring blocking status

Are you using --async-stdio or asyncstdio: or old version of Websocat?

my logs:

What is the command line?

[PanAeon]

Are you using --async-stdio or asyncstdio: or old version of Websocat?

websocat 1.13.0

What is the command line?

it's either websocat or websocat -E

It may not be explicit but my best guess it's implicitly closes the stream when one of its end closed. Which means that when stdin is closed the connection is closed by default.
If this is intended behaviour then simple warning message about closing stdin might suffice