R1 memory stream configuration gets lost when nats-server is restarted [v2.10.20]

[facefirstMH]

Observed behavior

R1 memory stream configuration gets lost when nats-server is restarted.

Expected behavior

My expectation would have been that the configuration of a memory stream is durable.
Meaning, that the data was in memory and ok to be lost, but instead the stream's config is gone too.
This now requires that loss of a memory stream is detected and needs recreating.

Server and client version

2.10.20

Host environment

No response

Steps to reproduce

 ~
$ nats stream add --storage=memory --subjects=foo --defaults test
Stream test was created

Information for Stream test created 2024-12-03 12:43:07

              Subjects: foo
              Replicas: 1
               Storage: Memory

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
            Direct Get: true
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

State:

              Messages: 0
                 Bytes: 0 B
        First Sequence: 0
         Last Sequence: 0
      Active Consumers: 0

 ~
$ nats stream info test
Information for Stream test created 2024-12-03 12:43:07

              Subjects: foo
              Replicas: 1
               Storage: Memory

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
            Direct Get: true
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

State:

              Messages: 0
                 Bytes: 0 B
        First Sequence: 0
         Last Sequence: 0
      Active Consumers: 0


 ~
$ echo restarted nats server
restarted nats server

 ~
$ nats stream info test
[local-5222] ? Select a Stream  [Use arrows to move, type to filter]
> analytics
  insights
  logs
  match_events
nats.exe: error: could not pick a Stream to operate on: interrupt

The last stream info shows that the stream is not present anymore.

[ripienaar]

the stream is stored in memory, restarting the server clears the memory so this is expected behaviour

[MauriceVanVeen]

Would say this is at least inconsistent, so it might be classified as a bug still.

If you run only one server and it's not clustered. Creating an in-memory stream and restarting the server also removes the stream. As indeed the stream is only in-memory.

However, if you have a clustered setup. The in-memory stream is recreated on startup because the stream assignment was stored in the meta state. An in-memory stream staying for a clustered setup is more a side-effect of clustering. High-level this seems inconsistent though. Shouldn't we fix that?

[ripienaar]

When clustered you would still lose all the data in the stream, to me it seems fine that clustered would retain the knowledge of the stream as we then get a reliable storage of the configuration which you simply dont have in 1 server mode. It's a sensible enhancement clustering brings.

We have this in a fair few places of things that get persisted in raft that otherwise is never stored

[facefirstMH]

Meaning the R1 memory stream is the only stream that needs extra management code to detect its disappearance and re-create it as well as it's consumers.

The downstream issue of a disappearing stream is that (at least the c pull consumer api) did not raise an error that would indicate loss of the backing stream. (I believe the consumer message won't be received without the stream being there.) An explicit not exist type error when consuming would have allowed the process creating the stream and consuming from it, to notice the issue and take action.

I tried monitoring for the streams existence in another program and recreating it from there. Didn't go so well either. Probably because the consumer state in the app was from the previous incarnation and/or does not exist in the new stream.

We now restart the consumer app whenever the nats server goes down.