Back-off strategy of handling messages which fail

[chrismoutray]

Hi,

Hopefully someone can point me in the right direction or tell me I'm being stupid! :). I'm looking for a way to configure Rebus message processing with a back-off strategy for specific messages which fail frequently.

I have a basic Rebus setup with 2 subscriptions for; DispatchMessage and OperationMessage.
Now imagine my queue is flooded with 100's of DispatchMessage's and then after small delay 1 OperationMessage. Btw the messages aren't related.

Sometimes there will be a period of time where the handler for the DispatchMessage keeps failing because its actually making a call to a 3rd party API with failure timeout (30 seconds).

The problem I've observed is that the OperationMessage is not processed until the all DispatchMessage's have been processed, I guess this is expected because of the order they were added to the queue. But I'm wondering if there is a way to make Rebus skip the DispatchMessage's (even its retries) for a period of time leaving Rebus free to process the OperationMessage first and then try the DispatchMessage's later.

Perhaps this isn't a problem to be solved in Rebus in which case I guess the DispatchMessage handler could skip trying the 3rd party API, and re-queue the message with a delay (deferred).

// my basic setup
services.AddRebus(configure =>
    configure
        .Logging(logging => logging.Serilog())
        .Transport(transport => ...)
        .Routing(routingConfigurer => ...)
        .Options(options => options.EnableDiagnosticSources())
        .Serialization(serialization => serialization.UseNewtonsoftJson(CreateJsonSerializerSettings())),
    onCreated: async (IBus bus) => 
    {
        await bus.Subscribe<DispatchMessage>();
        await bus.Subscribe<OperationMessage>();
    });

[mookid8000]

Hi @chrismoutray , if you would like to build a custom retry strategy with variable delays, you might want to look into the transport message API, which is capable of "deferring transport messages" (i.e. send the message currently being handled with a delay)

await bus.Advanced.TransportMessage.Defer(TimeSpan.FromMinutes(1));

and the special rbs2-defer-count header (conveniently accessible via Headers.DeferCount), which is automatically incremented each time a transport message is deferred. With these two mechanisms and a little bit of clever code it's possible to build pretty sophisticated retry strategies.

Please remember to back up your clever retry strategy with some integration tests. 🙂