Pause generation

[system1system2]

Right now we have Interrupt and Skip, which are great. But oftentimes, especially when generating 16-image batches or doing a big X/Y plot, I'd really like to have a Pause button.

[Gerschel]

What would the expectation be for a pause button?

Let's assume there was one, there are different ways to implement it. An easy to implement method would be, set a pause flag and enter a loop that listens for pause flag to flip into resume. You wouldn't be able to start another process without breaking the one that's hanging around. If you're just going to refill your coffee, but want to come back and monitor output, this works.

For a more complex pause, maybe for a purpose such as running another function, and then resume on the original. It'll need to save users settings and position through process, then exit entirely. Then an import project and resume would need to be created. A saved project could break though on an update, example being similar to the situation when sampler_index was renamed to sampler_name, your project file would use the wrong parameter name and format in that situation, and wouldn't work without manual intervention.

I like the idea of pause and save project. In fact, I wish we had a start project function, which allowed us to have unique settings for the entire interface. This way I could find my images in the named project folder, for example, if I used the same word prompts in two projects, I could find the outputs segregated by project, then I could name each project by goal I think I'm going for.

[system1system2]

The use case that triggered my request is this:

I'm generating a big X/Y plot on my Mac M1. So while A1111 is working, I do other things. Among those, all of a sudden, I have to run a GPU-intensive workload, like watching a video or joining a video call. If the two things run concurrently, the whole system gets quite slow in even the most basic window animations. If the video is 5min long, then fine. If the video is 30min long or it's a 1h video call, I'd prefer to pause A1111 and watch it, rather than having to interrupt the generation and restart from scratch.

It sounds like my use case matches your "more complex pause" situation.

[Scn64]

Yes, I'd love a pause button too. I've been running a big batch of images for several hours. I have a little over one hour left but would like to play a GPU intensive game. Would be great to just pause it, play the game for a couple hours, then resume it when I'm done.

[CrazyPieGuy]

I would also like a feature like this. A naive pause would be useful as well as the more complicated pause. My use case is upscaling images. It requires generating large amounts of images that can take many hours to days of processing time. It would be nice to be able to pause the generation so that I can use my GPU for other tasks, and then resume the generation once I do not need my GPU.

[gndqb]

Same here, pause would be awesome. Running videos for many hours, there is need to pause and then resume from there.

[kuso-ge]

I'm surprised no one has gotten around to implementing this

[kuso-ge]

maybe the pause doesn't need to happen during the current generation but for the next one so it's easier to implement?

[noisefloordev]

That's definitely the way to go. Finish or cancel the current gen (cancellation is probably better: the user wants his GPU back, or the server is about to disappear) so it doesn't need to leave a half-finished generation running or the queue locked. Then you only need the state of the generation (input params + current seed/subseed) to restart it. That way everything is client-side, so it should work for @KeithSieman's use case where you might be coming up in a new server instance.

It also could allow restarting a job that fails partway through unexpectedly: send the current state to the web client at the start of each generation, so if the server crashes or the VM disappears, the web UI already has the info it needs to resume where it left off.

I'm sure there are corner cases that would take more work, like extensions that have ongoing state during a job that needs serialization (eg. loopbacks), and you'd want to be able to serialize and reload extension settings (maybe that's already possible as part of generation info).

[KeithSieman]

Just thought of another edge use-case for some kind of a pause functionality.

Azure spot VMs, when being evicted for capacity, have the capability of using Azure Scheduled Events to trigger a series of actions within the VM. The eviction could trigger the pause functionality and then write progress and whatever else it needs to a file immediately, then be resumed once the VM is back online again. There is a 30 second window between eviction and deallocation, so it'd have to be quick acting.

[hellridergit]

I came here with just this idea. My use case scenario is running larger X/Y/Z plots that are troublesome to split into just X/Y plots and enqueue them with the agent-scheduler extension.

Often throughout the day I need to quickly start some rendering jobs for 20-30 minutes and it would be great to be able to pause the generation even if it had to discard just that one current image on pause and restart it on resume.