Effective branch & release management

[Pwuts]

Current state of affairs

We are working on two types of changes:

• Atomic changes that improve the stability and/or performance of Auto-GPT
• Re-architectural / breaking changes

Current workflow

We work in master, merging pull requests when they are deemed ready, and doing a release whenever we feel it is appropriate. Bigger changes are often split up into smaller PRs, even if the end goal is only achieved with all of the changes combined.

Desired properties of new workflow

• Merge atomic pull requests when the following conditions are met:
  • it addresses an acknowledged issue or desire for improvement
  • it has been reviewed by a maintainer and is up to standards
  • it has been tested to not introduce any regression
• Publish releases regularly (e.g. weekly), to ensure we publish improvements at a steady pace
• Allow work on non-atomic changes for as long as it takes to get them ready for release, without blocking or being blocked by unrelated changes or releases

Challenges in implementing new workflow

The unwritten rule that big PRs may be split up into smaller incremental changes conflicts with the desire to publish regular, curated releases. Our current branch/release management does not provide a clear way to process partial or breaking PRs without having them end up in the next release.
Currently, PRs which contain breaking changes have to remain open until the time is right, blocking progress on the overall objective when the PR only implements a part of the envisioned solution.

Proposal

There are multiple options to solve the challenges above:

1. Release branches
   For every major version, a release branch is created off master, and for each release, changes are either selected and cherry-picked from master, or non-breaking PRs are merged directly into the release branch, which is merged back into master whenever necessary, with master containing all changes that are not destined for the next release.

2. Feature branches (practical inverse of Release branches)
   For breaking changes, a feature branch + corresponding draft PR is created. The objective and todo's are listed in the draft PR, and it is merged to master only after all of the todo's are checked off, indicating it is ready to be shipped in a release. So no step-by-step addressing an issue with smaller incremental PRs to master: this all happens in a feature branch.

   Using feature branches should be combined with the requirement that anything merged into master must be ready to ship.

[Pwuts]

After giving this some thought, I think option 2 (consistently using feature branches) is our best option. It may sound straightforward, but it is not how we currently work.

Catalysts and maintainers would have to be more strict in reviewing and curating PRs, only merging ones that are actually ready to ship.

Bigger pull requests would become more common, and devs working on them must be made aware that they should sync their fork with master regularly, to prevent conflicts from accumulating.

This would also mean that the scope of my PR #4208 must be extended to include at least some of the envisioned performance improvements. I'm not too enthusiastic about solving more merge conflicts, but I guess it will have to do.

[ntindle]

I’m okay with option 2 if and only if it has the same branch protections as master as that forces smaller real reviews