-
Notifications
You must be signed in to change notification settings - Fork 10.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New Release? #31585
Comments
A new release is overdue but depends on solving certain cross-platform issues in the CI build process. Meanwhile for the main platforms, see #31530 for update hints. |
What are those issues that need sorting out? Is there an issue/PR tracking this? Creating a PR could bring in people that are willing to help with the process (like myself) |
#30644, but it should probably be refreshed now. This is very far down my stack today, but as I recall there were various possible blockers:
Obviously I need to dive back into this especially in view of yt-dlp/yt-dlp#6220, which I would also like to acquire, if only for more regular versioned releases. |
But isn't the auto-update installer already there, depending on the how the yt-dl instance was installed? Once a build has been deployed to the release page and to PyPi, it can be pulled by almost all installation types. The problem being addressed is really to get the small number of builds in the release tested on all the different platforms. |
This comment was marked as off-topic.
This comment was marked as off-topic.
Will #30644 keep tracking this and should I post my remarks on there? I think maybe instead of perfecting the release process it might make sense to release something so as to get updates to the end user. If releases were not automated, whats problematic about building like that, then passing those executables to the user to be tried on various platforms? The related release can be marked as I would also advise to look into the These are the repository configuration options of the release and nighly action: vars.PUSH_VERSION_COMMIT: Update master branch with a push
vars.BUILD_NIGHTLY: Build a nightly release on every commit
# Both need to be set to push build to archive repo as release
vars.ARCHIVE_REPO: repo like `yt-dlp/yt-dlp-builds`
secrets.ARCHIVE_REPO_TOKEN: token with contents:write
# Push to PyPI and brew if each is set
secrets.PYPI_TOKEN: Token for PyPI
secrets.BREW_TOKEN: Token for brew
Are the test adjustments needed for the release for now? Is that preparation to be testing if the builds would work as expected? If so, I'd think that can be moved back for now?
Do we need to supply
This should be handled by 6220. I can help with the remaining blockers there.
Yes, understandable. If we encounter anything in particular we might need to push that back until somebody with experience on that system or with those dependencies can chime in. I think those issues will become apparent once we create/try creating the actual The steps that I would propose for now are:
Feel free to contact me via E-Mail if you would like to move to a different communication channel for this. Alternatively let me know of anything else I can do. |
This comment was marked as off-topic.
This comment was marked as off-topic.
Just to show that a release might appear quite soon, and not in the far future. From #31530 (comment)
No, although it's true that the GH workflow ecosystem forces you into an upgrade race as various dependencies are moved to new versions, much as in the rest of the s/w world. Actually, for yt-dl itself, Py2.7 is fine and dandy for the most part: the #31530 fix did reveal some 2.6 hold-outs but also surprisingly many verbose logs showing 2.7 still in use. The previous build and release system, not well documented, depended on some local setup that had been passed down through the former maintainers. Lured by the gleaming promise of online builds, I completely failed to reproduce the necessary local setup. That brings us to last month. Now, I find that GH is about to junk ("deprecated") versions of Linux that support the Pythons needed for yt-dl's CI testing. So we'll have to rework that completely using our own runners, etc. This is too hard for now and the existing tests will have to do: boo to Python >= 3.10 (though I don't expect significant issues there). Therefore the build and release workflow doesn't depend on the test workflow. I took the yt-dlp scheme as a pattern, as suggested, rather than building directly on the excellent work in #30644. Thus Users of new macOSes that have no Python will still have to install a Python and arrange for it to be found by the shebang of the POSIX build. Currently a tricky part is print((Changelog if args.markdown else PlainChangelog)(commits.groups(), args.repo)) Less simple, the 200 commits since the last release aren't as disciplined as the yt-dlp script expected. I think the only practical design for grouping these commits automatically is to use git commands to identify what each commit changed and then map back from there. For instance:
Given this, the solution is to take some output of an as-good-as-can-be-expected hack on the existing script and modify the CI process to accept some manually tweaked changelog delta that will be prepended with its version ID to the Obviously any other suggestions would be happily considered. |
Good to hear there is a path forward!
Yes, I believe the 18.04 runner is being removed in April if I remember correctly. I assume you're aware of this action to setup Python? https://github.com/actions/setup-python ... Example: https://github.com/LizardByte/Themerr-plex/blob/master/.github/workflows/CI.yml#L36-L49 Ubuntu 20.04 and 22.04 also still have python 2.7 that can be manually installed. Here's a build in an Ubuntu 22.04 dockerfile that uses Python 2.7 for reference: https://github.com/LizardByte/PlexyGlass/blob/nightly/Dockerfile#L4-L24 ... Debian/Ubuntu has too many packages that still rely on Python 2.x to completely remove it. I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there? |
Is there a new PR, some branch or similar that I can take a look at and implement missing factors to help out?
This is why I suggested shimming it for now, as manual mapping will be required anyways. Regarding future commits, it might make sense to adjust the commit message to be closer to dlp format OR (I thought about it before but scrapped it since its easier to just rely on message only) search for the files using file or text search if the format should be kept the same (
Assuming that this is building on the release workflow of dlp the PyPI publish is integrated in those steps It might make sense to diff with dlp build workflow as some new changes were coming. A relevant branch can be found here (though many changes were directly related to nightly and |
Homebrew project leader here 👋🏻. I think even getting a tagged prerelease here with failing CI will allow Homebrew to update to a version that will work for downloading from YouTube and stop you getting so many issues filed by our users. |
Thanks for your interest! Almost any commit to master is meant to be reliable and its CI test will have succeeded, having been exercised on another branch first. If you would like a special tag applied to some commit, that's easy. If you're able to take an unofficial nightly build (from a downstream), which has the advantage of reporting a different version and showing a disclaimer, that's even easier. |
Upcoming.
I suppose that a manually adjusted changelog delta could be offered to the workflow as a var, even if it's ~15kB?
For now I commented out the package management interfaces. The existing PyPi update process would have been manual AFAIK with any account details held by former maintainers; presumably we can regenerate them if they can't be recalled.
The workflow files
all started out from the yt-dlp equivalents in a meld session. So yes. |
Thanks for your great software that I personally use!
Yes, our hope would be any sort of tag applied to a release that would show up in https://github.com/ytdl-org/youtube-dl/releases that we could update to 🙇🏻 |
Which release asset is used to create the Homebrew package? |
A nightly release stays up unless some terrible bug is found that means downloading its assets would be a bad idea. In that sense they're all better than the 2021 release. Ofc you can still check out the bad tag and run the broken release if you want. |
it's not about bugs but about a reproducible build. i'm maintaining a distro and a package build script contains an upstream URL for the source tarball, a checksum of the tarball, and a few script instructions to build and install it. that means if a nightly tarball is only available for half a week the distro package recipe is basically useless. |
Check out the nightly release pages. I think you'll see that what you want is available, so long as GitHub keeps hosting the pages. Obvs don't build in a brand-new release in case some anomaly ((c) Post Office Ltd) has got through the CI and local tests. |
yeah, looks good. the nightlies from 2023 are still available. i guess it'd be good to make a link on the homepage or make a "final" tag here where you refer to the nightlies repo, so everyone seeing the new "release" here gets the news. |
hm, couldn't find this by searching the issues... It seems that some if not all of the issues mentioned are with making a binary release, but a new tag without any artifacts would be more than sufficient for having something for linux distros to upgrade to if they don't want to chase nightlies. People who use the prebuilts are presumably going to use nightlies anyway, they don't really care whether something is an official release. |
easiest installation currently (does not require
or to add
to your requirements.txt or your library requirements |
I don't have an issue installing youtube-dl, it's just that the README says that "nothing can be done" when a distro has outdated package, but making a release tag is definitely something that has to be done if the distro only packages release tags I'm pretty sure that at least in the single case of Tumbleweed, the reason the package is stuck on the 2021 version is that from their point of view there is no newer version. as a Linux user, I obviously don't have issues with using git since I can just install git using the package manager |
There's nothing to stop anyone packaging a nightly build (it has tags, etc) but it'll immediately be obsolete because of web site turnover. This is why the hassle of porting back the build process from the nightly repo hasn't seemed worthwhile.
Unlike the POSIX-ish single file bundled executable, the single file Windows build contains a Python executable and only needs the user to install ffmpeg for a good experience. The POSIX-ish targets are expected to have or be able to install a |
I'm not sure what problem that addresses. There is already an Actions workflow that creates nightly releases. A new release for every commit is not wanted. If you're suggesting cross-posting nightly releases, it's not just a matter of posting the release in the main repo: the releases are linked to the repo, so the nightly release updates from the nightly repo. It would be confusing for people to download a release and then have it update from a different source. |
hm, indeed https://github.com/ytdl-org/ytdl-nightly/releases has source tarballs... I guess it's just not on the radar for some packagers :/ |
If the nightly releases are the source of truth, this repository loses its value. It on the other hand also does not correlate to to a release being blocked on this repo. Nightly builds are being made and tested and using a workflow like that on this repository will require little changes. You can omit the duplication that does not match the actual development state since the two repositories are diverged. Instead, consider creating a prerelease on this repository, with future posibility of making it a regular release. Automation can be fine tuned to achieve this easily. I will offer my help on this topic again: Feel free to contact me through email if you want any help or someone to discuss with in private |
Technically not much would have to be done to pull the nightly workflow into the main repo, changing it to make pre-releases, but it would actually have to be done and with more detailed work, at least initially and especially when designating such a pre-release as a stable release. First off, the actions in the nightly workflow need to be brought up-to-date in GitHub's version hell actions ecosystem. Otherwise I just suggest that, if a packager needs an up-to-date tagged commit to ingest into its workflow, the nightly repo is a good choice; a packager can install the tar.gz as if from the main repo but presumably would have to patch the build process if delivering a single-file build for which |
Replying to:
No, nightlies tagged in a different repo are not releases. This is exactly what led to the current state:
|
Most downstream packagers are not going to be OK with arbitrarily selecting a pre-release from a repo other than the actual upstream repo; some have policies that forbid doing this. E.g. #31585 (comment) |
Also, Kodi is struggling a lot to keep this updated https://github.com/xbmc/repo-scripts/commits/matrix/script.module.youtube.dl |
There are no pre-releases. A nightly release may be retrospectively marked The complaint linked above referred to the ytdl-patched/youtube-dl (superseded by the nightly build repo here). Unlike ytdl-patched/youtube-dl, the nightly build repo is clearly an adjunct of the development repo. Maybe a nightly tag (s/t like |
The complaints linked are about tagging a regular release in THIS repo. None other. The project is this repository, and like I said before, rebasing or adding a nightly commit means they are DISJOINTED, and can no longer be seen as the same project / state. Creating a regular, stable git tag on this repository is trivial. Downstream packagers can do the remaining work. No binaries are required. Using the nightly build workflow on this repository to create binaries is optional, albeit also trivial. Interestingly doing both on Trying to reason against doing either is an interesting choice though |
So there is now https://github.com/ytdl-org/youtube-dl/tag/2024.07.11-nightly and so also https://github.com/ytdl-org/youtube-dl/archive/refs/tags/2024.07.11-nightly.tar.gz. Let's see if that helps. I have avoided, once GH allowed me to delete it, having an actual release page here since it isn't one. Otherwise, or anyway, I stand by the previous comments
For packagers? I don't see how that applies otherwise. |
Tried to clarify in #32876. Running that workflow when the repository is deemed stable will produce a stable release. |
Will the pypi packages be updated with any missed Nightly releases? Hasn't been updated since 2021. |
@hudsonm62 If you want @dependabot to keep it updated, you can do what I do. Add it as a submodule.
|
ah good idea, thank you sir I shall look into this for my use case |
Checklist
Question
Preface:
There have been a number of fixes done since the last release (2021-12-17) {over a years worth} The error messages refer using youtube-dl -U (which is already in place). All the errors I am experiencing are already closed issues, but no no release so youtube-dl -U won't fix it. Issue #31583
Question:
Can either the instructions be updated (and the error message refering to youtube-dl -U be dropped) to "install" via a github pull from Master
or
Can a new release be cut?
Thanks
The text was updated successfully, but these errors were encountered: