Update from Binary, where do the binaries come from?
-
I understand that updating from source causes RetroPie-Setup to download the latest sources from various GitHub repository branches (depending on the emulator), and compile them. But this is incredibly slow on my RPi 3.
So I decided to only update from binaries. This seems fast, but which binaries are installed? Where do they come from? It seems the binary versions are lower than the latest release versions on github.
I looked in https://www.github.com/retropie/retropie-setup but I could not find any reference to binaries. I find lots of scripts that download sources, but no binaries.
So where does RetroPie download the binaries from?
For example, I updated Amiberry from binaries, and ended up with version 2.18.2, an old version. The newest binary on https://www.github.com/midwan/amiberry/releases is version 2.19, the latest source version is already 2.21.
So where the heck is RetroPie getting this old Amiberry binary version 2.18 from?
I suspect that there's some secret repository somewhere with hundreds of outdated binaries. And RetroPie is hard-coded to download binaries from that repo? Is this true? If yes, where is this mysterious repository full of outdated versions?
-
Yeah, RetroPie is hard coded with this repo. It's really messy with all kinds of variable-ception going on in the code.
Digging through the code, I found out the install_bin function (the code that lets you install binaries in the script) references to variable "__binary_host", which is part of variable "__binary_url". This code contains the URL where the binaries are located. Here is the root directory of what I found:
https://files.retropie.org.uk/binaries/
(This URL is forbidden, so you can only access the files from their respective URLs in this repo.)
-
Excellent, thank you!
So it is exactly like I thought.
Why is this undocumented, and access to the URL is forbidden?
What is the rationale to use a secret/forbidden URL "https://files.retropie.org.uk/binaries/" for this, instead of a GitHub repo? If it was a GitHub repo, then at least all of us could take a look at the binaries, and/or re-direct to later versions ourselves.
Maybe the filesizes are a problem, because the binaries take up a lot of space?
And how are users supposed to figure out when the files on https://files.retropie.org.uk/binaries/ are updated? I suppose they are updated with every version bump of RetroPie? But to which versions?
This is completely opaque. There's absolutely no way of telling which versions are on there, and when they change.
-
@rsn8887 yep something sinister going on with this super secret repo of magically unkowns!
-
Indeed :)
I suppose the reasoning for this is as follows. Probably the contents of https://files.retropie.org.uk/binaries/ are only updated whenever RetroPie receives a new version number. And the binaries on the server are simply the same versions that are included always in the latest RetroPie image.
So this server allows users to update their RetroPie installation to the latest binaries without haveing to re-write their complete image. But they will never be newer than that, even if an emulator received another update in the meantime.
This would also explain why the RetroPie binaries seem to "lag behind" both the github source and github release pages of the emulators.
Another possibility is that the contents of https://files.retropie.org.uk/binaries/ are just updated whenever the devs feel like it, completely independent of RetroPie versioning. If that is the case, then version control of RetroPie is only a very loose concept, since the binaries are not associated with any certain RetroPie version at all.
In either case, without any transparent version control and history of changes, there's currently no way for users to see which binaries were updated and when. There's also no way of predicting what versions will be installed on an 'install from binaries.'
-
-
@rsn8887 said in Update from Binary, where do the binaries come from?:
I understand that updating from source causes RetroPie-Setup to download the latest sources from various GitHub repository branches (depending on the emulator), and compile them. But this is incredibly slow on my RPi 3.
This is not necessarily true -- for example with RetroArch I believe that RetroPie-Setup uses a more recent RetroArch commit from github for building from source than it does for the binary, but there is not a guarantee that it will be the latest commit.
-
@markwkidd said in Update from Binary, where do the binaries come from?:
@rsn8887 said in Update from Binary, where do the binaries come from?:
I understand that updating from source causes RetroPie-Setup to download the latest sources from various GitHub repository branches (depending on the emulator), and compile them. But this is incredibly slow on my RPi 3.
This is not necessarily true -- for example with RetroArch I believe that RetroPie-Setup uses a more recent RetroArch commit from github for building from source than it does for the binary, but there is not a guarantee that it will be the latest commit.
But at least we can look at the .sh scripts in the RetroPie-Setup repository and see exactly which repo, commit, and branch is used. It might not be the latest, but it is clear which one it is.
With the binary server, there's no telling.
-
@rsn8887 said in Update from Binary, where do the binaries come from?:
With the binary server, there's no telling
While the binary package is not versioned, you can always check the emulator version at runtime. There's nothing hidden or obscured, everything is in the setup script.
-
@mitu said in Update from Binary, where do the binaries come from?:
@rsn8887 said in Update from Binary, where do the binaries come from?:
With the binary server, there's no telling
While the binary package is not versioned, you can always check the emulator version at runtime. There's nothing hidden or obscured, everything is in the setup script.
No. The setup script doesn't show me how the binaries from files.retropie.org were built, which versions of them were built to make a certain image, which commands where used to build them. In fact there's no information about the binaries the setup script at all. It is impossible to build my own image that is identical to the retropie distribution image, because I wouldn't know which binaries where used, and the binaries on the server will probably already have been updated.
-
@rsn8887 For each package that has the source available (most of packages included in RetroPie), there is procedure that builds the package from source - this is the procedure used to produce the binaries hosted on the RetroPie site. There are some exceptions to this - for things like Drastic or CoolCV, for which the upstream authors don't provide sources.
You can, in fact, build your own RetroPie image by installing each package from source and you can modify each package's
build from sourceinstructions. You can even replicate the RetroPie image creation using the scripts in theadminscriptmodule folder.As I said, there's not hidden or obscured - if you're familiar with shell programming and compiling programs from source, you can easily read the RetroPie setup script.
-
I see, thanks for the detailed answer. If the admin scripts make it indeed possible to replicate the official image, then you are right, all the information is there and it is not a big deal that the binaries are hosted outside of version control.
-
@rsn8887 The only reason binaries exist is convenience - compiling on a low powered SoC like the Raspberry Pi takes a lot of resources (CPU, disk and memory) and a LOT of time. You'll notice that, despite supporting multiple platforms, the Pi is the only one that gets binaries. The other platforms only have the option to install from source (x86 Debian distros, Odroid, Asus Tinkerbox).
-
Perhaps it would be helpful if there were a readme file on https://files.retropie.org.uk/binaries/ saying which versions are currently hosted.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.