The installation of scraper package is failling.

windg

RPi4
Official Psu
Sd-card 32 gb (2yr old)
RetroPie 4.8.4 (manual installation on top of Buster legacy)

~~I try to install this scraper package(Steven Selph) but it is failing. I don't know if there is a conflit with the skyscraper that i have already on the system. Thanks~~

~~log file : https://pastebin.com/AkDtTbNd~~

I just find that is a upstream issue.

Lolonois

@windg could you be more specific abt the upstream?

AFAIK math/bits (the missing include error in the pastebin-log) was introduced with golang 1.9 onwards.

The golang.sh script module does a sloppy version comparison, thus go1.8.7 may be enforced/installed. I propose a quick patch, which pulls in golang 1.11 (which is currently used in Buster) and does proper version comparison:

diff --git a/scriptmodules/supplementary/golang.sh b/scriptmodules/supplementary/golang.sh
index 0ac552df..58fa4589 100644
--- a/scriptmodules/supplementary/golang.sh
+++ b/scriptmodules/supplementary/golang.sh
@@ -18,14 +18,15 @@ function _get_goroot_golang() {
 }
 
 function install_bin_golang() {
-    local version
+    local version="0"
+    local min_version="1.11.13"
+
     if [[ -e "$md_inst/bin/go" ]]; then
-        local version=$(GOROOT="$md_inst" "$md_inst/bin/go" version | sed 's/.*\(go1[^ ]*\).*/\1/')
+        local version=$(GOROOT="$md_inst" "$md_inst/bin/go" version | sed 's/.*go\(.*\)\s.*/\1/')
     fi
+
     printMsgs "console" "Current Go version: $version"
-    if [[ ! "${version}" < "go1.8.7" ]]; then
-        return 0
-    fi
+    compareVersions "${version}" ge "${min_version}" && return 0
 
     rm -rf "$md_inst"
     mkdir -p "$md_inst"
@@ -40,6 +41,6 @@ function install_bin_golang() {
     if isPlatform "aarch64"; then
         arch="arm64"
     fi
-    printMsgs "console" "Downloading go1.8.7.linux-$arch.tar.gz"
-    downloadAndExtract "https://storage.googleapis.com/golang/go1.8.7.linux-$arch.tar.gz" "$md_inst" --strip-components 1
+    printMsgs "console" "Downloading go${min_version}.linux-$arch.tar.gz"
+    downloadAndExtract "https://storage.googleapis.com/golang/go${min_version}.linux-$arch.tar.gz" "$md_inst" --strip-components 1
 }

windg

@Lolonois I find this older topic but doesn't have details : https://retropie.org.uk/forum/topic/32847/could-not-install-scraper-steven-selph?_=1690218810409

mitu

@windg Update RetroPie-Setup and re-try the installation.

windg

@mitu Thanks to looking into this!

I try to reinstal, after update the setup-script and it's failing again :

Log started at: Mon 24 Jul 2023 09:26:04 PM EEST

RetroPie-Setup version: 4.8.4 (5fceb82)
System: rpi4 (armv7l) - Raspbian GNU/Linux 10 (buster) - Linux retropie 5.10.103-v7l+ #1529 SMP Tue Mar 8 12:24:00 GMT 2022 armv7l GNU/Linux

= = = = = = = = = = = = = = = = = = = = =
Installing dependencies for 'scraper' : Scraper for EmulationStation by Steven Selph
= = = = = = = = = = = = = = = = = = = = =


= = = = = = = = = = = = = = = = = = = = =
Installing (binary) 'golang' : Golang binary install
= = = = = = = = = = = = = = = = = = = = =

Current Go version: go1.8.7
/home/pi/RetroPie-Setup/tmp/build/scraper /home/pi

= = = = = = = = = = = = = = = = = = = = =
Getting sources for 'scraper' : Scraper for EmulationStation by Steven Selph
= = = = = = = = = = = = = = = = = = = = =

package math/bits: unrecognized import path "math/bits" (import path does not begin with hostname)
Note: checking out '43ed8a0b364ed2d8521d0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 43ed8a0 Add 403 errors
/home/pi
/home/pi/RetroPie-Setup/tmp/build/scraper /home/pi

= = = = = = = = = = = = = = = = = = = = =
Building 'scraper' : Scraper for EmulationStation by Steven Selph
= = = = = = = = = = = = = = = = = = = = =

src/google.golang.org/protobuf/encoding/protowire/wire.go:15:2: cannot find package "math/bits" in any of:
	/opt/retropie/supplementary/golang/src/math/bits (from $GOROOT)
	/home/pi/RetroPie-Setup/tmp/build/scraper/src/math/bits (from $GOPATH)
/home/pi
/home/pi/RetroPie-Setup/tmp/build/scraper /home/pi
Could not successfully install Scraper for EmulationStation by Steven Selph (/home/pi/RetroPie-Setup/tmp/build/scraper/scraper not found).
/home/pi

Log ended at: Mon 24 Jul 2023 09:26:31 PM EEST
Total running time: 0 hours, 0 mins, 27 secs

mitu

Try removing the golang package first.

windg

I try sudo apt-get remove go1.8.7 and sudo apt-get remove golang but it says that this package is not installed.

mitu

@windg The RetroPie golang package, not the system package.

windg

@mitu Many thanks is working now !

Mathes75

Have the same issue, sorry to ask - but what's the difference between the RetroPie golang package and the system package? How to remove the RetroPie golang package?

windg

@Mathes75 To uninstall the golang :

First update RetroPie-Setup script, then, go to RetroPie Menu >> RetroPie Setup >> Manage packages >> Manage dependency packages >> uninstall and reinstall golang package. After that try to install again the scraper .

Mathes75

@windg OK. I was able to locate the package, uninstall and reinstalls go1.11.13, but still no luck with scraper :(

Here is the interesting part from the log...

# cd /home/pi/RetroPie-Setup/tmp/build/scraper/src/github.com/J-Swift/thegamesdb-swagger-client-go; git pull --ff-only
You are not currently on a branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.

    git pull <remote> <branch>

package github.com/J-Swift/thegamesdb-swagger-client-go: exit status 1
HEAD is now at 43ed8a0 Add 403 errors
error: patch failed: ds/ss.go:230
error: ds/ss.go: patch does not apply
error: patch failed: ds/ss_mame.go:2
error: ds/ss_mame.go: patch does not apply
error: patch failed: ss/ss.go:14
error: ss/ss.go: patch does not apply
/home/pi
/home/pi/RetroPie-Setup/tmp/build/scraper /home/pi

= = = = = = = = = = = = = = = = = = = = =
Building 'scraper' : Scraper for EmulationStation by Steven Selph
= = = = = = = = = = = = = = = = = = = = =

# golang.org/x/oauth2/internal
src/golang.org/x/oauth2/internal/token.go:140:10: lc.v.CompareAndSwap undefined (type atomic.Value has no field or method CompareAndSwap)
/home/pi
/home/pi/RetroPie-Setup/tmp/build/scraper /home/pi
Could not successfully install Scraper for EmulationStation by Steven Selph (/home/pi/RetroPie-Setup/tmp/build/scraper/scraper not found).
/home/pi

windg

I can confirm this, it looks that something it is not working correct.

Mathes75

OK. First switch brain on, then start actions, then complain.

I got Scraper installed by copying scraper files from githib.com folder in temp to folder that is not found above, but now it doesn't work. No new ROM info is fetched. Error message is referring to /opt/retropie/supplementary/scraper/scraper not found. All files seem to be there...

Lolonois

The issue is twofold.

@Mathes75 said in The installation of scraper package is failling.:

error: patch failed: ds/ss.go:230

This can be remediated by running sudo RetroPie-Setup/retropie_packages.sh scraper clean and then sudo RetroPie-Setup/retropie_packages.sh scraper

But hold your horses: The second issue needs attention too.

@Mathes75 said in The installation of scraper package is failling.:

src/golang.org/x/oauth2/internal/token.go:140:10: lc.v.CompareAndSwap undefined (ty...

This requires a newer golang version (was introduced in 1.17 AFAIK). My bad on suggesting 1.11 in the first place. However, since 1.17 some commands have been deprecated, which also needed to be reflected in the scraper.sh scriptmodule.

TL;DR: Here is a patch.

Save the patchfile somewhere on your Pi, then run in folder /home/pi/RetroPie-Setup this command: git apply /path/to/patchfile. No worries, you can undo the patch with git apply --reverse /path/to/patchfile

Then run in this order:

sudo RetroPie-Setup/retropie_packages.sh golang
sudo RetroPie-Setup/retropie_packages.sh scraper clean
sudo RetroPie-Setup/retropie_packages.sh scraper

It should provide you a scraper binary. I did not try to scrape but you pls try and report back.

HTH

PS: Once more this is only life-support for the Scraper. The upstream git repo of Scraper hat it's last.update in 2019 the retropie scriptmodule uses some duct tape (read patches) to keep it alive. I suggest using skyscraper. This skyscraper branch is from late 2022.

windg

Thanks for your time to looking into this.
I try the patch but i have some errors when i try to apply it.

pi@retropie:~/RetroPie-Setup $ git apply /home/pi/rp_modules_golang_scraper.patch
error: patch failed: scriptmodules/supplementary/golang.sh:18
error: scriptmodules/supplementary/golang.sh: patch does not apply
error: patch failed: scriptmodules/supplementary/scraper.sh:22
error: scriptmodules/supplementary/scraper.sh: patch does not apply

I believe that is not worth your time to try to fix it, there better alternatives out there. Thanks again !

mitu

@windg Try cleaning up the scraper sources first, before attempting to re-build.

windg

@mitu I have remove both scraper and golang packages but it's still failing.

= = = = = = = = = = = = = = = = = = = = =
Getting sources for 'scraper' : Scraper for EmulationStation by Steven Selph
= = = = = = = = = = = = = = = = = = = = =

go: go.mod file not found in current directory or any parent directory.
	'go get' is no longer supported outside a module.
	To build and install a command, use 'go install' with a version,
	like 'go install example.com/cmd@latest'
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.
fatal: cannot change to '/home/pi/RetroPie-Setup/tmp/build/scraper/src/github.com/J-Swift/thegamesdb-swagger-client-go/': No such file or directory
fatal: cannot change to '/home/pi/RetroPie-Setup/tmp/build/scraper/src/github.com/sselph/scraper': No such file or directory
/home/pi

= = = = = = = = = = = = = = = = = = = = =
Building 'scraper' : Scraper for EmulationStation by Steven Selph
= = = = = = = = = = = = = = = = = = = = =

no required module provides package github.com/sselph/scraper: go.mod file not found in current directory or any parent directory; see 'go help modules'
Could not successfully install Scraper for EmulationStation by Steven Selph (/home/pi/RetroPie-Setup/tmp/build/scraper/scraper not found).

mitu

@windg Yeah, it won't work. I'll take a look later on, but that may be the last patchwork for this package.

windg

I believe even if you fix it it will be broken again soon or later. Maybe it's time to drop this package and save your time.