[SOFT] New Scraper in the works

[Folly]

@kiro said in [SOFT] New Scraper in the works:

Enjoy your weekend!

You too.

[Folly]

@kiro

Today I changed how the dependencies for retroscraper-remote are installed in my WIP mamedev.sh module-script as I wasn't happy about how it was.

It took me quite some time to figure things out but I am quite happy with the result as it looks like it's a bit faster.

This is the commit I made.

This is how the function looks now :

function retroscraper_remote_depends_mamedev () {
	#install pip if the pip module is not installed
    if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then
        su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py"
        su $user -c "python3 /tmp/get-pip.py"
    fi

    #update pip if there is a new release
	[[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip"

    #install python modules when not detected as installed
    #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online"
    #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix
	local retroscraper_remote_module
	local retroscraper_remote_modules=()
	retroscraper_remote_modules=(
	wheel==0.40.0
	setuptools==67.7.2
	googletrans==4.0.0rc1
	Pillow==9.2.0
	requests==2.21.0
	httpimport==1.1.0
	)
	for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do 
	[[ $(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'") != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module"
	done
}

I think you will find it interesting too.

What is your opinion about also using a fixed version for wheel and setuptools ?

If you have a question regarding this, then let me know.

Edit :
After testing it more it still seems that it's a bit slow on the raspberry pi.
Will have a second look if I can improve it.

[Folly]

@kiro

By changing the function slightly I doubled the speed.
See this commit

This is the function now :

function retroscraper_remote_depends_mamedev () {
	#install pip if the pip module is not installed
    if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then
        su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py"
        su $user -c "python3 /tmp/get-pip.py"
    fi

    #update pip if there is a new release (2>&1 is used to redirect stderr to stdout so we can use the if function on the stderr info)
	[[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip"

    #install python modules when not detected as installed
    #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online"
    #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix
	local pip_list_output
	local retroscraper_remote_module
	local retroscraper_remote_modules=()
	retroscraper_remote_modules=(
	wheel==0.40.0
	setuptools==67.7.2
	googletrans==4.0.0rc1
	Pillow==9.2.0
	requests==2.21.0
	httpimport==1.1.0
	)
	#this command was the first test and filters out the required modules but has no speed advantage vs the used command
	#pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'|awk \"/${retroscraper_remote_modules[0]}/ || /${retroscraper_remote_modules[1]}/ || /${retroscraper_remote_modules[2]}/ || /${retroscraper_remote_modules[3]}/ ||  /${retroscraper_remote_modules[4]}/ || /${retroscraper_remote_modules[5]}/\"")
	pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'")
	
	for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do 
	[[ $pip_list_output != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module"
	done
}

[kiro]

@Folly Hey thanks! Very useful indeed. I like the specific version switch, since sometimes newer versions of libraries have breaking changes.

[Folly]

@kiro said in [SOFT] New Scraper in the works:

I like the specific version switch, since sometimes newer versions of libraries have breaking changes

Sure, I think so too.
I think you know but with this command you can list the outdated versions :

python3 -m pip list --outdated

So we can find the latest versions and update the versions one at a time in the function I made.
That way we can test if a newer module version breaks the script.
If not we can update the versions in the script.

What do you think ?

[Folly]

@kiro

I updated all modules to the latest version.
Only these 3 are changed : Pillow==9.5.0 requests==2.30.0 httpimport==1.3.0

See this commit

This is the function now :

function retroscraper_remote_depends_mamedev () {
	#install pip if the pip module is not installed
    if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then
        su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py"
        su $user -c "python3 /tmp/get-pip.py"
    fi

    #update pip if there is a new release (2>&1 is used to redirect stderr to stdout so we can use the if function on the stderr info)
	[[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip"

    #install python modules when not detected as installed
    #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online"
    #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix
	local pip_list_output
	local retroscraper_remote_module
	local retroscraper_remote_modules=()
	retroscraper_remote_modules=(
	wheel==0.40.0
	setuptools==67.7.2
	googletrans==4.0.0rc1
	Pillow==9.5.0
	requests==2.30.0
	httpimport==1.3.0
	)
	#this command was the first test and filters out the required modules but has no speed advantage vs the used command
	#pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'|awk \"/${retroscraper_remote_modules[0]}/ || /${retroscraper_remote_modules[1]}/ || /${retroscraper_remote_modules[2]}/ || /${retroscraper_remote_modules[3]}/ ||  /${retroscraper_remote_modules[4]}/ || /${retroscraper_remote_modules[5]}/\"")
	pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'")
	
	for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do 
	[[ $pip_list_output != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module"
	done
}

[Folly]

Hi @kiro ,

I think I found an issue when scraping msx stuff.
For some games a number was detected as name and :113 is added after the zip file.
These "name" numbers / *.zip:113 aren't correctly displayed also in emulationstation.

Here is part of the retropie log :

System: msx2 | Game : Game World - 126 Games
System: msx2 | Game : Gun Fright
System: msx2 | Game : 95372
Gw126.zip:113

System: msx2 | Game : 95372
Gw126.zip:113

System: msx2 | Game : 95372
Gw126.zip:113

System: msx2 | Game : Hades
System: msx2 | Game : Desperado
System: msx2 | Game : Hafanuda Koi Koi - Gostop Godori
System: msx2 | Game : Miracle Warriors - Seal Of The Dark Lord
System: msx2 | Game : Gyrodine
System: msx2 | Game : Harapeko Pakkun
System: msx2 | Game : Hang-on
System: msx2 | Game : Hb-3600 Controller + Hb-6000 5.25" Dsdd Floppy Drive
System: msx2 | Game : Hanafuta
System: msx2 | Game : Hbd-50
System: msx2 | Game : 95524
Hboxing.rom:113

System: msx2 | Game : Hbd-f1 (floppy Controller + 3.5" Dsdd Floppy Drive)
System: msx2 | Game : Head & Tail
System: msx2 | Game : Hbd-20w (floppy Controller + 3.5" Dsdd Floppy Drive)
System: msx2 | Game : Heavy Boxing
System: msx2 | Game : Heist, The
System: msx2 | Game : 95372
Gw126.zip:113

System: msx2 | Game : hfox2.zip

Can you have a look ?

[kiro]

Hi @Folly ! Sorry I didn't get your meaning. Does the number also show in the gamelists? thanks!

Forget it, it seems to happen with just two games. Should get fixed in the nightly refresh. Sorry for that!

[Folly]

@kiro

FYI, In emulationstation both name(number) and *.zip:113 are displayed over each other in one line.

Thanks I will try tomorrow.

[kiro]

@Folly I'm in the process of adding around 100k amiga roms to the DB, so the nightly refresh takes a little bit more time than expected. Probably the fixes will not be visible yet.

[Folly]

@kiro

Great !
I loved the Amiga too.
The latest Amiberry is quite good though, definitely need to play it more often ;-)