[SOFT] New Scraper in the works

kiro

@Folly there seems to be an issue with the database and some video naming, I fixed the video naming conventions in the front-end, but still need some work in the backend. Thanks again! And yes, great teamwork!

Enjoy your weekend!

Folly

@kiro said in [SOFT] New Scraper in the works:

Enjoy your weekend!

You too.

Folly

@kiro

Today I changed how the dependencies for retroscraper-remote are installed in my WIP mamedev.sh module-script as I wasn't happy about how it was.

It took me quite some time to figure things out but I am quite happy with the result as it looks like it's a bit faster.

This is the commit I made.

This is how the function looks now :

function retroscraper_remote_depends_mamedev () {
	#install pip if the pip module is not installed
    if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then
        su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py"
        su $user -c "python3 /tmp/get-pip.py"
    fi

    #update pip if there is a new release
	[[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip"

    #install python modules when not detected as installed
    #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online"
    #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix
	local retroscraper_remote_module
	local retroscraper_remote_modules=()
	retroscraper_remote_modules=(
	wheel==0.40.0
	setuptools==67.7.2
	googletrans==4.0.0rc1
	Pillow==9.2.0
	requests==2.21.0
	httpimport==1.1.0
	)
	for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do 
	[[ $(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'") != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module"
	done
}

I think you will find it interesting too.

What is your opinion about also using a fixed version for wheel and setuptools ?

If you have a question regarding this, then let me know.

Edit :
After testing it more it still seems that it's a bit slow on the raspberry pi.
Will have a second look if I can improve it.

Folly

@kiro

By changing the function slightly I doubled the speed.
See this commit

This is the function now :

function retroscraper_remote_depends_mamedev () {
	#install pip if the pip module is not installed
    if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then
        su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py"
        su $user -c "python3 /tmp/get-pip.py"
    fi

    #update pip if there is a new release (2>&1 is used to redirect stderr to stdout so we can use the if function on the stderr info)
	[[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip"

    #install python modules when not detected as installed
    #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online"
    #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix
	local pip_list_output
	local retroscraper_remote_module
	local retroscraper_remote_modules=()
	retroscraper_remote_modules=(
	wheel==0.40.0
	setuptools==67.7.2
	googletrans==4.0.0rc1
	Pillow==9.2.0
	requests==2.21.0
	httpimport==1.1.0
	)
	#this command was the first test and filters out the required modules but has no speed advantage vs the used command
	#pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'|awk \"/${retroscraper_remote_modules[0]}/ || /${retroscraper_remote_modules[1]}/ || /${retroscraper_remote_modules[2]}/ || /${retroscraper_remote_modules[3]}/ ||  /${retroscraper_remote_modules[4]}/ || /${retroscraper_remote_modules[5]}/\"")
	pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'")
	
	for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do 
	[[ $pip_list_output != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module"
	done
}

kiro

@Folly Hey thanks! Very useful indeed. I like the specific version switch, since sometimes newer versions of libraries have breaking changes.

Folly

@kiro said in [SOFT] New Scraper in the works:

I like the specific version switch, since sometimes newer versions of libraries have breaking changes

Sure, I think so too.
I think you know but with this command you can list the outdated versions :

python3 -m pip list --outdated

So we can find the latest versions and update the versions one at a time in the function I made.
That way we can test if a newer module version breaks the script.
If not we can update the versions in the script.

What do you think ?

Folly

@kiro

I updated all modules to the latest version.
Only these 3 are changed : Pillow==9.5.0 requests==2.30.0 httpimport==1.3.0

See this commit

This is the function now :

function retroscraper_remote_depends_mamedev () {
	#install pip if the pip module is not installed
    if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then
        su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py"
        su $user -c "python3 /tmp/get-pip.py"
    fi

    #update pip if there is a new release (2>&1 is used to redirect stderr to stdout so we can use the if function on the stderr info)
	[[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip"

    #install python modules when not detected as installed
    #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online"
    #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix
	local pip_list_output
	local retroscraper_remote_module
	local retroscraper_remote_modules=()
	retroscraper_remote_modules=(
	wheel==0.40.0
	setuptools==67.7.2
	googletrans==4.0.0rc1
	Pillow==9.5.0
	requests==2.30.0
	httpimport==1.3.0
	)
	#this command was the first test and filters out the required modules but has no speed advantage vs the used command
	#pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'|awk \"/${retroscraper_remote_modules[0]}/ || /${retroscraper_remote_modules[1]}/ || /${retroscraper_remote_modules[2]}/ || /${retroscraper_remote_modules[3]}/ ||  /${retroscraper_remote_modules[4]}/ || /${retroscraper_remote_modules[5]}/\"")
	pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'")
	
	for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do 
	[[ $pip_list_output != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module"
	done
}

Folly

Hi @kiro ,

I think I found an issue when scraping msx stuff.
For some games a number was detected as name and :113 is added after the zip file.
These "name" numbers / *.zip:113 aren't correctly displayed also in emulationstation.

Here is part of the retropie log :

System: msx2 | Game : Game World - 126 Games
System: msx2 | Game : Gun Fright
System: msx2 | Game : 95372
Gw126.zip:113

System: msx2 | Game : 95372
Gw126.zip:113

System: msx2 | Game : 95372
Gw126.zip:113

System: msx2 | Game : Hades
System: msx2 | Game : Desperado
System: msx2 | Game : Hafanuda Koi Koi - Gostop Godori
System: msx2 | Game : Miracle Warriors - Seal Of The Dark Lord
System: msx2 | Game : Gyrodine
System: msx2 | Game : Harapeko Pakkun
System: msx2 | Game : Hang-on
System: msx2 | Game : Hb-3600 Controller + Hb-6000 5.25" Dsdd Floppy Drive
System: msx2 | Game : Hanafuta
System: msx2 | Game : Hbd-50
System: msx2 | Game : 95524
Hboxing.rom:113

System: msx2 | Game : Hbd-f1 (floppy Controller + 3.5" Dsdd Floppy Drive)
System: msx2 | Game : Head & Tail
System: msx2 | Game : Hbd-20w (floppy Controller + 3.5" Dsdd Floppy Drive)
System: msx2 | Game : Heavy Boxing
System: msx2 | Game : Heist, The
System: msx2 | Game : 95372
Gw126.zip:113

System: msx2 | Game : hfox2.zip

Can you have a look ?

kiro

Hi @Folly ~~! Sorry I didn't get your meaning. Does the number also show in the gamelists? thanks!~~

Forget it, it seems to happen with just two games. Should get fixed in the nightly refresh. Sorry for that!

Folly

@kiro

FYI, In emulationstation both name(number) and *.zip:113 are displayed over each other in one line.

Thanks I will try tomorrow.

kiro

@Folly I'm in the process of adding around 100k amiga roms to the DB, so the nightly refresh takes a little bit more time than expected. Probably the fixes will not be visible yet.

Folly

@kiro

Great !
I loved the Amiga too.
The latest Amiberry is quite good though, definitely need to play it more often ;-)