[SOFT] New Scraper in the works
-
No change but I am doing the scraping from my script which uses the -rpie version.
So you probably have to change some in there too. -
@Folly I've updated the -rpie version, are you sure you have the latest version from the repository? Thanks
-
I see you have just made a commit.
I just tried again, same result.
My script runs the retroscraper-rpie version "online" so it should be the latest.Did the dependencies change ?
-
@Folly no, I did not change any dependencies...do you have somehow a cache enabled in your setup that would not get the latest version? Not sure...I'm scanning my whole collection again and it seems to be working fine....
-
@kiro said in [SOFT] New Scraper in the works:
do you have somehow a cache enabled in your setup that would not get the latest version?
No I haven't, I also did a re-run of the RetroPie-Setup script.
-
@Folly if you try to get retroscraper and run it locally, do you have the same problem?
-
I just did, with this command :
python3 retroscraper.py --systems arcade --relativepaths
That works, I have videos now.
My script has a different command, will do that test also locally.
I tested this command, with similar options with the one in my script :
python3 retroscraper.py --systems arcade --recursive --relativepaths --mediadir media/emulationstation --nobackup
That works locally too now.
-
@Folly cool, let me know how it goes.
-
Running this locally also works :
curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3 - --systems arcade --recursive --relativepaths --mediadir media/emulationstation --nobackup
From my script however it still has the same issue.
I will reboot my system to see if this will help. -
@Folly yes, I totally forgot about the retroscraper -remote version, just uploaded the fix there too
-
Yea, I see it too now.
I was thinking also that it was the -rpie version but it's the remote indeed.
So It didn't have the fix yet.I rebooted and now it works again, great.
But the reboot would not have been necessary, I believe now.Very strange indeed that it somehow didn't work anymore but it seems it has been fixed ;-)
Thanks for fixing.
Good team work ! -
@Folly there seems to be an issue with the database and some video naming, I fixed the video naming conventions in the front-end, but still need some work in the backend. Thanks again! And yes, great teamwork!
Enjoy your weekend!
-
-
Today I changed how the dependencies for retroscraper-remote are installed in my WIP mamedev.sh module-script as I wasn't happy about how it was.
It took me quite some time to figure things out but I am quite happy with the result as it looks like it's a bit faster.
This is the commit I made.
This is how the function looks now :
function retroscraper_remote_depends_mamedev () { #install pip if the pip module is not installed if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py" su $user -c "python3 /tmp/get-pip.py" fi #update pip if there is a new release [[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip" #install python modules when not detected as installed #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online" #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix local retroscraper_remote_module local retroscraper_remote_modules=() retroscraper_remote_modules=( wheel==0.40.0 setuptools==67.7.2 googletrans==4.0.0rc1 Pillow==9.2.0 requests==2.21.0 httpimport==1.1.0 ) for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do [[ $(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'") != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module" done }
I think you will find it interesting too.
What is your opinion about also using a fixed version for wheel and setuptools ?
If you have a question regarding this, then let me know.
Edit :
After testing it more it still seems that it's a bit slow on the raspberry pi.
Will have a second look if I can improve it. -
By changing the function slightly I doubled the speed.
See this commitThis is the function now :
function retroscraper_remote_depends_mamedev () { #install pip if the pip module is not installed if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py" su $user -c "python3 /tmp/get-pip.py" fi #update pip if there is a new release (2>&1 is used to redirect stderr to stdout so we can use the if function on the stderr info) [[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip" #install python modules when not detected as installed #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online" #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix local pip_list_output local retroscraper_remote_module local retroscraper_remote_modules=() retroscraper_remote_modules=( wheel==0.40.0 setuptools==67.7.2 googletrans==4.0.0rc1 Pillow==9.2.0 requests==2.21.0 httpimport==1.1.0 ) #this command was the first test and filters out the required modules but has no speed advantage vs the used command #pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'|awk \"/${retroscraper_remote_modules[0]}/ || /${retroscraper_remote_modules[1]}/ || /${retroscraper_remote_modules[2]}/ || /${retroscraper_remote_modules[3]}/ || /${retroscraper_remote_modules[4]}/ || /${retroscraper_remote_modules[5]}/\"") pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'") for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do [[ $pip_list_output != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module" done }
-
@Folly Hey thanks! Very useful indeed. I like the specific version switch, since sometimes newer versions of libraries have breaking changes.
-
@kiro said in [SOFT] New Scraper in the works:
I like the specific version switch, since sometimes newer versions of libraries have breaking changes
Sure, I think so too.
I think you know but with this command you can list the outdated versions :python3 -m pip list --outdated
So we can find the latest versions and update the versions one at a time in the function I made.
That way we can test if a newer module version breaks the script.
If not we can update the versions in the script.What do you think ?
-
I updated all modules to the latest version.
Only these 3 are changed : Pillow==9.5.0 requests==2.30.0 httpimport==1.3.0See this commit
This is the function now :
function retroscraper_remote_depends_mamedev () { #install pip if the pip module is not installed if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py" su $user -c "python3 /tmp/get-pip.py" fi #update pip if there is a new release (2>&1 is used to redirect stderr to stdout so we can use the if function on the stderr info) [[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip" #install python modules when not detected as installed #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online" #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix local pip_list_output local retroscraper_remote_module local retroscraper_remote_modules=() retroscraper_remote_modules=( wheel==0.40.0 setuptools==67.7.2 googletrans==4.0.0rc1 Pillow==9.5.0 requests==2.30.0 httpimport==1.3.0 ) #this command was the first test and filters out the required modules but has no speed advantage vs the used command #pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'|awk \"/${retroscraper_remote_modules[0]}/ || /${retroscraper_remote_modules[1]}/ || /${retroscraper_remote_modules[2]}/ || /${retroscraper_remote_modules[3]}/ || /${retroscraper_remote_modules[4]}/ || /${retroscraper_remote_modules[5]}/\"") pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'") for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do [[ $pip_list_output != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module" done }
-
Hi @kiro ,
I think I found an issue when scraping msx stuff.
For some games a number was detected as name and :113 is added after the zip file.
These "name" numbers / *.zip:113 aren't correctly displayed also in emulationstation.Here is part of the retropie log :
System: msx2 | Game : Game World - 126 Games System: msx2 | Game : Gun Fright System: msx2 | Game : 95372 Gw126.zip:113 System: msx2 | Game : 95372 Gw126.zip:113 System: msx2 | Game : 95372 Gw126.zip:113 System: msx2 | Game : Hades System: msx2 | Game : Desperado System: msx2 | Game : Hafanuda Koi Koi - Gostop Godori System: msx2 | Game : Miracle Warriors - Seal Of The Dark Lord System: msx2 | Game : Gyrodine System: msx2 | Game : Harapeko Pakkun System: msx2 | Game : Hang-on System: msx2 | Game : Hb-3600 Controller + Hb-6000 5.25" Dsdd Floppy Drive System: msx2 | Game : Hanafuta System: msx2 | Game : Hbd-50 System: msx2 | Game : 95524 Hboxing.rom:113 System: msx2 | Game : Hbd-f1 (floppy Controller + 3.5" Dsdd Floppy Drive) System: msx2 | Game : Head & Tail System: msx2 | Game : Hbd-20w (floppy Controller + 3.5" Dsdd Floppy Drive) System: msx2 | Game : Heavy Boxing System: msx2 | Game : Heist, The System: msx2 | Game : 95372 Gw126.zip:113 System: msx2 | Game : hfox2.zip
Can you have a look ?
-
Hi @Folly
! Sorry I didn't get your meaning. Does the number also show in the gamelists? thanks!Forget it, it seems to happen with just two games. Should get fixed in the nightly refresh. Sorry for that!
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.