[SOFT] New Scraper in the works
-
@Folly Fix is the repository. Let me know if this solves your problems or not.
Thanks one more time!
-
No change but I am doing the scraping from my script which uses the -rpie version.
So you probably have to change some in there too. -
@Folly I've updated the -rpie version, are you sure you have the latest version from the repository? Thanks
-
I see you have just made a commit.
I just tried again, same result.
My script runs the retroscraper-rpie version "online" so it should be the latest.Did the dependencies change ?
-
@Folly no, I did not change any dependencies...do you have somehow a cache enabled in your setup that would not get the latest version? Not sure...I'm scanning my whole collection again and it seems to be working fine....
-
@kiro said in [SOFT] New Scraper in the works:
do you have somehow a cache enabled in your setup that would not get the latest version?
No I haven't, I also did a re-run of the RetroPie-Setup script.
-
@Folly if you try to get retroscraper and run it locally, do you have the same problem?
-
I just did, with this command :
python3 retroscraper.py --systems arcade --relativepaths
That works, I have videos now.
My script has a different command, will do that test also locally.
I tested this command, with similar options with the one in my script :
python3 retroscraper.py --systems arcade --recursive --relativepaths --mediadir media/emulationstation --nobackup
That works locally too now.
-
@Folly cool, let me know how it goes.
-
Running this locally also works :
curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3 - --systems arcade --recursive --relativepaths --mediadir media/emulationstation --nobackup
From my script however it still has the same issue.
I will reboot my system to see if this will help. -
@Folly yes, I totally forgot about the retroscraper -remote version, just uploaded the fix there too
-
Yea, I see it too now.
I was thinking also that it was the -rpie version but it's the remote indeed.
So It didn't have the fix yet.I rebooted and now it works again, great.
But the reboot would not have been necessary, I believe now.Very strange indeed that it somehow didn't work anymore but it seems it has been fixed ;-)
Thanks for fixing.
Good team work ! -
@Folly there seems to be an issue with the database and some video naming, I fixed the video naming conventions in the front-end, but still need some work in the backend. Thanks again! And yes, great teamwork!
Enjoy your weekend!
-
-
Today I changed how the dependencies for retroscraper-remote are installed in my WIP mamedev.sh module-script as I wasn't happy about how it was.
It took me quite some time to figure things out but I am quite happy with the result as it looks like it's a bit faster.
This is the commit I made.
This is how the function looks now :
function retroscraper_remote_depends_mamedev () { #install pip if the pip module is not installed if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py" su $user -c "python3 /tmp/get-pip.py" fi #update pip if there is a new release [[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip" #install python modules when not detected as installed #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online" #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix local retroscraper_remote_module local retroscraper_remote_modules=() retroscraper_remote_modules=( wheel==0.40.0 setuptools==67.7.2 googletrans==4.0.0rc1 Pillow==9.2.0 requests==2.21.0 httpimport==1.1.0 ) for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do [[ $(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'") != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module" done }
I think you will find it interesting too.
What is your opinion about also using a fixed version for wheel and setuptools ?
If you have a question regarding this, then let me know.
Edit :
After testing it more it still seems that it's a bit slow on the raspberry pi.
Will have a second look if I can improve it. -
By changing the function slightly I doubled the speed.
See this commitThis is the function now :
function retroscraper_remote_depends_mamedev () { #install pip if the pip module is not installed if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py" su $user -c "python3 /tmp/get-pip.py" fi #update pip if there is a new release (2>&1 is used to redirect stderr to stdout so we can use the if function on the stderr info) [[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip" #install python modules when not detected as installed #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online" #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix local pip_list_output local retroscraper_remote_module local retroscraper_remote_modules=() retroscraper_remote_modules=( wheel==0.40.0 setuptools==67.7.2 googletrans==4.0.0rc1 Pillow==9.2.0 requests==2.21.0 httpimport==1.1.0 ) #this command was the first test and filters out the required modules but has no speed advantage vs the used command #pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'|awk \"/${retroscraper_remote_modules[0]}/ || /${retroscraper_remote_modules[1]}/ || /${retroscraper_remote_modules[2]}/ || /${retroscraper_remote_modules[3]}/ || /${retroscraper_remote_modules[4]}/ || /${retroscraper_remote_modules[5]}/\"") pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'") for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do [[ $pip_list_output != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module" done }
-
@Folly Hey thanks! Very useful indeed. I like the specific version switch, since sometimes newer versions of libraries have breaking changes.
-
@kiro said in [SOFT] New Scraper in the works:
I like the specific version switch, since sometimes newer versions of libraries have breaking changes
Sure, I think so too.
I think you know but with this command you can list the outdated versions :python3 -m pip list --outdated
So we can find the latest versions and update the versions one at a time in the function I made.
That way we can test if a newer module version breaks the script.
If not we can update the versions in the script.What do you think ?
-
I updated all modules to the latest version.
Only these 3 are changed : Pillow==9.5.0 requests==2.30.0 httpimport==1.3.0See this commit
This is the function now :
function retroscraper_remote_depends_mamedev () { #install pip if the pip module is not installed if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py" su $user -c "python3 /tmp/get-pip.py" fi #update pip if there is a new release (2>&1 is used to redirect stderr to stdout so we can use the if function on the stderr info) [[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip" #install python modules when not detected as installed #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online" #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix local pip_list_output local retroscraper_remote_module local retroscraper_remote_modules=() retroscraper_remote_modules=( wheel==0.40.0 setuptools==67.7.2 googletrans==4.0.0rc1 Pillow==9.5.0 requests==2.30.0 httpimport==1.3.0 ) #this command was the first test and filters out the required modules but has no speed advantage vs the used command #pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'|awk \"/${retroscraper_remote_modules[0]}/ || /${retroscraper_remote_modules[1]}/ || /${retroscraper_remote_modules[2]}/ || /${retroscraper_remote_modules[3]}/ || /${retroscraper_remote_modules[4]}/ || /${retroscraper_remote_modules[5]}/\"") pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'") for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do [[ $pip_list_output != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module" done }
-
Hi @kiro ,
I think I found an issue when scraping msx stuff.
For some games a number was detected as name and :113 is added after the zip file.
These "name" numbers / *.zip:113 aren't correctly displayed also in emulationstation.Here is part of the retropie log :
System: msx2 | Game : Game World - 126 Games System: msx2 | Game : Gun Fright System: msx2 | Game : 95372 Gw126.zip:113 System: msx2 | Game : 95372 Gw126.zip:113 System: msx2 | Game : 95372 Gw126.zip:113 System: msx2 | Game : Hades System: msx2 | Game : Desperado System: msx2 | Game : Hafanuda Koi Koi - Gostop Godori System: msx2 | Game : Miracle Warriors - Seal Of The Dark Lord System: msx2 | Game : Gyrodine System: msx2 | Game : Harapeko Pakkun System: msx2 | Game : Hang-on System: msx2 | Game : Hb-3600 Controller + Hb-6000 5.25" Dsdd Floppy Drive System: msx2 | Game : Hanafuta System: msx2 | Game : Hbd-50 System: msx2 | Game : 95524 Hboxing.rom:113 System: msx2 | Game : Hbd-f1 (floppy Controller + 3.5" Dsdd Floppy Drive) System: msx2 | Game : Head & Tail System: msx2 | Game : Hbd-20w (floppy Controller + 3.5" Dsdd Floppy Drive) System: msx2 | Game : Heavy Boxing System: msx2 | Game : Heist, The System: msx2 | Game : 95372 Gw126.zip:113 System: msx2 | Game : hfox2.zip
Can you have a look ?
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.