[SOFT] New Scraper in the works

Folly

I did try to scrape all_in1 but the images are "empty".
Or do we have to wait until others try it too ?
Any news about the videos ?

kiro

@Folly hey sorry, I'm away from home at the moment, I'll be back on Wednesday and check by then! Sorry!

Folly

@kiro

Aha, no problem.
Have good days, see you later ;-)

kiro

@Folly I've just tried again the rztiger and it does work for me, the only difference is that I've got the .7z roms, can you point me (to my private email to avoid issues) to the .zip version of these roms??

and the gamelist:

<?xml version='1.0' encoding='utf-8'?><gameList>
	<game>
		<rating></rating>
		<name>R-Zone: Batman Forever</name>
		<marquee>/home/pi/RetroPie/roms/tigerrz/marquees/rzbatfor-marquee.png</marquee>
		<image>/home/pi/RetroPie/roms/tigerrz/images/rzbatfor-image.png</image>
		<publisher>Unknown</publisher>
		<releasedate></releasedate>
		<players></players>
		<video>/home/pi/RetroPie/roms/tigerrz/videos/rzbatfor-video.mp4</video>
		<genre></genre>
		<path>/home/pi/RetroPie/roms/tigerrz/rzbatfor.7z</path>
		<developer>Unknown</developer>
		<thumbnail/>
		<desc>Batman Forever (handheld)</desc>
		<playcount>0</playcount>
		<lastplayed>0</lastplayed>
	</game>
	<game>
		<rating></rating>
		<name>R-Zone: Indy 500</name>
		<marquee>/home/pi/RetroPie/roms/tigerrz/marquees/rzindy500-marquee.png</marquee>
		<image>/home/pi/RetroPie/roms/tigerrz/images/rzindy500-image.png</image>
		<publisher>Unknown</publisher>
		<releasedate></releasedate>
		<players></players>
		<video>/home/pi/RetroPie/roms/tigerrz/videos/rzindy500-video.mp4</video>
		<genre></genre>
		<path>/home/pi/RetroPie/roms/tigerrz/rzindy500.7z</path>
		<developer>Unknown</developer>
		<thumbnail/>
		<desc>Indy 500 (handheld)</desc>
		<playcount>0</playcount>
		<lastplayed>0</lastplayed>
	</game>
	<game>
		<rating></rating>
		<name>R-Zone: Battle Arena Toshinden</name>
		<marquee>/home/pi/RetroPie/roms/tigerrz/marquees/rztoshden-marquee.png</marquee>
		<image>/home/pi/RetroPie/roms/tigerrz/images/rztoshden-image.png</image>
		<publisher>Unknown</publisher>
		<releasedate></releasedate>
		<players></players>
		<video>/home/pi/RetroPie/roms/tigerrz/videos/rztoshden-video.mp4</video>
		<genre></genre>
		<path>/home/pi/RetroPie/roms/tigerrz/rztoshden.7z</path>
		<developer>Unknown</developer>
		<thumbnail/>
		<desc>Battle Arena (handheld)</desc>
		<playcount>0</playcount>
		<lastplayed>0</lastplayed>
	</game>
</gameList>

Folly

@kiro

Hi again.

I tested .7z and .zip files and both worked now.
I think the .7z files came from the mame-0.231-merged pack but this is not available anymore.

You've got mail

kiro

@Folly Cool, I'll test with the romset you mentioned.

kiro

I've uploaded the latest version, adding the --sort flag, it works like this:

--sort ORIGDIR DESTDIR

Where origdir is the place where all your roms are stored (they can all be mixed and in one single directoryt) and it will identify them and copy (not move) them over to your destdir/SYSTEM folder, where SYSTEM is the name of the system the roms belong to.

Have fun!

Folly

@kiro

I am experimenting with running retroscraper "online".

The advantage would be that the script is always up to date and that the basic script doesn't need to be stored anywhere.

I want to look if this is a good way of adding it to my script.
That way I only have to install the regular depends.

I already use a python script called gdrivedl in my module-script.
This is an example of running gdrivedl.py online using only the help option :

curl https://raw.githubusercontent.com/matthuisman/gdrivedl/master/gdrivedl.py | python3 - -h

So I wanted to see if I could do the same with your "retroscraper-rpie".
Basically, with the same help option and it actually works as expected :

curl https://raw.githubusercontent.com/zayamatias/retroscraper-rpie/main/retroscraper.py|python3 - -h

However your script uses more dependant scripts.
So when it comes to the actual scraping, it will miss these dependant scripts. (apicalls, checksums, scrapfunctions)

So I have a question :
Is there a way to make one script including these dependant scripts so it can be run online ?

kiro

@Folly said in [SOFT] New Scraper in the works:

curl https://raw.githubusercontent.com/zayamatias/retroscraper-rpie/main/retroscraper.py|python3 - -h

Hi Folly, I guess it would be possible to include these files in the main python file, but it will add complexity to the maintenance of the code (at least for me :-) ). Let me check if something can be done through the command line or any other alternative. will keep you posted.

EDIT: Just discovered the httpimport module which actually allows you to import dependencies from the web. I'm testing it right now. It may have an impact on the initialization of the script though.

kiro

@Folly , some update. Try this:

curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3

It has 2 consequences:

1-There's a new dependency (httpimport module, to be installed with pip)
2-Startup is slower since it grabs everything form the repository directly.

Let me know if it works for you.

Folly

@kiro

Thanks !
I will try it when I can and let you know how it goes.

Folly

@Folly

Ok, I tested it out and it seems the idea could work.

But I have a few problems to overcome.

I have added the dependency installs in the depends function of my module script .
However I have to find a way to skip it properly so it will not try to when it's not needed.
Or I have to separate so it can be installed in a menu option.

Was hoping that it would run ok the first time.
It did work in the terminal
But from the script it didn't work.
I had to change the cmd, in my module script, and run it with su $user -c because the dependencies are installed under the normal user and therefor the script has also be run with the normal user.
So that one is solved.

To my surprise the waiting time of getting the script and it's dependencies remotely doesn't take that much time
Of-course you have to wait a little while but it's acceptable in my opinion and the scraping speed it just as normal.

Sadly I ran into an issue.
The crc and the sha1sum are saved in the name in the gamelist.xml.
This is an example :

<name>Boulder Dash,149604,0C0F7FF29CBE1CAA5A22AA0CF291BF5560DDD358</name>

Do you have an idea what it could be ?

Edit :
For archiving and looking up purposes I will add here what code I added.

Getting the retroscraper-remote depends in my module-script :

    local pip=$( su $user -c "python3 -c 'import pip'" 2>&1 )
    local succ="ModuleNotFoundError"
    echo "$pip"
    if [[ $pip == *"$succ"* ]]; then
        su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py"
        su $user -c "python3 /tmp/get-pip.py"
    fi
    su $user -c "python3 -m pip install --user --upgrade pip wheel setuptools"
    su $user -c "python3 -m pip install --user googletrans==4.0.0rc1 Pillow==9.2.0 requests==2.21.0 httpimport==1.1.0"

Test command from the terminal :

curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3 - --systems deco_cassette --relativepaths --mediadir media/emulationstation --nobackup

Test command from the module-script :

su $user -c "curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3 - --systems deco_cassette --relativepaths --mediadir media/emulationstation --nobackup"

kiro

@Folly Argh! Sorry, I uploaded a test version (it's been a while since I didn't touch the code, sorry) I've updated it! Should be fine now..

Folly

@kiro

Nice, so there is probably no issue.
Will do test again when I can.

I hope it's the correct route for me.
Nevertheless it's already quite interesting that we managed to run it this way.

kiro

@Folly indeed, I never thought about that possibility to be honest. Let me know how it goes!

Folly

@kiro

Nice, it's working ;-)

kiro

@Folly Cool... I'm still in the process of importing all mess roms...it's a pain in the .... anyway, as soon as that is done, I'll start adding the images from your playlists..

Folly

@kiro

A pain ?
I can imagine.
Just be patient there is no hurry.

Hopefully we can manage that the outputs are be becoming a good alternative for the predefined gamelists,
because the scraping speed is much higher than downloading speed of the predefined ones.
So all in all that would really be great.

kiro

Hi @Folly! Finally all roms from mame-sl are in the backend and I've added the pictures found in your gamelists where missing, phew... hope this helps!

Supported systems with figures here: http://77.68.23.83/

Folly

@kiro

Very cool that you managed all of this.
I can only imagine how much work it must be.
Thumbs up ^^ :-)

I will do some scraping when I can ;-)