[SOFT] New Scraper in the works

Folly

Good idea.
Curious if that will fix it.

Just did a small test with galaga only in arcade and it works perfectly.
So overall there seems to be no issue with other things.

Have a good evening.

kiro

@Folly cool! There are plus 800k roms in the DB and still a lot missing probably, that's why the more the scraper is used the better it will become, hopefully!

Folly

@kiro

Just tried tigerrz on multiple computers, sadly no recognition at all.
Before you added the media the game names were detected so then there was recognition to some degree, only without media.

I also tried an all scrape from your module-script.
9 retropie files are recognised faulty as "World Champion Cards"
system : retropie | game : World Champion Cards
Seems to be an issue.

Should I make a debug log for you ?

kiro

@Folly Hi, the auto script didn't insert the roms as expected, I'm looking at it right now (I saw this first thing after waking up :-)) Should be ok now --edit-- the autoscript will look for at least 3 scrappings of the same rom by different people before actually inserting it into the DB, to avoid false positives (someone putting a wrong file in their rom directories). In this case, as I know it was you, I've forced the insertion.

Now, there is an issue with very small files, where the checksums give always strange behavior, in principle when there was a check that the system existed, it would skip retropie, since now it has been disabled to accommodate for any type of system names, it scrapes 'retropie too', I'll adapt the script so it does not, at the same time, you can remove the gamelist.xml created in that folder. Updated version in Git.

Folly

@kiro

Updated, and its working ;-)
Great.

Is it possible to add the release date info and genre too in the future ?

kiro

@Folly great! If I have the info it should be present, although I'm. Although I'm not taking it from your gamelists for the moment it should be easy.

DTEAM

@kiro @Folly

Sometime, site like screenscraper recognize roms from "No Intro " romset but not from the MAME romset as ColecoVision or Videopac. Are you working on a solution for that?

kiro

@DTEAM retrosctaper recognizes the ROMs based on a checksum, not in names which in my opinion is a nonsense. So even if the names are whatever, your rom, as long as the checksum is in the DB, should be recognized.

Folly

@kiro

I did try to scrape all_in1 but the images are "empty".
Or do we have to wait until others try it too ?
Any news about the videos ?

kiro

@Folly hey sorry, I'm away from home at the moment, I'll be back on Wednesday and check by then! Sorry!

Folly

@kiro

Aha, no problem.
Have good days, see you later ;-)

kiro

@Folly I've just tried again the rztiger and it does work for me, the only difference is that I've got the .7z roms, can you point me (to my private email to avoid issues) to the .zip version of these roms??

and the gamelist:

<?xml version='1.0' encoding='utf-8'?><gameList>
	<game>
		<rating></rating>
		<name>R-Zone: Batman Forever</name>
		<marquee>/home/pi/RetroPie/roms/tigerrz/marquees/rzbatfor-marquee.png</marquee>
		<image>/home/pi/RetroPie/roms/tigerrz/images/rzbatfor-image.png</image>
		<publisher>Unknown</publisher>
		<releasedate></releasedate>
		<players></players>
		<video>/home/pi/RetroPie/roms/tigerrz/videos/rzbatfor-video.mp4</video>
		<genre></genre>
		<path>/home/pi/RetroPie/roms/tigerrz/rzbatfor.7z</path>
		<developer>Unknown</developer>
		<thumbnail/>
		<desc>Batman Forever (handheld)</desc>
		<playcount>0</playcount>
		<lastplayed>0</lastplayed>
	</game>
	<game>
		<rating></rating>
		<name>R-Zone: Indy 500</name>
		<marquee>/home/pi/RetroPie/roms/tigerrz/marquees/rzindy500-marquee.png</marquee>
		<image>/home/pi/RetroPie/roms/tigerrz/images/rzindy500-image.png</image>
		<publisher>Unknown</publisher>
		<releasedate></releasedate>
		<players></players>
		<video>/home/pi/RetroPie/roms/tigerrz/videos/rzindy500-video.mp4</video>
		<genre></genre>
		<path>/home/pi/RetroPie/roms/tigerrz/rzindy500.7z</path>
		<developer>Unknown</developer>
		<thumbnail/>
		<desc>Indy 500 (handheld)</desc>
		<playcount>0</playcount>
		<lastplayed>0</lastplayed>
	</game>
	<game>
		<rating></rating>
		<name>R-Zone: Battle Arena Toshinden</name>
		<marquee>/home/pi/RetroPie/roms/tigerrz/marquees/rztoshden-marquee.png</marquee>
		<image>/home/pi/RetroPie/roms/tigerrz/images/rztoshden-image.png</image>
		<publisher>Unknown</publisher>
		<releasedate></releasedate>
		<players></players>
		<video>/home/pi/RetroPie/roms/tigerrz/videos/rztoshden-video.mp4</video>
		<genre></genre>
		<path>/home/pi/RetroPie/roms/tigerrz/rztoshden.7z</path>
		<developer>Unknown</developer>
		<thumbnail/>
		<desc>Battle Arena (handheld)</desc>
		<playcount>0</playcount>
		<lastplayed>0</lastplayed>
	</game>
</gameList>

Folly

@kiro

Hi again.

I tested .7z and .zip files and both worked now.
I think the .7z files came from the mame-0.231-merged pack but this is not available anymore.

You've got mail

kiro

@Folly Cool, I'll test with the romset you mentioned.

kiro

I've uploaded the latest version, adding the --sort flag, it works like this:

--sort ORIGDIR DESTDIR

Where origdir is the place where all your roms are stored (they can all be mixed and in one single directoryt) and it will identify them and copy (not move) them over to your destdir/SYSTEM folder, where SYSTEM is the name of the system the roms belong to.

Have fun!

Folly

@kiro

I am experimenting with running retroscraper "online".

The advantage would be that the script is always up to date and that the basic script doesn't need to be stored anywhere.

I want to look if this is a good way of adding it to my script.
That way I only have to install the regular depends.

I already use a python script called gdrivedl in my module-script.
This is an example of running gdrivedl.py online using only the help option :

curl https://raw.githubusercontent.com/matthuisman/gdrivedl/master/gdrivedl.py | python3 - -h

So I wanted to see if I could do the same with your "retroscraper-rpie".
Basically, with the same help option and it actually works as expected :

curl https://raw.githubusercontent.com/zayamatias/retroscraper-rpie/main/retroscraper.py|python3 - -h

However your script uses more dependant scripts.
So when it comes to the actual scraping, it will miss these dependant scripts. (apicalls, checksums, scrapfunctions)

So I have a question :
Is there a way to make one script including these dependant scripts so it can be run online ?

kiro

@Folly said in [SOFT] New Scraper in the works:

curl https://raw.githubusercontent.com/zayamatias/retroscraper-rpie/main/retroscraper.py|python3 - -h

Hi Folly, I guess it would be possible to include these files in the main python file, but it will add complexity to the maintenance of the code (at least for me :-) ). Let me check if something can be done through the command line or any other alternative. will keep you posted.

EDIT: Just discovered the httpimport module which actually allows you to import dependencies from the web. I'm testing it right now. It may have an impact on the initialization of the script though.

kiro

@Folly , some update. Try this:

curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3

It has 2 consequences:

1-There's a new dependency (httpimport module, to be installed with pip)
2-Startup is slower since it grabs everything form the repository directly.

Let me know if it works for you.

Folly

@kiro

Thanks !
I will try it when I can and let you know how it goes.

Folly

@Folly

Ok, I tested it out and it seems the idea could work.

But I have a few problems to overcome.

I have added the dependency installs in the depends function of my module script .
However I have to find a way to skip it properly so it will not try to when it's not needed.
Or I have to separate so it can be installed in a menu option.

Was hoping that it would run ok the first time.
It did work in the terminal
But from the script it didn't work.
I had to change the cmd, in my module script, and run it with su $user -c because the dependencies are installed under the normal user and therefor the script has also be run with the normal user.
So that one is solved.

To my surprise the waiting time of getting the script and it's dependencies remotely doesn't take that much time
Of-course you have to wait a little while but it's acceptable in my opinion and the scraping speed it just as normal.

Sadly I ran into an issue.
The crc and the sha1sum are saved in the name in the gamelist.xml.
This is an example :

<name>Boulder Dash,149604,0C0F7FF29CBE1CAA5A22AA0CF291BF5560DDD358</name>

Do you have an idea what it could be ?

Edit :
For archiving and looking up purposes I will add here what code I added.

Getting the retroscraper-remote depends in my module-script :

    local pip=$( su $user -c "python3 -c 'import pip'" 2>&1 )
    local succ="ModuleNotFoundError"
    echo "$pip"
    if [[ $pip == *"$succ"* ]]; then
        su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py"
        su $user -c "python3 /tmp/get-pip.py"
    fi
    su $user -c "python3 -m pip install --user --upgrade pip wheel setuptools"
    su $user -c "python3 -m pip install --user googletrans==4.0.0rc1 Pillow==9.2.0 requests==2.21.0 httpimport==1.1.0"

Test command from the terminal :

curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3 - --systems deco_cassette --relativepaths --mediadir media/emulationstation --nobackup

Test command from the module-script :

su $user -c "curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3 - --systems deco_cassette --relativepaths --mediadir media/emulationstation --nobackup"