[SOFT] New Scraper in the works

kiro

@Folly Hmm I could... but I'm not sure it will have an impact, namely because I have no system 'arcadia' in the db :-) I could create new systems which parent as 'arcade' and then it should be solved... I guess this need some thinking...

Correction, arcadia belongs to the system 'Emerson Radio Arcadia 2001' not entirely sure if this is correct or not...

Folly

@kiro

Mmmm, Yes difficult.
Indeed, my fault, arcadia is not "non-arcade" but emerson arcadia 2001.
Indeed, it needs some thinking.

This is my "database" with all mame drivers (mame(arcade)/mess(non-arcade) :
https://github.com/FollyMaddy/RetroPie-Share/blob/main/00-databases-00/mame/mame0249_systems_sorted_info
Could be something to look up things.

I tried now at home putting all in arcade roms directory.

python3 /opt/retropie/supplementary/retroscraper/retroscraper.py --mediadir media/emulationstation --systems arcade
Starting scraping
Scanning Files
Doing System arcade
Scanning Directory
Found 3 roms for the system
This is an arcade system
--it waits a long time here--
System: arcade | Game : rzindy500.zip
System: arcade | Game : rzbatfor.zip
System: arcade | Game : rztoshden.zip
Writing Gamelist
Scan Done!
SCRAPPING ENDED --- Thank you for using retroscraper!!

Sadly still no media and info in gamelist.

kiro

@Folly There's a nightly batch that will run and create the linking this evening (at midnight)

I see that retroscraper has detected the rom though (i see SHA for rzindy as 22ADC5B5AE9A26FA11697984293BBEEFF3DE69DD67), strange that the scraper has not added anything to the gamelist...

Let's see tomorrow how it goes, and if the .zip files are embedded into the same game as the 7z files..

Folly

@kiro

Good idea.
Curious if that will fix it.

Just did a small test with galaga only in arcade and it works perfectly.
So overall there seems to be no issue with other things.

Have a good evening.

kiro

@Folly cool! There are plus 800k roms in the DB and still a lot missing probably, that's why the more the scraper is used the better it will become, hopefully!

Folly

@kiro

Just tried tigerrz on multiple computers, sadly no recognition at all.
Before you added the media the game names were detected so then there was recognition to some degree, only without media.

I also tried an all scrape from your module-script.
9 retropie files are recognised faulty as "World Champion Cards"
system : retropie | game : World Champion Cards
Seems to be an issue.

Should I make a debug log for you ?

kiro

@Folly Hi, the auto script didn't insert the roms as expected, I'm looking at it right now (I saw this first thing after waking up :-)) Should be ok now --edit-- the autoscript will look for at least 3 scrappings of the same rom by different people before actually inserting it into the DB, to avoid false positives (someone putting a wrong file in their rom directories). In this case, as I know it was you, I've forced the insertion.

Now, there is an issue with very small files, where the checksums give always strange behavior, in principle when there was a check that the system existed, it would skip retropie, since now it has been disabled to accommodate for any type of system names, it scrapes 'retropie too', I'll adapt the script so it does not, at the same time, you can remove the gamelist.xml created in that folder. Updated version in Git.

Folly

@kiro

Updated, and its working ;-)
Great.

Is it possible to add the release date info and genre too in the future ?

kiro

@Folly great! If I have the info it should be present, although I'm. Although I'm not taking it from your gamelists for the moment it should be easy.

DTEAM

@kiro @Folly

Sometime, site like screenscraper recognize roms from "No Intro " romset but not from the MAME romset as ColecoVision or Videopac. Are you working on a solution for that?

kiro

@DTEAM retrosctaper recognizes the ROMs based on a checksum, not in names which in my opinion is a nonsense. So even if the names are whatever, your rom, as long as the checksum is in the DB, should be recognized.

Folly

@kiro

I did try to scrape all_in1 but the images are "empty".
Or do we have to wait until others try it too ?
Any news about the videos ?

kiro

@Folly hey sorry, I'm away from home at the moment, I'll be back on Wednesday and check by then! Sorry!

Folly

@kiro

Aha, no problem.
Have good days, see you later ;-)

kiro

@Folly I've just tried again the rztiger and it does work for me, the only difference is that I've got the .7z roms, can you point me (to my private email to avoid issues) to the .zip version of these roms??

and the gamelist:

<?xml version='1.0' encoding='utf-8'?><gameList>
	<game>
		<rating></rating>
		<name>R-Zone: Batman Forever</name>
		<marquee>/home/pi/RetroPie/roms/tigerrz/marquees/rzbatfor-marquee.png</marquee>
		<image>/home/pi/RetroPie/roms/tigerrz/images/rzbatfor-image.png</image>
		<publisher>Unknown</publisher>
		<releasedate></releasedate>
		<players></players>
		<video>/home/pi/RetroPie/roms/tigerrz/videos/rzbatfor-video.mp4</video>
		<genre></genre>
		<path>/home/pi/RetroPie/roms/tigerrz/rzbatfor.7z</path>
		<developer>Unknown</developer>
		<thumbnail/>
		<desc>Batman Forever (handheld)</desc>
		<playcount>0</playcount>
		<lastplayed>0</lastplayed>
	</game>
	<game>
		<rating></rating>
		<name>R-Zone: Indy 500</name>
		<marquee>/home/pi/RetroPie/roms/tigerrz/marquees/rzindy500-marquee.png</marquee>
		<image>/home/pi/RetroPie/roms/tigerrz/images/rzindy500-image.png</image>
		<publisher>Unknown</publisher>
		<releasedate></releasedate>
		<players></players>
		<video>/home/pi/RetroPie/roms/tigerrz/videos/rzindy500-video.mp4</video>
		<genre></genre>
		<path>/home/pi/RetroPie/roms/tigerrz/rzindy500.7z</path>
		<developer>Unknown</developer>
		<thumbnail/>
		<desc>Indy 500 (handheld)</desc>
		<playcount>0</playcount>
		<lastplayed>0</lastplayed>
	</game>
	<game>
		<rating></rating>
		<name>R-Zone: Battle Arena Toshinden</name>
		<marquee>/home/pi/RetroPie/roms/tigerrz/marquees/rztoshden-marquee.png</marquee>
		<image>/home/pi/RetroPie/roms/tigerrz/images/rztoshden-image.png</image>
		<publisher>Unknown</publisher>
		<releasedate></releasedate>
		<players></players>
		<video>/home/pi/RetroPie/roms/tigerrz/videos/rztoshden-video.mp4</video>
		<genre></genre>
		<path>/home/pi/RetroPie/roms/tigerrz/rztoshden.7z</path>
		<developer>Unknown</developer>
		<thumbnail/>
		<desc>Battle Arena (handheld)</desc>
		<playcount>0</playcount>
		<lastplayed>0</lastplayed>
	</game>
</gameList>

Folly

@kiro

Hi again.

I tested .7z and .zip files and both worked now.
I think the .7z files came from the mame-0.231-merged pack but this is not available anymore.

You've got mail

kiro

@Folly Cool, I'll test with the romset you mentioned.

kiro

I've uploaded the latest version, adding the --sort flag, it works like this:

--sort ORIGDIR DESTDIR

Where origdir is the place where all your roms are stored (they can all be mixed and in one single directoryt) and it will identify them and copy (not move) them over to your destdir/SYSTEM folder, where SYSTEM is the name of the system the roms belong to.

Have fun!

Folly

@kiro

I am experimenting with running retroscraper "online".

The advantage would be that the script is always up to date and that the basic script doesn't need to be stored anywhere.

I want to look if this is a good way of adding it to my script.
That way I only have to install the regular depends.

I already use a python script called gdrivedl in my module-script.
This is an example of running gdrivedl.py online using only the help option :

curl https://raw.githubusercontent.com/matthuisman/gdrivedl/master/gdrivedl.py | python3 - -h

So I wanted to see if I could do the same with your "retroscraper-rpie".
Basically, with the same help option and it actually works as expected :

curl https://raw.githubusercontent.com/zayamatias/retroscraper-rpie/main/retroscraper.py|python3 - -h

However your script uses more dependant scripts.
So when it comes to the actual scraping, it will miss these dependant scripts. (apicalls, checksums, scrapfunctions)

So I have a question :
Is there a way to make one script including these dependant scripts so it can be run online ?

kiro

@Folly said in [SOFT] New Scraper in the works:

curl https://raw.githubusercontent.com/zayamatias/retroscraper-rpie/main/retroscraper.py|python3 - -h

Hi Folly, I guess it would be possible to include these files in the main python file, but it will add complexity to the maintenance of the code (at least for me :-) ). Let me check if something can be done through the command line or any other alternative. will keep you posted.

EDIT: Just discovered the httpimport module which actually allows you to import dependencies from the web. I'm testing it right now. It may have an impact on the initialization of the script though.