[SOFT] New Scraper in the works

Folly

Good morning.

You can take the regular .mp4 ones, I see that the .ia.pm4 aren't used in the accompanied gamelist.
(As it should be, game seems to have no audio ~~seems 1 files has no audio, perhaps something to fix later~~)

The picture with the name should be the logo/marquee (*m.png).
Perhaps you can add the R-Zone logo sideways, just like with your other marquees, you can find logos in this theme :
https://github.com/DTEAM-1/cygnus-blue-flames

Indeed, the cartridge should be the screenshot/image.

Have a great day too !

kiro

@Folly I've updated the medias and they should be working (cannot really test without having the roms myself). I think I'll have to work on creating a script to extract the info from your gamelists and import it into the DB, shouldn't be very hard to do.

kiro

@Folly In the meantime I've also added the option to chose path from the retrropie_setup script. The funny thing is if you chose '.' as your media path, it will write in the same folder as the roms... (it covers a request made by someone in the forum)

Folly

@kiro said in [SOFT] New Scraper in the works:

@Folly I've updated the medias and they should be working (cannot really test without having the roms myself).

Tested, but it doesn't work.
No descriptions and no media or do I have to update the script ?

I think I'll have to work on creating a script to extract the info from your gamelists and import it into the DB, shouldn't be very hard to do.

Could be a good idea.

kiro

@Folly Hmm...

I've tested myself and it does work..

       <game>
                <rating></rating>
                <name>R-Zone: Batman Forever</name>
                <marquee>/home/pi/RetroPie/roms/arcade/marquees/rzbatfor-marquee.png</marquee>
                <image>/home/pi/RetroPie/roms/arcade/images/rzbatfor-image.png</image>
                <publisher>Unknown</publisher>
                <releasedate></releasedate>
                <players></players>
                <video>/home/pi/RetroPie/roms/arcade/videos/rzbatfor-video.mp4</video>
                <genre></genre>
                <path>/home/pi/RetroPie/roms/arcade/rzbatfor.7z</path>
                <developer>Unknown</developer>
                <thumbnail/>
                <desc>Batman Forever (handheld)</desc>
                <playcount>0</playcount>
                <lastplayed>0</lastplayed>
        </game>
        <game>
                <rating></rating>
                <name>R-Zone: Battle Arena Toshinden</name>
                <marquee>/home/pi/RetroPie/roms/arcade/marquees/rztoshden-marquee.png</marquee>
                <image>/home/pi/RetroPie/roms/arcade/images/rztoshden-image.png</image>
                <publisher>Unknown</publisher>
                <releasedate></releasedate>
                <players></players>
                <video>/home/pi/RetroPie/roms/arcade/videos/rztoshden-video.mp4</video>
                <genre></genre>
                <path>/home/pi/RetroPie/roms/arcade/rztoshden.7z</path>
                <developer>Unknown</developer>
                <thumbnail/>
                <desc>Battle Arena (handheld)</desc>
                <playcount>0</playcount>
                <lastplayed>0</lastplayed>
        </game>
        <game>
                <rating></rating>
                <name>R-Zone: Indy 500</name>
                <marquee>/home/pi/RetroPie/roms/arcade/marquees/rzindy500-marquee.png</marquee>
                <image>/home/pi/RetroPie/roms/arcade/images/rzindy500-image.png</image>
                <publisher>Unknown</publisher>
                <releasedate></releasedate>
                <players></players>
                <video>/home/pi/RetroPie/roms/arcade/videos/rzindy500-video.mp4</video>
                <genre></genre>
                <path>/home/pi/RetroPie/roms/arcade/rzindy500.7z</path>
                <developer>Unknown</developer>
                <thumbnail/>
                <desc>Indy 500 (handheld)</desc>
                <playcount>0</playcount>
                <lastplayed>0</lastplayed>
        </game>

So not sure what could be the issue???

Folly

@kiro

I have .zip files.
Did you add recognition for them too ?

kiro

@Folly Aha! that's the issue ;-)

The problem is not with the file extensions, but rather with the checksums, how the retroscraper works is that:

Will calculate SHA1,MD5 and CRC of the file and try to match it with the DB, if there's no match, then it will try to match the filename (without extension), BUT in order to do this, the location of the rom should be in one of the standard systems, which it does not happen probably if you use a setup like yours (name of the system in es_config.xml), so, it is not safe to assign a game to a filename for which the system is not sure to which system it actually belongs... phewww

There's a workaround I could use for this which is to link your system name to the 'arcade' system, and then it should recognize it...

Of course, those using other names will have the same issue if the checksums do not match.

Makes sense?

Quick one, the system names are the same as the folders in your google drive? I can adapt this for testing purposes

(just added 'tigerrz', if it is the same name as in es_systems it should recognize it now, hopefully, haven't tried it)

Folly

@kiro

Ok, I get it.

Indeed it's not in the arcade folder but in the tigerrz.
DTEAM began categorizing the mame roms so the are separated to have their own system name.
So this has been extended to the project too.

What if I post the sha1sums, could that help too ?

kiro

@Folly try again, if "tigerrz" is the system name in es_config, it should recognize the files, and capture the checksums automatically and add them to the DB

Folly

@kiro

Seems your backed isn't running.
CANNOT CONNECT TO THE BACKEND, PLEASE TRY AGAIN LATER

kiro

@Folly Hmm I'm actually connected, so it should be working?

Folly

@kiro

I am now on a fast-ferry back to home.
I think it is blocked somehow.

Tried to use a vpn, and it began but freezes.

I probably have to try it when I am back home again.

kiro

@Folly No worries, have a safe trip!

Folly

@kiro

Thanks !

Folly

@kiro

I had a good look at where tigerrz belongs to.
Turns out these are MESS drivers so theoretically the do not actually belong to the arcade section.
They should belong to the arcadia section.
I that something you can change ?

kiro

@Folly Hmm I could... but I'm not sure it will have an impact, namely because I have no system 'arcadia' in the db :-) I could create new systems which parent as 'arcade' and then it should be solved... I guess this need some thinking...

Correction, arcadia belongs to the system 'Emerson Radio Arcadia 2001' not entirely sure if this is correct or not...

Folly

@kiro

Mmmm, Yes difficult.
Indeed, my fault, arcadia is not "non-arcade" but emerson arcadia 2001.
Indeed, it needs some thinking.

This is my "database" with all mame drivers (mame(arcade)/mess(non-arcade) :
https://github.com/FollyMaddy/RetroPie-Share/blob/main/00-databases-00/mame/mame0249_systems_sorted_info
Could be something to look up things.

I tried now at home putting all in arcade roms directory.

python3 /opt/retropie/supplementary/retroscraper/retroscraper.py --mediadir media/emulationstation --systems arcade
Starting scraping
Scanning Files
Doing System arcade
Scanning Directory
Found 3 roms for the system
This is an arcade system
--it waits a long time here--
System: arcade | Game : rzindy500.zip
System: arcade | Game : rzbatfor.zip
System: arcade | Game : rztoshden.zip
Writing Gamelist
Scan Done!
SCRAPPING ENDED --- Thank you for using retroscraper!!

Sadly still no media and info in gamelist.

kiro

@Folly There's a nightly batch that will run and create the linking this evening (at midnight)

I see that retroscraper has detected the rom though (i see SHA for rzindy as 22ADC5B5AE9A26FA11697984293BBEEFF3DE69DD67), strange that the scraper has not added anything to the gamelist...

Let's see tomorrow how it goes, and if the .zip files are embedded into the same game as the 7z files..

Folly

@kiro

Good idea.
Curious if that will fix it.

Just did a small test with galaga only in arcade and it works perfectly.
So overall there seems to be no issue with other things.

Have a good evening.

kiro

@Folly cool! There are plus 800k roms in the DB and still a lot missing probably, that's why the more the scraper is used the better it will become, hopefully!