[SOFT] New Scraper in the works
-
@Folly Hmm...
I've tested myself and it does work..
<game> <rating></rating> <name>R-Zone: Batman Forever</name> <marquee>/home/pi/RetroPie/roms/arcade/marquees/rzbatfor-marquee.png</marquee> <image>/home/pi/RetroPie/roms/arcade/images/rzbatfor-image.png</image> <publisher>Unknown</publisher> <releasedate></releasedate> <players></players> <video>/home/pi/RetroPie/roms/arcade/videos/rzbatfor-video.mp4</video> <genre></genre> <path>/home/pi/RetroPie/roms/arcade/rzbatfor.7z</path> <developer>Unknown</developer> <thumbnail/> <desc>Batman Forever (handheld)</desc> <playcount>0</playcount> <lastplayed>0</lastplayed> </game> <game> <rating></rating> <name>R-Zone: Battle Arena Toshinden</name> <marquee>/home/pi/RetroPie/roms/arcade/marquees/rztoshden-marquee.png</marquee> <image>/home/pi/RetroPie/roms/arcade/images/rztoshden-image.png</image> <publisher>Unknown</publisher> <releasedate></releasedate> <players></players> <video>/home/pi/RetroPie/roms/arcade/videos/rztoshden-video.mp4</video> <genre></genre> <path>/home/pi/RetroPie/roms/arcade/rztoshden.7z</path> <developer>Unknown</developer> <thumbnail/> <desc>Battle Arena (handheld)</desc> <playcount>0</playcount> <lastplayed>0</lastplayed> </game> <game> <rating></rating> <name>R-Zone: Indy 500</name> <marquee>/home/pi/RetroPie/roms/arcade/marquees/rzindy500-marquee.png</marquee> <image>/home/pi/RetroPie/roms/arcade/images/rzindy500-image.png</image> <publisher>Unknown</publisher> <releasedate></releasedate> <players></players> <video>/home/pi/RetroPie/roms/arcade/videos/rzindy500-video.mp4</video> <genre></genre> <path>/home/pi/RetroPie/roms/arcade/rzindy500.7z</path> <developer>Unknown</developer> <thumbnail/> <desc>Indy 500 (handheld)</desc> <playcount>0</playcount> <lastplayed>0</lastplayed> </game>So not sure what could be the issue???
-
I have .zip files.
Did you add recognition for them too ? -
@Folly Aha! that's the issue ;-)
The problem is not with the file extensions, but rather with the checksums, how the retroscraper works is that:
Will calculate SHA1,MD5 and CRC of the file and try to match it with the DB, if there's no match, then it will try to match the filename (without extension), BUT in order to do this, the location of the rom should be in one of the standard systems, which it does not happen probably if you use a setup like yours (name of the system in es_config.xml), so, it is not safe to assign a game to a filename for which the system is not sure to which system it actually belongs... phewww
There's a workaround I could use for this which is to link your system name to the 'arcade' system, and then it should recognize it...
Of course, those using other names will have the same issue if the checksums do not match.
Makes sense?
Quick one, the system names are the same as the folders in your google drive? I can adapt this for testing purposes
(just added 'tigerrz', if it is the same name as in es_systems it should recognize it now, hopefully, haven't tried it)
-
Ok, I get it.
Indeed it's not in the arcade folder but in the tigerrz.
DTEAM began categorizing the mame roms so the are separated to have their own system name.
So this has been extended to the project too.What if I post the sha1sums, could that help too ?
-
@Folly try again, if "tigerrz" is the system name in es_config, it should recognize the files, and capture the checksums automatically and add them to the DB
-
Seems your backed isn't running.
CANNOT CONNECT TO THE BACKEND, PLEASE TRY AGAIN LATER -
@Folly Hmm I'm actually connected, so it should be working?
-
I am now on a fast-ferry back to home.
I think it is blocked somehow.Tried to use a vpn, and it began but freezes.
I probably have to try it when I am back home again.
-
@Folly No worries, have a safe trip!
-
Thanks !
-
I had a good look at where tigerrz belongs to.
Turns out these are MESS drivers so theoretically the do not actually belong to the arcade section.
They should belong to the arcadia section.
I that something you can change ? -
@Folly Hmm I could... but I'm not sure it will have an impact, namely because I have no system 'arcadia' in the db :-) I could create new systems which parent as 'arcade' and then it should be solved... I guess this need some thinking...
Correction, arcadia belongs to the system 'Emerson Radio Arcadia 2001' not entirely sure if this is correct or not...
-
Mmmm, Yes difficult.
Indeed, my fault, arcadia is not "non-arcade" but emerson arcadia 2001.
Indeed, it needs some thinking.This is my "database" with all mame drivers (mame(arcade)/mess(non-arcade) :
https://github.com/FollyMaddy/RetroPie-Share/blob/main/00-databases-00/mame/mame0249_systems_sorted_info
Could be something to look up things.I tried now at home putting all in arcade roms directory.
python3 /opt/retropie/supplementary/retroscraper/retroscraper.py --mediadir media/emulationstation --systems arcade Starting scraping Scanning Files Doing System arcade Scanning Directory Found 3 roms for the system This is an arcade system --it waits a long time here-- System: arcade | Game : rzindy500.zip System: arcade | Game : rzbatfor.zip System: arcade | Game : rztoshden.zip Writing Gamelist Scan Done! SCRAPPING ENDED --- Thank you for using retroscraper!!Sadly still no media and info in gamelist.
-
@Folly There's a nightly batch that will run and create the linking this evening (at midnight)
I see that retroscraper has detected the rom though (i see SHA for rzindy as 22ADC5B5AE9A26FA11697984293BBEEFF3DE69DD67), strange that the scraper has not added anything to the gamelist...
Let's see tomorrow how it goes, and if the .zip files are embedded into the same game as the 7z files..
-
Good idea.
Curious if that will fix it.Just did a small test with galaga only in arcade and it works perfectly.
So overall there seems to be no issue with other things.Have a good evening.
-
@Folly cool! There are plus 800k roms in the DB and still a lot missing probably, that's why the more the scraper is used the better it will become, hopefully!
-
Just tried tigerrz on multiple computers, sadly no recognition at all.
Before you added the media the game names were detected so then there was recognition to some degree, only without media.I also tried an all scrape from your module-script.
9 retropie files are recognised faulty as "World Champion Cards"
system : retropie | game : World Champion Cards
Seems to be an issue.Should I make a debug log for you ?
-
@Folly Hi, the auto script didn't insert the roms as expected, I'm looking at it right now (I saw this first thing after waking up :-)) Should be ok now --edit-- the autoscript will look for at least 3 scrappings of the same rom by different people before actually inserting it into the DB, to avoid false positives (someone putting a wrong file in their rom directories). In this case, as I know it was you, I've forced the insertion.
Now, there is an issue with very small files, where the checksums give always strange behavior, in principle when there was a check that the system existed, it would skip retropie, since now it has been disabled to accommodate for any type of system names, it scrapes 'retropie too', I'll adapt the script so it does not, at the same time, you can remove the gamelist.xml created in that folder. Updated version in Git.
-
Updated, and its working ;-)
Great.Is it possible to add the release date info and genre too in the future ?
-
@Folly great! If I have the info it should be present, although I'm. Although I'm not taking it from your gamelists for the moment it should be easy.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.