[SOFT] New Scraper in the works

kiro

@Folly Well, here's the difficult question... how do we do that.. ok there are 2 things to consider:

1- I'll have to manually add the images to the DB, that is not really a concern as long as I understand to which game each image corresponds, which should be fairly easy (thanks to filenames).

2-What do we do if there's already a screenshot available for the game in question, for example, Archimedes. (although I see that most of your images correspond rather to 'systems' than games? And some images seem to be fairly similar (game&watch for instance)

So, if you want to make a small test, pack all the all_in_1 images, which I see are missing from my backend, into a file and I'll try to put them in the right place, once done, we can try to see if scraping gets these images. makes sense?

Folly

@kiro
I see your point.
Perhaps we can start with an easy one, tigerrz (Tiger R-Zone) which has just 3 games.
I scraped it earlier with your scraper and, as far as I know there are no marquees, images and videos in your DB.

kiro

@Folly Let's try it :-)

Folly

@kiro

Here is the googledrive of DTEAM with predefined gamelists, which is used by our project :
https://drive.google.com/drive/folders/1f_jXMG0XMBdyOOBpz8CHM6AFj9vC1R6m

You can just download the file tigerrz.

I just tested my files with your scraper and these have no media and no description and full name etc :
rzbatfor.zip rzindy500.zip rztoshden.zip

Don't know if .7z files have media, etc in your DB.

kiro

@Folly thx!

But, when I try to download (I guess right click then download on folder), I get

Download failed...do I need permissions or am I not downloading what I'm supposed to be downloading?

Folly

@kiro

I just tested, and it works here so it should work for you too.
You should just click on the tigerrz and a zip file is created and downloaded.

I recently had the same problem working on an open network, though I think you are not on an open network now, right.

kiro

@Folly Not sure, I tried an incognito window and it seems to work... maybe because since I'm logged in with my Gmail account to chrome it somehow messes things up? not sure...Anyway, It's a bit on the late side so I'll be importing the images tomorrow first thing and will let you know as soon as I'm done. I see there's a screenshot, a marquee, and a video (almost for every game) correct?

Folly

@kiro

Aha, so it's working now.
Yes indeed, all should be there.

Good evening !

kiro

Morning @Folly, I've been checking on the files, couple of questions:

All three games have 4 files in them, of these files, 2 are .mp4 (just .mp4 and .ia.mp4) when I try to play the ia.mp4 it does not show properly..should I consider the .mp4 one for the video?

Then, I have the logo, which I suppose should be the marque, and then the last one which looks like a cartridge, should this be the screenshot?

Have a great day!

Folly

@kiro

Good morning.

You can take the regular .mp4 ones, I see that the .ia.pm4 aren't used in the accompanied gamelist.
(As it should be, game seems to have no audio ~~seems 1 files has no audio, perhaps something to fix later~~)

The picture with the name should be the logo/marquee (*m.png).
Perhaps you can add the R-Zone logo sideways, just like with your other marquees, you can find logos in this theme :
https://github.com/DTEAM-1/cygnus-blue-flames

Indeed, the cartridge should be the screenshot/image.

Have a great day too !

kiro

@Folly I've updated the medias and they should be working (cannot really test without having the roms myself). I think I'll have to work on creating a script to extract the info from your gamelists and import it into the DB, shouldn't be very hard to do.

kiro

@Folly In the meantime I've also added the option to chose path from the retrropie_setup script. The funny thing is if you chose '.' as your media path, it will write in the same folder as the roms... (it covers a request made by someone in the forum)

Folly

@kiro said in [SOFT] New Scraper in the works:

@Folly I've updated the medias and they should be working (cannot really test without having the roms myself).

Tested, but it doesn't work.
No descriptions and no media or do I have to update the script ?

I think I'll have to work on creating a script to extract the info from your gamelists and import it into the DB, shouldn't be very hard to do.

Could be a good idea.

kiro

@Folly Hmm...

I've tested myself and it does work..

       <game>
                <rating></rating>
                <name>R-Zone: Batman Forever</name>
                <marquee>/home/pi/RetroPie/roms/arcade/marquees/rzbatfor-marquee.png</marquee>
                <image>/home/pi/RetroPie/roms/arcade/images/rzbatfor-image.png</image>
                <publisher>Unknown</publisher>
                <releasedate></releasedate>
                <players></players>
                <video>/home/pi/RetroPie/roms/arcade/videos/rzbatfor-video.mp4</video>
                <genre></genre>
                <path>/home/pi/RetroPie/roms/arcade/rzbatfor.7z</path>
                <developer>Unknown</developer>
                <thumbnail/>
                <desc>Batman Forever (handheld)</desc>
                <playcount>0</playcount>
                <lastplayed>0</lastplayed>
        </game>
        <game>
                <rating></rating>
                <name>R-Zone: Battle Arena Toshinden</name>
                <marquee>/home/pi/RetroPie/roms/arcade/marquees/rztoshden-marquee.png</marquee>
                <image>/home/pi/RetroPie/roms/arcade/images/rztoshden-image.png</image>
                <publisher>Unknown</publisher>
                <releasedate></releasedate>
                <players></players>
                <video>/home/pi/RetroPie/roms/arcade/videos/rztoshden-video.mp4</video>
                <genre></genre>
                <path>/home/pi/RetroPie/roms/arcade/rztoshden.7z</path>
                <developer>Unknown</developer>
                <thumbnail/>
                <desc>Battle Arena (handheld)</desc>
                <playcount>0</playcount>
                <lastplayed>0</lastplayed>
        </game>
        <game>
                <rating></rating>
                <name>R-Zone: Indy 500</name>
                <marquee>/home/pi/RetroPie/roms/arcade/marquees/rzindy500-marquee.png</marquee>
                <image>/home/pi/RetroPie/roms/arcade/images/rzindy500-image.png</image>
                <publisher>Unknown</publisher>
                <releasedate></releasedate>
                <players></players>
                <video>/home/pi/RetroPie/roms/arcade/videos/rzindy500-video.mp4</video>
                <genre></genre>
                <path>/home/pi/RetroPie/roms/arcade/rzindy500.7z</path>
                <developer>Unknown</developer>
                <thumbnail/>
                <desc>Indy 500 (handheld)</desc>
                <playcount>0</playcount>
                <lastplayed>0</lastplayed>
        </game>

So not sure what could be the issue???

Folly

@kiro

I have .zip files.
Did you add recognition for them too ?

kiro

@Folly Aha! that's the issue ;-)

The problem is not with the file extensions, but rather with the checksums, how the retroscraper works is that:

Will calculate SHA1,MD5 and CRC of the file and try to match it with the DB, if there's no match, then it will try to match the filename (without extension), BUT in order to do this, the location of the rom should be in one of the standard systems, which it does not happen probably if you use a setup like yours (name of the system in es_config.xml), so, it is not safe to assign a game to a filename for which the system is not sure to which system it actually belongs... phewww

There's a workaround I could use for this which is to link your system name to the 'arcade' system, and then it should recognize it...

Of course, those using other names will have the same issue if the checksums do not match.

Makes sense?

Quick one, the system names are the same as the folders in your google drive? I can adapt this for testing purposes

(just added 'tigerrz', if it is the same name as in es_systems it should recognize it now, hopefully, haven't tried it)

Folly

@kiro

Ok, I get it.

Indeed it's not in the arcade folder but in the tigerrz.
DTEAM began categorizing the mame roms so the are separated to have their own system name.
So this has been extended to the project too.

What if I post the sha1sums, could that help too ?

kiro

@Folly try again, if "tigerrz" is the system name in es_config, it should recognize the files, and capture the checksums automatically and add them to the DB

Folly

@kiro

Seems your backed isn't running.
CANNOT CONNECT TO THE BACKEND, PLEASE TRY AGAIN LATER

kiro

@Folly Hmm I'm actually connected, so it should be working?