[SOFT] New Scraper in the works

kiro

@Folly Hi Again! I'm trying the subfolders, and it is somehow working, except that I have a starnge behaviour from emulationstation:

This is a snippet of the created gamelist:

        <game>
                <rating></rating>
                <name>3d Asteroids</name>
                <marquee>./marquees/3D Asteroids (1987) (Atari) (Prototype) [a1].png</marquee>
                <image>./images/3D Asteroids (1987) (Atari) (Prototype) [a1].png</image>
                <publisher></publisher>
                <releasedate></releasedate>
                <players></players>
                <video>./videos/3D Asteroids (1987) (Atari) (Prototype) [a1].mp4</video>
                <genre></genre>
                <path>./myroms/3D Asteroids (1987) (Atari) (Prototype) [a1].zip</path>
                <developer></developer>
                <thumbnail/>
                <desc>This is an unencrypted pre-release version of the game that became Asteroids for the 7800.</desc>
                <playcount>0</playcount>
                <lastplayed>0</lastplayed>
        </game>

So roms in subfolders are detected and integrated into the gamelist, even in relative mode.

BUT, see emulationstation:

and when I enter the 'myroms' subfolder:

Is this the expected behavior from emulationstation?

(same behaviour when using absolute paths)

Anyway, just added the --recursive flag that will do exactly that, recurse over subfolders in the systems path. Let me know when you have a chance to test it. I do not know how to remove those subpaths in emualtionstation menu though.

Folly

@kiro

Nice ;-)

Yes, it works with and without the option --relativepaths.
Though I am missing that the script will not display what's being recognised.

The subfolder behaviour is expected.
Subfolders are shown when they contain roms.
Dependant on the theme they are mostly showing up as gray text in emulationstation.
Obviously such folder has to be there otherwise you can not select a rom from a subfolder.
As far as I know it's possible to add media to a folder just like it's done with files but for the folder no media has been added in the gamelist.xml now.

kiro

@Folly said in [SOFT] New Scraper in the works:

Though I am missing that the script will not display what's being recognised.

What do you mean by this?

Folly

@kiro

As test 1 placed 1 rom in the roms folder en 1 in a subfolder called sub :

pi@Bullseye-VM:~/RetroPie/roms/playchoice10$ find *
pc_1942.zip
sub
sub/pc_bball.zip

I would expect that the name of 1942 and bball are shown as usual.
Though they are missing in the output of the scan :

pi@Bullseye-VM:~/retroscraper-rpie$ python3 retroscraper.py --systems playchoice10 --recursive --relativepaths
Starting retroscraper - be Patient :-)
Starting retroscraper - be Patient :-)
Loading RetroScraper config File
Loading systems from Backend
Scanning Systems ['playchoice10']
Loading companies from backend
Starting scraping
Scanning Files
SCANNING DIRECTORY
SCRAPPING ENDED --- Thank you for using retroscraper!!

This is shown with a normal scan with both in roms directory :

Game : 1942 (playchoice-10)
Game : Baseball (playchoice-10)

kiro

Hmmmm strange, le me try that.. cannot replicate that... is the rom you copied actually with an accepted extension (I would assume yes, but just trying to understand)

I did some tests:

/home/pi/retroscraper-rpie/retroscraper.py --systems ags --bezels --sysbezels --relative --recursive 
Starting retroscraper - be Patient :-)
Starting retroscraper - be Patient :-)
Loading RetroScraper config File
Loading systems from Backend
Scanning Systems ['ags']
Loading companies from backend
Starting scraping
Scanning Files
SCANNING DIRECTORY
Game : test.zip
WRITING GAMELIST
SCRAPPING ENDED --- Thank you for using retroscraper

Folly

@kiro said in [SOFT] New Scraper in the works:

--relative

Can I now use --relative instead of --relativepaths ?

--recursive

Removing this option and detected roms in the roms folder are shown again.

Very strange that you don't experience this behaviour.

kiro

@Folly :-) I should've put --relativepaths, good sight !

Anyway , I tried one rom in system path, one rom in subfolder and one rom in both, and it seems to work?

I'll see if the git version is the latest one...it is... hmmm can you try --debug and see if something shows up?

Folly

@kiro

I updated the script on my pi4 and it worked correctly showing the games in the script output when using a subfolder.

I did the same on my VM.
The first time it did not display the games when using a subfolder.
The next tries were always successful so I can't reproduce the error any more so I can't use --debug anymore.
I had a generated file from my script 0 rom-index 0 in the roms folder, this could have had some influence, but moving it back and it still worked fine so I doubt it.

Let's say it's solved for now.
If it comes back again we can have a second look.

Btw. the pc_mario.mp4 works as expected on the pi4 ;-) (video and sound OK)

Folly

@kiro

Hi,

Here you can read about using folders in gamelists using descriptions and media :
https://github.com/RetroPie/EmulationStation/blob/master/GAMELISTS.md

I did a quick test and it's not only restricted to :

name - string, the displayed name for the folder.
desc - string, the description for the folder.
image - image_path, the path to an image to display for the folder.
thumbnail - image_path, the path to a smaller image to display for the folder.

Quick example :

<folder>
		<rating></rating>
		<name>subfolder</name>
		<marquee>./marquees/pc_1942.png</marquee>
		<image>./images/pc_1942.png</image>
		<video>./videos/pc_mario.mp4</video>
		<path>./more</path>
		<thumbnail/>
		<desc>description</desc>
</folder>

Looks like this :

kiro

@Folly thanks for that, although I'm not sure what would be the use for that in an automatic scraping... I mean, the folder tag could be created automatically, but what about description and image and the rest? Fill this info based on what exactly? Something to think about...

Folly

@kiro

I agree, it's not something to implement directly into the script.
I only mention it because you questioned why a folder looked the way it did.
So it's only for the knowledge.

Folly

@kiro

Just want to say :
I really appreciate the effort you putting in it.
Keep up the good work ;-)

kiro

@Folly Thanks, this is just my way to give back something to the retro community for all the fun over the years! even if it is just a silly scraper :-)

kiro

@Folly just wanted to give you a quick update, I'm still in the process of converting problematic videos, not easy with over 40k to go over (even being a script), will keep you posted. Take care!

Folly

@kiro

That's quite a lot.
How long does it take, do you think.
Just curious.

kiro

@Folly, well it depends on the number of videos that actually need conversion (impossible to estimate), but it also depends on me doing the script right the first time and not having to launch it over and over, the internet going off just in the middle of the process which means I need to restart all over, you know, the usual stuff... if everything goes ok it should be fine by this evening or tomorrow morning the latest. Luckily once the conversion is done, it should be fine. Funny thing: most of the videos not working seem to have been captured with mame, go figure :-) And, this means other databases will probably have the same issue.

Folly

@kiro

Nice to know.
I will see the result in time ;-)
Good luck !

kiro

@Folly Oh, BTW, I started playing around with a script for retropie-setup, it is in the git repository and it is a copy of another scraper's script. It works as far as the installation and running of the scraper goes. If you want to play with it, put it in ~/RetroPie-Setup/scriptmodules/supplementary and it should appear under the optional modules of retropie-setup. Just added the system selection option to rsetup ;-) ... videos still going ...

kiro

@Folly Hi, good news, the script for the videos is done, all videos should be working (at least those with a similar problem as pc-mario).

Folly

@kiro

Great !
Will test it this evening on my pi4.

I already see the improvement on my VM.
There are less errors reported after closing emulationstation.
Superb ;-)