[SOFT] New Scraper in the works

kiro

@Folly Thanks a lot for the testing! Really appreciate it, so hopefully, the video issue is solved.

I started working on the relative paths, --relativepaths is added as an option and should put everything in the gamelist under a relative path. Let me know when you have tested (no urgency).

Tomorrow I'll add the --mediadir option ;-)

Thanks!

Folly

@kiro said in [SOFT] New Scraper in the works:

so hopefully, the video issue is solved.

Sadly no, not on the pi4.
Unless you changed something with the videos.

What do you think, looking at the VLC logs ?

kiro

@Folly, I might have said the wrong way :-) The scraper is downloading the videos properly, but, some of the videos in the backend are encoded in a way the pi is not able to display. I'll dig into that today trying to understand what's the difference between a working and a non-working video so I can eventually re-encode those not working. Will keep you updated.

This is the encoding for the pc-mario video:

Stream #0:0(und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), yuv444p, 512x480 [SAR 5:4 DAR 4:3], 74 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)

This is the encoding for a working video:

Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 143 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)

The only differences I see are the (high) and yuv as well as resolution.... any idea (any expert reading this) why the first video would not play on a pi?

I've tried converting from high 4:4:4 predictive to just high and it seems to be working:

I'll be creating a script soon to run over all videos and convert those under the same predictive profile.

Will keep you updated.

Folly

@kiro said in [SOFT] New Scraper in the works:

@Folly I just tried that video and it works for me ...

Could you try it on another pc maybe?

I saw that your computer had somewhat the same error, although it played well.

I will see what my VM says with the old pc_mario.mp4 and try the new one on my VM and PI4 when I am home again.

My VM gives the same errors, although it plays :

[src/libmpg123/layer3.c:2184] error: part2_3_length (1536) too large for available bit count (1368)
[00007f15f4003510] gl gl: Initialized libplacebo v2.72.0 (API v72)
[00007f15f47c4080] chain filter error: Too high level of recursion (3)
[00007f15f47c7690] main filter error: Failed to create video converter
.. many more chain/main errors
[00007f15f40013b0] main vout display error: Failed to adapt decoder format to display
[00007f15f40013b0] main vout display error: Failed to adapt decoder format to display
[00007f15f00484b0] main video output error: video output creation failed
[00007f1604c74da0] main decoder error: failed to create video output

As you said it looks like the video has issues.

I renewed the video on my VM.
Succes, the errors are gone for pc_mario.mp4 !

Folly

@kiro

Tested --relativepaths :

pi@Bullseye-VM:~/retroscraper-rpie$ python3 retroscraper.py --systems playchoice10 --relativepaths

Perfect !

kiro

@Folly Nice! Thanks!

I'm going through all the videos with this 'Predictive' profile to fix them. Once this is done I'll start with the --mediadir flag.

Again, thanks a lot for all this testing.

Folly

@kiro

Good idea.
Not tested yet on my pi4, but indeed, when all videos are converted they probably will work too.

Wanted to ask.
Does the script have the ability to scan also subfolders ?

Folly

@kiro

As far as I know you removed the backup function for gamelist.xml.
If so you can remove it from the --help
~~--nobackup Do not backup gamelist.xml file~~

kiro

@Folly Hi, the backup functionality was not removed, at least not intentionally :-) Is it not doing backups anymore?

Also, it will in principle scan subdirs, and exclude '/videos/', '/images/' and '/marquees/' from the scan.. although I have not tested it (I do not have subfolders in my rom collection honestly)

Although thinking about it, probably the --relativepath option will not work in this case due to how I extract the path from the filename... will have to rethink the relative logic...

Folly

@kiro

By default the backup function doesn't work anymore, so you should look at this.

I did a test with placing the megaplay roms directory inside playchoice10.
Then the subfolder isn't scanned/detected.

Will do a second test with a subfolder.
Placed some playchoice10 files in a subfolder and indeed they aren't detected.

kiro

@Folly Will check on the default backup issue.... and will put the subfolders in my todo list :-)

backup should be ok now...(it was a matter of the logic being inverted)

Folly

@kiro

By default backing up gamelist.xml works again.
Tested the --nobackup function and that works too.

kiro

@Folly thanks! Yes the double negative logic just screwed with my brain...

Folly

@kiro

Haha, Yes I can imagine.

kiro

@Folly Hi Again! I'm trying the subfolders, and it is somehow working, except that I have a starnge behaviour from emulationstation:

This is a snippet of the created gamelist:

        <game>
                <rating></rating>
                <name>3d Asteroids</name>
                <marquee>./marquees/3D Asteroids (1987) (Atari) (Prototype) [a1].png</marquee>
                <image>./images/3D Asteroids (1987) (Atari) (Prototype) [a1].png</image>
                <publisher></publisher>
                <releasedate></releasedate>
                <players></players>
                <video>./videos/3D Asteroids (1987) (Atari) (Prototype) [a1].mp4</video>
                <genre></genre>
                <path>./myroms/3D Asteroids (1987) (Atari) (Prototype) [a1].zip</path>
                <developer></developer>
                <thumbnail/>
                <desc>This is an unencrypted pre-release version of the game that became Asteroids for the 7800.</desc>
                <playcount>0</playcount>
                <lastplayed>0</lastplayed>
        </game>

So roms in subfolders are detected and integrated into the gamelist, even in relative mode.

BUT, see emulationstation:

and when I enter the 'myroms' subfolder:

Is this the expected behavior from emulationstation?

(same behaviour when using absolute paths)

Anyway, just added the --recursive flag that will do exactly that, recurse over subfolders in the systems path. Let me know when you have a chance to test it. I do not know how to remove those subpaths in emualtionstation menu though.

Folly

@kiro

Nice ;-)

Yes, it works with and without the option --relativepaths.
Though I am missing that the script will not display what's being recognised.

The subfolder behaviour is expected.
Subfolders are shown when they contain roms.
Dependant on the theme they are mostly showing up as gray text in emulationstation.
Obviously such folder has to be there otherwise you can not select a rom from a subfolder.
As far as I know it's possible to add media to a folder just like it's done with files but for the folder no media has been added in the gamelist.xml now.

kiro

@Folly said in [SOFT] New Scraper in the works:

Though I am missing that the script will not display what's being recognised.

What do you mean by this?

Folly

@kiro

As test 1 placed 1 rom in the roms folder en 1 in a subfolder called sub :

pi@Bullseye-VM:~/RetroPie/roms/playchoice10$ find *
pc_1942.zip
sub
sub/pc_bball.zip

I would expect that the name of 1942 and bball are shown as usual.
Though they are missing in the output of the scan :

pi@Bullseye-VM:~/retroscraper-rpie$ python3 retroscraper.py --systems playchoice10 --recursive --relativepaths
Starting retroscraper - be Patient :-)
Starting retroscraper - be Patient :-)
Loading RetroScraper config File
Loading systems from Backend
Scanning Systems ['playchoice10']
Loading companies from backend
Starting scraping
Scanning Files
SCANNING DIRECTORY
SCRAPPING ENDED --- Thank you for using retroscraper!!

This is shown with a normal scan with both in roms directory :

Game : 1942 (playchoice-10)
Game : Baseball (playchoice-10)

kiro

Hmmmm strange, le me try that.. cannot replicate that... is the rom you copied actually with an accepted extension (I would assume yes, but just trying to understand)

I did some tests:

/home/pi/retroscraper-rpie/retroscraper.py --systems ags --bezels --sysbezels --relative --recursive 
Starting retroscraper - be Patient :-)
Starting retroscraper - be Patient :-)
Loading RetroScraper config File
Loading systems from Backend
Scanning Systems ['ags']
Loading companies from backend
Starting scraping
Scanning Files
SCANNING DIRECTORY
Game : test.zip
WRITING GAMELIST
SCRAPPING ENDED --- Thank you for using retroscraper

Folly

@kiro said in [SOFT] New Scraper in the works:

--relative

Can I now use --relative instead of --relativepaths ?

--recursive

Removing this option and detected roms in the roms folder are shown again.

Very strange that you don't experience this behaviour.