[SOFT] New Scraper in the works
-
@Folly ... This is the 'frontend' to the backend:
It may work perfectly as it is more of an experiment for me.
There you may want to check if the videos you see (they are the same as the one grabbed by the scraper) are working as the ones downloaded.
Thx
-
@kiro
Yes, they are the same :pi@raspberrypi:~/Downloads $ sha1sum download.mp4 5370f9a66196ae8a3ab82fe81347617c45a82f8a download.mp4 pi@raspberrypi:~/Downloads $ cd .. pi@raspberrypi:~ $ cd RetroPie/roms/playchoice10/videos pi@raspberrypi:~/RetroPie/roms/playchoice10/videos $ sha1sum pc_mario.mp4 5370f9a66196ae8a3ab82fe81347617c45a82f8a pc_mario.mp4The video gives also black screen in vlc.
I think it could be somehow a codec problem.
Perhaps I am missing a codec dependency . -
F Folly referenced this topic on
-
-
It played from within your 'frontend' on my pi.
Will try a VM to see what it does.Not home now, but I will run it from the CLI on the pi when I can.
Just to see if it gives some useful info. -
Just tried on my VM Bulleseye X86_64 but I get this message now :
$ python3 retroscraper.py --systems playchoice10 Loading RetroScraper config File Loading systems from Backend Scanning Systems ['playchoice10'] Loading companies from backend Starting scraping Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner self.run() File "/usr/lib/python3.9/threading.py", line 892, in run self._target(*self._args, **self._kwargs) File "/home/pi/retroscraper-rpie/scrapfunctions.py", line 967, in scanSystems if (system['name'].lower() not in selectedSystems) and (selectedSystems!=[]) and not doallsystems: AttributeError: 'NoneType' object has no attribute 'lower' SCRAPPING ENDED --- Thank you for using retroscraper!!Could be that I am getting blocked or having a bad internet connection.
Will do an update and try again later. -
@Folly Hmm this seems like another issue.... can you use the --debug feature??
By the way I've updated the script in the repo as there was a bug with the debug function (It always went debug)
-
I am traveling now on a boot for my work and I am almost on my destination so have to work soon.
Will do it later today ;-)
-
@Folly Sure! no worries! I have the same issue :-)
Also, pls check if playchoice10 is the name of your system in es_systems:
<system> <name>playchoice10</name> ..... </system>Just trying to discard issues.
Thx! -
It's not working on the pi4 so the issue seems to be pi related.
Though it's working in the VM and the video plays and the others seem to work too.
This is the VLC output of pc_mario.mp4 only playing sound on the PI4 :
vlc pc_mario.mp4 VLC media player 3.0.17.4 Vetinari (revision 3.0.13-8-g41878ff4f2) [016a9b98] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface. [a2102b80] gles2 gl: Initialized libplacebo v2.72.0 (API v72) [a210cb50] mmal_xsplitter vout display error: Failed to open Xsplitter:mmal_vout module [b38555b8] avcodec decoder: Using DRM Video Accel for hardware decoding [src/libmpg123/layer3.c:2184] error: part2_3_length (1536) too large for available bit count (1368) [b38555b8] main decoder error: buffer deadlock prevented QObject::~QObject: Timers cannot be stopped from another threadThis is the VLC output of moglf.mp4 playing both video and sound on the PI4 :
vlc pc_moglf.mp4 VLC media player 3.0.17.4 Vetinari (revision 3.0.13-8-g41878ff4f2) [0006cb98] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface. [a2306f88] gles2 gl: Initialized libplacebo v2.72.0 (API v72) [a2304e40] mmal_xsplitter vout display error: Failed to open Xsplitter:mmal_vout module [b3852340] avcodec decoder: Using DRM Video Accel for hardware decoding QObject::~QObject: Timers cannot be stopped from another threadffmpeg gives these video formats :
pc_mario.mp4 :Duration: 00:00:30.05, start: 0.000000, bitrate: 145 kb/s Stream #0:0(und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), yuv444p, 512x480 [SAR 5:4 DAR 4:3], 74 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)pc_moglf.mp4 :
Duration: 00:00:30.05, start: 0.000000, bitrate: 214 kb/s Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 143 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default) -
@Folly Thanks a lot for the testing! Really appreciate it, so hopefully, the video issue is solved.
I started working on the relative paths, --relativepaths is added as an option and should put everything in the gamelist under a relative path. Let me know when you have tested (no urgency).
Tomorrow I'll add the --mediadir option ;-)
Thanks!
-
@kiro said in [SOFT] New Scraper in the works:
so hopefully, the video issue is solved.
Sadly no, not on the pi4.
Unless you changed something with the videos.What do you think, looking at the VLC logs ?
-
@Folly, I might have said the wrong way :-) The scraper is downloading the videos properly, but, some of the videos in the backend are encoded in a way the pi is not able to display. I'll dig into that today trying to understand what's the difference between a working and a non-working video so I can eventually re-encode those not working. Will keep you updated.
This is the encoding for the pc-mario video:
Stream #0:0(und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), yuv444p, 512x480 [SAR 5:4 DAR 4:3], 74 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
This is the encoding for a working video:
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 143 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
The only differences I see are the (high) and yuv as well as resolution.... any idea (any expert reading this) why the first video would not play on a pi?
I've tried converting from high 4:4:4 predictive to just high and it seems to be working:
I'll be creating a script soon to run over all videos and convert those under the same predictive profile.
Will keep you updated.
-
@kiro said in [SOFT] New Scraper in the works:
@Folly I just tried that video and it works for me ...
Could you try it on another pc maybe?
I saw that your computer had somewhat the same error, although it played well.
I will see what my VM says with the old pc_mario.mp4 and try the new one on my VM and PI4 when I am home again.
My VM gives the same errors, although it plays :
[src/libmpg123/layer3.c:2184] error: part2_3_length (1536) too large for available bit count (1368) [00007f15f4003510] gl gl: Initialized libplacebo v2.72.0 (API v72) [00007f15f47c4080] chain filter error: Too high level of recursion (3) [00007f15f47c7690] main filter error: Failed to create video converter .. many more chain/main errors [00007f15f40013b0] main vout display error: Failed to adapt decoder format to display [00007f15f40013b0] main vout display error: Failed to adapt decoder format to display [00007f15f00484b0] main video output error: video output creation failed [00007f1604c74da0] main decoder error: failed to create video outputAs you said it looks like the video has issues.
I renewed the video on my VM.
Succes, the errors are gone for pc_mario.mp4 ! -
Tested --relativepaths :
pi@Bullseye-VM:~/retroscraper-rpie$ python3 retroscraper.py --systems playchoice10 --relativepathsPerfect !
-
@Folly Nice! Thanks!
I'm going through all the videos with this 'Predictive' profile to fix them. Once this is done I'll start with the --mediadir flag.
Again, thanks a lot for all this testing.
-
Good idea.
Not tested yet on my pi4, but indeed, when all videos are converted they probably will work too.Wanted to ask.
Does the script have the ability to scan also subfolders ? -
As far as I know you removed the backup function for gamelist.xml.
If so you can remove it from the --help
--nobackup Do not backup gamelist.xml file -
@Folly Hi, the backup functionality was not removed, at least not intentionally :-) Is it not doing backups anymore?
Also, it will in principle scan subdirs, and exclude '/videos/', '/images/' and '/marquees/' from the scan.. although I have not tested it (I do not have subfolders in my rom collection honestly)
Although thinking about it, probably the --relativepath option will not work in this case due to how I extract the path from the filename... will have to rethink the relative logic...
-
By default the backup function doesn't work anymore, so you should look at this.
I did a test with placing the megaplay roms directory inside playchoice10.
Then the subfolder isn't scanned/detected.Will do a second test with a subfolder.
Placed some playchoice10 files in a subfolder and indeed they aren't detected. -
@Folly Will check on the default backup issue.... and will put the subfolders in my todo list :-)
backup should be ok now...(it was a matter of the logic being inverted)
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.