[SOFT] New Scraper in the works
-
@Folly Thanks, will have a look right now.
On the other hand, I've updated both versions to cater for systems with 'non-standard names' such as 'shooters' :-) The only drawback is that the scraper will not be able to do a name search, due to the fact that it does not understand which is the actual system, but if checksums are in the DB they'll show up properly.
-
Cool will give it a go.
Yea, it works really nice !
Seems a bit slower.
Does it have to search longer in the database with an unknown system ? -
I am getting this error, nothing works anymore :
pi@raspberrypi:~/retroscraper-rpie $ python3 retroscraper.py --systems arcade Loading RetroScraper config File Traceback (most recent call last): File "/home/pi/retroscraper-rpie/retroscraper.py", line 150, in <module> trans = complete['en'] TypeError: list indices must be integers or slices, not str
-
@Folly Hi yes, I'm correcting the issue with the PNG, which means I had to stop the backend for a while and resync DBs (that was the issue hopefully) should be up and running anytime soon. Need to make that error more clear :-) Thanks!
-
Aha, that's good ;-)
Pfff, I was a bit afraid everything was broken.Perfect, now showing this :
pi@raspberrypi:~/retroscraper-rpie $ python3 retroscraper.py --systems megaplay Loading RetroScraper config File CANNOT CONNECT TO THE BACKEND, PLEASE TRY AGAIN LATER
-
@Folly I've uploaded new versions with the friendly error message... DB is in sync again, png issue should be solved hopefully :-) now the system is up and running again, let me know how it goes when you can test. Thanks!
I'll be having a look at relative paths this week. I was thinking of adding a --realtivepaths flag and a --mediadir flag, first will add the './' instead of the full path, and second would add the possibility to choose the media down dir (starting by './' would make it relative to the current system path). Was thinking also of a '--singlemediadir' to put everything under the same folder, which would work only together with the '--mediadir' flag. What do you think?
-
Really cool if you can add all that.
Seems to be what I am looking for.Will do a clone now and report.
Looks very good !!!
š - videos are now correctly named *.mp4
š - now also similar speed between known systems and unknown systemsWas the *.png issue in the database or back-end ?
-
@Folly it was a backend issue... in order to gain speed the backend is just a collection of files created from the DB (so no actual engine running at the server)... the CRC's are symbolic links to the actual games objects, and for some reason, these links were not updated properly, so yes, I had to recreate 2.4 million links :-) luckily not manually!
-
I must say it' now very much improved.
I tested the newly scraped systems in emulationstattion and off-course they work.
However, still quite some videos only play sound.
An example is mamerom pc_mario from playchoice10. -
@Folly will have a look at those videos..thanks!
-
@Folly ... This is the 'frontend' to the backend:
It may work perfectly as it is more of an experiment for me.
There you may want to check if the videos you see (they are the same as the one grabbed by the scraper) are working as the ones downloaded.
Thx
-
@kiro
Yes, they are the same :pi@raspberrypi:~/Downloads $ sha1sum download.mp4 5370f9a66196ae8a3ab82fe81347617c45a82f8a download.mp4 pi@raspberrypi:~/Downloads $ cd .. pi@raspberrypi:~ $ cd RetroPie/roms/playchoice10/videos pi@raspberrypi:~/RetroPie/roms/playchoice10/videos $ sha1sum pc_mario.mp4 5370f9a66196ae8a3ab82fe81347617c45a82f8a pc_mario.mp4
The video gives also black screen in vlc.
I think it could be somehow a codec problem.
Perhaps I am missing a codec dependency . -
-
-
It played from within your 'frontend' on my pi.
Will try a VM to see what it does.Not home now, but I will run it from the CLI on the pi when I can.
Just to see if it gives some useful info. -
Just tried on my VM Bulleseye X86_64 but I get this message now :
$ python3 retroscraper.py --systems playchoice10 Loading RetroScraper config File Loading systems from Backend Scanning Systems ['playchoice10'] Loading companies from backend Starting scraping Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner self.run() File "/usr/lib/python3.9/threading.py", line 892, in run self._target(*self._args, **self._kwargs) File "/home/pi/retroscraper-rpie/scrapfunctions.py", line 967, in scanSystems if (system['name'].lower() not in selectedSystems) and (selectedSystems!=[]) and not doallsystems: AttributeError: 'NoneType' object has no attribute 'lower' SCRAPPING ENDED --- Thank you for using retroscraper!!
Could be that I am getting blocked or having a bad internet connection.
Will do an update and try again later. -
@Folly Hmm this seems like another issue.... can you use the --debug feature??
By the way I've updated the script in the repo as there was a bug with the debug function (It always went debug)
-
I am traveling now on a boot for my work and I am almost on my destination so have to work soon.
Will do it later today ;-)
-
@Folly Sure! no worries! I have the same issue :-)
Also, pls check if playchoice10 is the name of your system in es_systems:
<system> <name>playchoice10</name> ..... </system>
Just trying to discard issues.
Thx! -
It's not working on the pi4 so the issue seems to be pi related.
Though it's working in the VM and the video plays and the others seem to work too.
This is the VLC output of pc_mario.mp4 only playing sound on the PI4 :
vlc pc_mario.mp4 VLC media player 3.0.17.4 Vetinari (revision 3.0.13-8-g41878ff4f2) [016a9b98] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface. [a2102b80] gles2 gl: Initialized libplacebo v2.72.0 (API v72) [a210cb50] mmal_xsplitter vout display error: Failed to open Xsplitter:mmal_vout module [b38555b8] avcodec decoder: Using DRM Video Accel for hardware decoding [src/libmpg123/layer3.c:2184] error: part2_3_length (1536) too large for available bit count (1368) [b38555b8] main decoder error: buffer deadlock prevented QObject::~QObject: Timers cannot be stopped from another thread
This is the VLC output of moglf.mp4 playing both video and sound on the PI4 :
vlc pc_moglf.mp4 VLC media player 3.0.17.4 Vetinari (revision 3.0.13-8-g41878ff4f2) [0006cb98] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface. [a2306f88] gles2 gl: Initialized libplacebo v2.72.0 (API v72) [a2304e40] mmal_xsplitter vout display error: Failed to open Xsplitter:mmal_vout module [b3852340] avcodec decoder: Using DRM Video Accel for hardware decoding QObject::~QObject: Timers cannot be stopped from another thread
ffmpeg gives these video formats :
pc_mario.mp4 :Duration: 00:00:30.05, start: 0.000000, bitrate: 145 kb/s Stream #0:0(und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), yuv444p, 512x480 [SAR 5:4 DAR 4:3], 74 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
pc_moglf.mp4 :
Duration: 00:00:30.05, start: 0.000000, bitrate: 214 kb/s Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 143 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
-
@Folly Thanks a lot for the testing! Really appreciate it, so hopefully, the video issue is solved.
I started working on the relative paths, --relativepaths is added as an option and should put everything in the gamelist under a relative path. Let me know when you have tested (no urgency).
Tomorrow I'll add the --mediadir option ;-)
Thanks!
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.