[SOFT] New Scraper in the works

Folly

@kiro said in [SOFT] New Scraper in the works:

@Folly Thank you so much for all the testing.

I've uploaded new fixes into git (for both retroscraper versions) that should solve the missing games in the gamelists plus the bezels configuration issue.

Great !
I did some tests.

python3 retroscraper.py --systems atari7800 --cleanmedia

👍 - seems to be consistent now in detected games
👍 - seems to add all the detected games now in gamelist.xml
👍 - no game-specific configs are generated anymore

python3 retroscraper.py --systems atari7800 --bezel --cleanmedia

👍 - seems also to be consistent now in detected games
👍 - seems also to add all the detected games now in gamelist.xml
👍 - game-specific configs are generated as it should
👍 - bezel folder is created and bezel files are downloaded

I suspected that the bezel files would be in /home/pi/RetroPie/roms/atari7800/bezels but the were in /home/pi/RetroPie/overlays/atari7800/bezels.
Is it an idea to use this folder instead for the future ? :
/home/pi/RetroPie/roms/atari7800/media/retroarch/overlays/bezels
Using the same structure as I proposed for the emulationstation media :
/home/pi/RetroPie/roms/atari7800/media/emulationstation/<folders>
Then you can backup all in one time and with your option --cleanmedia you only have to delete 1 folder (media) which would make it much simpler.
Good idea ?

I'm struggling to reproduce the .png issue (I tried exactly with atari7800) and nothing. Just to clear out possible issues, do you have some sort of proxy at your location/machine that could be caching the API requests?

No I haven't.

I'll keep investigating in the meantime, and will start working on the relative path thing today.

I think that somehow string variables aren't updated correctly.
Though I have too little knowledge of your script.

Take care!

Same !

kiro

@Folly I've updated a new version of the 'light' retroscraper with a new flag '--debug' which will create a more complete log, maybe can you run it on the system tat still creates the png so we might understand what the problem is??

I've scanned my whole romset and no issues thousands of roms to be honest)

Thx!

Folly

@kiro

OK, I will do.

Found the log, but it's too big for pastebin.
Probably paste it on the google-drive.

kiro

@Folly thanks!

Folly

@kiro

~~retroscraper.log removed from google-drive.~~

Looks like the format variable is not updated correctly.

kiro

@Folly Thanks, will have a look right now.

On the other hand, I've updated both versions to cater for systems with 'non-standard names' such as 'shooters' :-) The only drawback is that the scraper will not be able to do a name search, due to the fact that it does not understand which is the actual system, but if checksums are in the DB they'll show up properly.

Folly

@kiro

Cool will give it a go.

Yea, it works really nice !

Seems a bit slower.
Does it have to search longer in the database with an unknown system ?

Folly

@kiro

I am getting this error, nothing works anymore :

pi@raspberrypi:~/retroscraper-rpie $ python3 retroscraper.py --systems arcade
Loading RetroScraper config File
Traceback (most recent call last):
  File "/home/pi/retroscraper-rpie/retroscraper.py", line 150, in <module>
    trans = complete['en']
TypeError: list indices must be integers or slices, not str

kiro

@Folly Hi yes, I'm correcting the issue with the PNG, which means I had to stop the backend for a while and resync DBs (that was the issue hopefully) should be up and running anytime soon. Need to make that error more clear :-) Thanks!

Folly

@kiro

Aha, that's good ;-)
Pfff, I was a bit afraid everything was broken.

Perfect, now showing this :

pi@raspberrypi:~/retroscraper-rpie $ python3 retroscraper.py --systems megaplay
Loading RetroScraper config File
CANNOT CONNECT TO THE BACKEND, PLEASE TRY AGAIN LATER

kiro

@Folly I've uploaded new versions with the friendly error message... DB is in sync again, png issue should be solved hopefully :-) now the system is up and running again, let me know how it goes when you can test. Thanks!

I'll be having a look at relative paths this week. I was thinking of adding a --realtivepaths flag and a --mediadir flag, first will add the './' instead of the full path, and second would add the possibility to choose the media down dir (starting by './' would make it relative to the current system path). Was thinking also of a '--singlemediadir' to put everything under the same folder, which would work only together with the '--mediadir' flag. What do you think?

Folly

@kiro

Really cool if you can add all that.
Seems to be what I am looking for.

Will do a clone now and report.
Looks very good !!!
👍 - videos are now correctly named *.mp4
👍 - now also similar speed between known systems and unknown systems

Was the *.png issue in the database or back-end ?

kiro

@Folly it was a backend issue... in order to gain speed the backend is just a collection of files created from the DB (so no actual engine running at the server)... the CRC's are symbolic links to the actual games objects, and for some reason, these links were not updated properly, so yes, I had to recreate 2.4 million links :-) luckily not manually!

Folly

@kiro

I must say it' now very much improved.
I tested the newly scraped systems in emulationstattion and off-course they work.
However, still quite some videos only play sound.
An example is mamerom pc_mario from playchoice10.

kiro

@Folly will have a look at those videos..thanks!

kiro

@Folly ... This is the 'frontend' to the backend:

http://77.68.23.83/

It may work perfectly as it is more of an experiment for me.

There you may want to check if the videos you see (they are the same as the one grabbed by the scraper) are working as the ones downloaded.

Thx

Folly

@kiro
Yes, they are the same :

pi@raspberrypi:~/Downloads $ sha1sum download.mp4 
5370f9a66196ae8a3ab82fe81347617c45a82f8a  download.mp4
pi@raspberrypi:~/Downloads $ cd ..
pi@raspberrypi:~ $ cd RetroPie/roms/playchoice10/videos
pi@raspberrypi:~/RetroPie/roms/playchoice10/videos $ sha1sum pc_mario.mp4 
5370f9a66196ae8a3ab82fe81347617c45a82f8a  pc_mario.mp4

The video gives also black screen in vlc.
I think it could be somehow a codec problem.
Perhaps I am missing a codec dependency .

kiro

@Folly I just tried that video and it works for me ...

Could you try it on another pc maybe?

Folly

@kiro

It played from within your 'frontend' on my pi.
Will try a VM to see what it does.

Not home now, but I will run it from the CLI on the pi when I can.
Just to see if it gives some useful info.

Folly

@kiro

Just tried on my VM Bulleseye X86_64 but I get this message now :

$ python3 retroscraper.py --systems playchoice10
Loading RetroScraper config File
Loading systems from Backend
Scanning Systems ['playchoice10']
Loading companies from backend
Starting scraping
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pi/retroscraper-rpie/scrapfunctions.py", line 967, in scanSystems
    if (system['name'].lower() not in selectedSystems) and (selectedSystems!=[]) and not doallsystems:
AttributeError: 'NoneType' object has no attribute 'lower'
SCRAPPING ENDED --- Thank you for using retroscraper!!

Could be that I am getting blocked or having a bad internet connection.
Will do an update and try again later.