[SOFT] New Scraper in the works

kiro

@Folly hi, thanks for the testing, can you give me a game name for the 7800 for which videos are downloaded as PNG? I cannot reproduce the issue.

Just for clarity, the games that are not in the gamelisy, have the proper extension? (Recognized one)? I'm sure yes but better be safe 🙂

Folly

@kiro

Do you need some sha1sums ?
Here is one :

sha1sum 'Choplifter (1987) (Atari).a78'
6457dd40aeadb9733602c4ad3498780ebd7bf0d1  Choplifter (1987) (Atari).a78

Looks like these games are missing :

'Water Ski (1988) (Froggo).a78'
'Winter Games (1987) (Atari) [!].a78'
'Xenophobe (1989) (Atari) [!].a78'
'Xevious (1987) (Atari).a78'

This is the output of my videos folder for atari7800 :

'3D Asteroids (1987) (Atari) (Prototype) [a1].png'
'3D Asteroids (1987) (Atari) (Prototype).png'
'Ace of Aces (1988) (Atari).png'
'Alien Brigade (1990) (Atari) (CC2 Hack).png'
'Alien Brigade (1990) (Atari) [!].png'
'Asteroids (1987) (Atari).png'
'Ballblazer (1987) (Atari-Lucasfilm).png'
'Barnyard Blaster (1988) (Atari) [!].png'
'Basketbrawl (1990) (Atari).png'
'Beef Drop 7800 (Oct 30-Rev 08) (2004) (CC2 Hack).png'
'Centipede (1987) (Atari).png'
'Choplifter (1987) (Atari).png'
'Commando (1989) (Atari).png'
'Crack'\''ed (1988) (Atari).png'
'Crossbow (1988) (Atari) (CC2 Hack).png'
'Crossbow (1988) (Atari).png'
'Dark Chambers (1988) (Atari).png'
'Desert Falcon (1987) (Atari).mp4'
'Desert Falcon (2004) (Atari) (HSC Hack) [a1].mp4'
'Desert Falcon (2004) (Atari) (HSC Hack).mp4'
'Dig Dug (1987) (Atari).png'
'Donkey Kong (1988) (Atari).png'
'Donkey Kong Jr (1988) (Atari).png'
'Double Dragon (1989) (Activision) [!].mp4'
'F-18 Hornet (1988) (Absolute) [!].mp4'
'Fatal Run (1990) (Atari).png'
'Fight Night (1988) (Atari).png'
'Food Fight (1987) (Atari).png'
'Galaga (1987) (Atari).png'
'Gato (198x) (Atari) (Prototype) [a1].mp4'
'Gato (198x) (Atari) (Prototype) [a2].mp4'
'Gato (198x) (Atari) (Prototype).mp4'
'Hat Trick (1987) (Atari).png'
'Ikari Warriors (1989) (Atari).png'
'Impossible Mission (1987) (Atari) [a1].mp4'
'Impossible Mission (1987) (Atari) [f1].mp4'
'Impossible Mission (1987) (Atari) [!].mp4'
'Jinks (1989) (Atari).png'
'Joust (1987) (Atari).png'
'Karateka (1987) (Atari).png'
'Klax (1992) (Atari) (Prototype) [a1].mp4'
'Klax (1992) (Atari) (Prototype) [f1].mp4'
'Klax (1992) (Atari) (Prototype).mp4'
'Klax (2004) (Impossible Level 1 Hack).mp4'
'Klax (2004) (Impossible Level 2 Hack).mp4'
'Klax (2004) (Impossible Level 3 Hack).mp4'
'Kung Fu Master (1989) (Absolute) [!].png'
'Mario Brothers (1988) (Atari).png'
'Mat Mania Challenge (1989) (Atari).png'
'Mean 18 Golf (1988) (Atari) [!].png'
'Meltdown (1990) (Atari).png'
'Midnight Mutants (1990) (Atari).png'
'Motor Psycho (1990) (Atari).png'
'Ms. Pac-Man (1987) (Atari).png'
'Ninja Golf (1990) (Atari).png'
'One-on-One Basketball (1987) (Atari).png'
'Pete Rose Baseball (1989) (Absolute) [!].png'
'Pit Fighter (1992) (Atari) (Prototype) [a1].png'
'Pit Fighter (1992) (Atari) (Prototype) [a2].png'
'Pit Fighter (1992) (Atari) (Prototype).png'
'Planet Smashers (1990) (Atari).png'
'Pole Position II (1987) (Atari).png'
'Rampage (1989) (Activision) [!].mp4'
'RealSports Baseball (1988) (Atari).png'
'Rescue on Fractalus (1984) (Atari) (Prototype) (CC2 Hack).png'
'Rescue on Fractalus (1984) (Atari) (Prototype).png'
'Robotron 2084 (1987) (Atari).png'
'Scrapyard Dog (1990) (Atari).png'
'Sentinel (1988) (Atari) [!].png'
'Summer Games (1987) (Atari) [!].png'
'Super Huey UH-IX (1989) (Atari).png'
'Super Skateboardin'\'' (1989) (Absolute) [!].png'
'Tank Command (1988) (Froggo) [!].png'
'Title Match Pro Wrestling (1989) (Absolute) [!].png'
'Tomcat - The F-14 (1989) (Absolute) [!].png'
'Touchdown Football (1988) (Atari).png'
'Tower Toppler (1988) (Atari) [!].png'
'Tubes (2004) (Bruce Tomlin).png'
'Tubes (Nov 3) (2004) (Bruce Tomlin).png'
'Tubes (Nov 3-Signed) (2004) (Bruce Tomlin).png'
'Water Ski (1988) (Froggo).png'
'Winter Games (1987) (Atari) [!].png'
'Xenophobe (1989) (Atari) [!].png'
'Xevious (1987) (Atari).png'

Edit :
The gamelist.xml contain the video filenames mentioned above.
*.png or *.mp4

kiro

@Folly thanks, very helpful, will have a look tomorrow.

Folly

@kiro

If I can I will test the same in my VMWARE tomorrow.
I suspect it will run fine in that VM.

kiro

@Folly if it works in the VM then I'm really lost🤣...the code is the same and it is getting the information from the same backend API...so there's no reason for it to behave differently

Folly

@kiro

Yes, one would think so.
Could it be some version issue for python3 ?

BTW.
I am fully updated on the pi4.

Folly

@kiro

Just tested atari7800 in my VM with retroscraper-rpie.
I see the same issues, so I replicated the problem here.
So that's basically a good thing.

I also noticed now that game-specific retroarch config files are generated in the roms directory while not using the option --bezel.
Adding the option --bezel will give the same results as without this option.
One would assume that a bezel folder is created containing the bezel images when adding this option.

In the VM the program doesn't always go back to the cli command prompt, after saying :
SCRAPPING ENDED --- Thank you for using retroscraper!!
(I have to press ctrl+c sometimes)

Multiple tries ended differently with the same romset :

Game : Tubes
Game : Tubes
Game : Tower Toppler
Game : Tubes
Game : Water Ski
Game : Xenophobe
SCRAPPING ENDED --- Thank you for using retroscraper!!

Game : Tubes
Game : Tubes
Game : Tubes
Game : Water Ski
Game : Winter Games
SCRAPPING ENDED --- Thank you for using retroscraper!!

To be secure I added the .local/bin path as it complained earlier with the setup.

pi@Bullseye-VM:~$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
pi@Bullseye-VM:~$ PATH="$HOME/.local/bin:$PATH"
pi@Bullseye-VM:~$ echo $PATH
/home/pi/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

Still the same results.

kiro

@Folly thanks for all the testing, will start looking at it on Monday when I'm back.

Folly

@kiro

That's ok, no problem.

Have a good weekend ;-)

kiro

@Folly hi folly...I've quickly checked the backend and it returns format MP4 for the videos of all the games you mentioned, it's surprising to say the least. I will have a deeper check tomorrow.

Folly

@kiro

I installed retroscraper-rpie on my debian buster vm 32bit x86 VM.
I was thinking perhaps it will work on this VM.

First I did a total update/upgrade/reboot.

This is my log installing it all :
https://pastebin.com/raw/gzJpdz1b

I scraped the atari7800 roms and had the the same results as earlier. (same results as described earlier)
(used : python3 retroscraper.py --systems atari7800 )

Perhaps you can find something in my log.

I'am not quite sure which dependancies are loaded by the script.
If somehow the same dependancies are already installed on the OS in different paths with sudo and apt, for example.
Could be that the wrong depandent programs are used from other paths (with an other version).
I checked this theory by forcing only "$HOME/.local/bin" in the PATH like this :

pi@VM-32bit:~/retroscraper-rpie$ PATH="$HOME/.local/bin"
pi@VM-32bit:~/retroscraper-rpie$ /usr/bin/python3 retroscraper.py --systems atari7800 --cleanmedia

Same result, so this doesn't seem the be the problem either.

kiro

@Folly Thank you so much for all the testing.

I've uploaded new fixes into git (for both retroscraper versions) that should solve the missing games in the gamelists plus the bezels configuration issue.

I'm struggling to reproduce the .png issue (I tried exactly with atari7800) and nothing. Just to clear out possible issues, do you have some sort of proxy at your location/machine that could be caching the API requests?

I'll keep investigating in the meantime, and will start working on the relative path thing today.

Take care!

Folly

@kiro said in [SOFT] New Scraper in the works:

@Folly Thank you so much for all the testing.

I've uploaded new fixes into git (for both retroscraper versions) that should solve the missing games in the gamelists plus the bezels configuration issue.

Great !
I did some tests.

python3 retroscraper.py --systems atari7800 --cleanmedia

👍 - seems to be consistent now in detected games
👍 - seems to add all the detected games now in gamelist.xml
👍 - no game-specific configs are generated anymore

python3 retroscraper.py --systems atari7800 --bezel --cleanmedia

👍 - seems also to be consistent now in detected games
👍 - seems also to add all the detected games now in gamelist.xml
👍 - game-specific configs are generated as it should
👍 - bezel folder is created and bezel files are downloaded

I suspected that the bezel files would be in /home/pi/RetroPie/roms/atari7800/bezels but the were in /home/pi/RetroPie/overlays/atari7800/bezels.
Is it an idea to use this folder instead for the future ? :
/home/pi/RetroPie/roms/atari7800/media/retroarch/overlays/bezels
Using the same structure as I proposed for the emulationstation media :
/home/pi/RetroPie/roms/atari7800/media/emulationstation/<folders>
Then you can backup all in one time and with your option --cleanmedia you only have to delete 1 folder (media) which would make it much simpler.
Good idea ?

I'm struggling to reproduce the .png issue (I tried exactly with atari7800) and nothing. Just to clear out possible issues, do you have some sort of proxy at your location/machine that could be caching the API requests?

No I haven't.

I'll keep investigating in the meantime, and will start working on the relative path thing today.

I think that somehow string variables aren't updated correctly.
Though I have too little knowledge of your script.

Take care!

Same !

kiro

@Folly I've updated a new version of the 'light' retroscraper with a new flag '--debug' which will create a more complete log, maybe can you run it on the system tat still creates the png so we might understand what the problem is??

I've scanned my whole romset and no issues thousands of roms to be honest)

Thx!

Folly

@kiro

OK, I will do.

Found the log, but it's too big for pastebin.
Probably paste it on the google-drive.

kiro

@Folly thanks!

Folly

@kiro

~~retroscraper.log removed from google-drive.~~

Looks like the format variable is not updated correctly.

kiro

@Folly Thanks, will have a look right now.

On the other hand, I've updated both versions to cater for systems with 'non-standard names' such as 'shooters' :-) The only drawback is that the scraper will not be able to do a name search, due to the fact that it does not understand which is the actual system, but if checksums are in the DB they'll show up properly.

Folly

@kiro

Cool will give it a go.

Yea, it works really nice !

Seems a bit slower.
Does it have to search longer in the database with an unknown system ?

Folly

@kiro

I am getting this error, nothing works anymore :

pi@raspberrypi:~/retroscraper-rpie $ python3 retroscraper.py --systems arcade
Loading RetroScraper config File
Traceback (most recent call last):
  File "/home/pi/retroscraper-rpie/retroscraper.py", line 150, in <module>
    trans = complete['en']
TypeError: list indices must be integers or slices, not str