[SOFT] New Scraper in the works

kiro

I've almost completed the retroscraper.sh script that would allow for seamless integration with retropie_setup script.

https://github.com/zayamatias/retroscraper-rpie/blob/main/retroscraper.sh

If you copy under the ~/RetroPie-Setup/scriptmodules/supplementary/ directory, you should see it under the optional packages:

Install it from source, once done go to configurations / options

Choose your preferences, decide wether to scan everything or just some system(s):

And let it run!

Let me know how it works!

Next steps, polish it a bit and hopefully integrate it with retropie?

Folly

@kiro

Had a quick look at the script.
Was wondering where the dependencies were installed.
Saw that you used su $user so obviously it's installed in the same path /home/pi/.local/bin .
So you made it uniform with the normal install, great !

Will have a look at the menu later.

kiro

@Folly the dependencies are basically a couple of python modules, not much more now, If I remember correctly. Still I have to improve the install/update script so things like pip dies not get installed everytime and checking that the modules are already installed. Retroscraoer itself will install in /opt/retropie/supplementary as every other tool from within retropie_setuo

Folly

@kiro

Indeed, you should implement checks so things will not install every time.
Are you planning to do it with checksums or checking files/versions ?

Btw.
Is a pip install as dependency not an option ?
(as apt install from the function depends)

kiro

@Folly pip could be installed by apt, but I've had bad experiences with the pip install from the distribution, not always being the latest version. I ws planing to check if running pip would succeed and if not install it. And the. Checking if the dependencies are there (or even asking to install, since it's just a couple of modules and the installation is faulty quick) I will see what's the least cumbersome option.

Folly

@kiro

On second thought, I think it not a really big deal.
The update/install procedure is reasonably quick and fail-proof.
Perhaps it's best to keep it the way it is.

kiro

@Folly Could be, I'm going to be checking that today. On the other hand, I'm trying to improve the performance of the script when running under retropie-setup, latest test seems to be positive.

kiro

Hi @Folly ! I just finished adding the --mediadir flag, if you put a directory after it, it will store all media under it, it has two possibilities:

You start the directory by a '/' it will create an absolute path and add the system name (just to avoid clashes) so for example:

--mediadir /home/pi/RetroPie/medias/

will save medias in

/home/pi/RetroPie/medias/system_name/

The second option is not to start with '/' which will consider the path as relative, and save the media under the system directory, for example:

--mediadir emulationstation/medias/

will save medias in

/system_path/emulation_station/medias/

Also, I've improved (I hope) scraping speed.

Cheers!

Folly

@kiro

Great !
I wil test it asap.

Also had a quick look at the module-script config/tool menu.
I like the toggle setting in your menu and I did a small scrape with it.

kiro

@Folly Cool, let me know how it goes.. I have the impression that under retropie_setup it gets stuck sometimes, let me see if you experience the same behavior. I've also updated the retropie_setup retroscraper.sh script so it will not install pip again if already installed :-)

Folly

@kiro

Here are a few minor things.
Typo :
Deleing Images

Echo'ed twice :
Writing Gamelist
Writing Gamelist

I did 2 tests.

python3 /opt/retropie/supplementary/retroscraper/retroscraper.py --mediadir media/emulationstation/ --systems upright --cleanmedia --relativepaths

Nice to see that all media is now in 1 directory.
Did not expect that but I think it's nice for editing (all in one place)
--mediadir media/emulationstation/ works as expected ;-)

Also tried the --mediadir with absolutepath.
This was my test :

python3 /opt/retropie/supplementary/retroscraper/retroscraper.py --mediadir /home/pi/RetroPie/roms/upright --systems upright --cleanmedia

I had a look at your explanation, it looks like it doesn't actually do the same as when adding a relative path.
This could be confusing.
Though I see what you want to accomplish.
Putting the gamelist media outside the roms directory with adding the systemname.
I did the above test and used a path with a roms directory.
According to the gamelist the media files would be in :
/home/pi/retropie/roms/upright/upright/
Though the directory isn't created and there is no media.
So there is probably an issue with creating the correct directory.

I think you need to remove adding the system name to the absolute path so the absolute path option can be used both ways for placing media outside the roms directory and inside the roms directory.
Or only add the system name if the path is outside the roms directory.

I hope I gave you some good feedback.

kiro

@Folly Hi, thanks for the typo :-)

The problem with having everything in a single directory comes when you're scanning multiple systems at the same type. Imagine you scan snes and md, and ask absolute path for images such as '/home/pi/media' (just as an example)... and you have one file 'xenon 2.zip' in snes, and the exact same file 'xenon 2.zip' in md... now, if I didn't use subfolders (system name) one image would be 'xenon 2-image.zip' and the other would be 'xenon 2-image.zip' ... see where I'm getting?

Another option would be to create 'xenon 2-snes-image.zip' and 'xenon 2-md-image.zip', but I'd be willing to say that from a management persepctive (imagine you had 120k roms in your whole rpie setup) it would be messier having everything in just 1 folder. Thoughts?

I mean, in the end, is just a matter of taste, I could add a '--nosysfolder' flag or something quite easily to avoid creating a folder per system if using relative paths.

Remember that the end objective is to allow you to scrap ALL your roms in a quick way.

Folly

@kiro

Good info.
Yes I am only seeing it from perspective scanning only one system.
I now see why you did it this way.

If you add a good description it shouldn't be a problem.

Though the creation of the folder did not work with my 2nd test.
What's your opinion.
Edit :
My fault I looked into the wrong folder RetroPie in stead of retropie.
So it does work ! , sorry.

kiro

@Folly maybe have a look at the logs? I know this could happen due to permission issues, sometimes... it should ouptut something like :

CANNOT CREATE DIRECTORY [Your/directory] - ERROR [Explanation] - PLS VERIFY AND TRY AGAIN

And then exit ... so strange is not exiting...have you got he latest version?

Folly

@kiro said in [SOFT] New Scraper in the works:

so strange is not exiting...have you got he latest version?

It does exit now always, so no problem.

I checked the --mediadir option again with absolute path.
The issue is that when using a path with the folder, for example, RetroPie (R and P are uppercase) RetroPie isn't used/created but retropie is. (everything with lowercase)

I see you have fixed the other issues ;-)

kiro

@Folly About the folders, didn't quite understand, so if you put /RetroPie/ will it create /retropie/ or you put /retropie/ and it will not be recognized as /RetroPie/ ? The latter being totally ok as directories under unix/linux are case sensitive..

Folly

@kiro said in [SOFT] New Scraper in the works:

if you put /RetroPie/ will it create /retropie/

This.

kiro

@Folly Ok, then probably I'm putting everything lowercase where I should not, going to check right now..

Should be fixed and uploaded to git...

Folly

@kiro

I get this :

python3 /opt/retropie/supplementary/retroscraper/retroscraper.py --mediadir /home/pi/RetroPie/roms/upright --systems upright --cleanmedia
Starting scraping
Scanning Files
Scanning Directory
ERROR SYSTEM upright: NO ROMS IN DIRECTORY /home/pi/RetroPie/roms/upright/
Scan Done!
SCRAPPING ENDED --- Thank you for using retroscraper!!

kiro

@Folly said in [SOFT] New Scraper in the works:

NO ROMS IN DIRECTORY

Ok, should be fixed, interesting. To avoid scanning through images, I remove anything whose path is the same as the 'media' path, of course, if the mediapath is the same as the system path, well guess what.... I just added the system name to the media path so it should be ok now.