[SOFT] New Scraper in the works

Folly

Just want to say :
I really appreciate the effort you putting in it.
Keep up the good work ;-)

kiro

@Folly Thanks, this is just my way to give back something to the retro community for all the fun over the years! even if it is just a silly scraper :-)

kiro

@Folly just wanted to give you a quick update, I'm still in the process of converting problematic videos, not easy with over 40k to go over (even being a script), will keep you posted. Take care!

Folly

@kiro

That's quite a lot.
How long does it take, do you think.
Just curious.

kiro

@Folly, well it depends on the number of videos that actually need conversion (impossible to estimate), but it also depends on me doing the script right the first time and not having to launch it over and over, the internet going off just in the middle of the process which means I need to restart all over, you know, the usual stuff... if everything goes ok it should be fine by this evening or tomorrow morning the latest. Luckily once the conversion is done, it should be fine. Funny thing: most of the videos not working seem to have been captured with mame, go figure :-) And, this means other databases will probably have the same issue.

Folly

@kiro

Nice to know.
I will see the result in time ;-)
Good luck !

kiro

@Folly Oh, BTW, I started playing around with a script for retropie-setup, it is in the git repository and it is a copy of another scraper's script. It works as far as the installation and running of the scraper goes. If you want to play with it, put it in ~/RetroPie-Setup/scriptmodules/supplementary and it should appear under the optional modules of retropie-setup. Just added the system selection option to rsetup ;-) ... videos still going ...

kiro

@Folly Hi, good news, the script for the videos is done, all videos should be working (at least those with a similar problem as pc-mario).

Folly

@kiro

Great !
Will test it this evening on my pi4.

I already see the improvement on my VM.
There are less errors reported after closing emulationstation.
Superb ;-)

kiro

@Folly thanks! In the meantime I've updated the retroscraper.sh script, seems to be working quite well, although it needs some cleaning and ironing, but functionality wise it seems to be ok.

kiro

I've almost completed the retroscraper.sh script that would allow for seamless integration with retropie_setup script.

https://github.com/zayamatias/retroscraper-rpie/blob/main/retroscraper.sh

If you copy under the ~/RetroPie-Setup/scriptmodules/supplementary/ directory, you should see it under the optional packages:

Install it from source, once done go to configurations / options

Choose your preferences, decide wether to scan everything or just some system(s):

And let it run!

Let me know how it works!

Next steps, polish it a bit and hopefully integrate it with retropie?

Folly

@kiro

Had a quick look at the script.
Was wondering where the dependencies were installed.
Saw that you used su $user so obviously it's installed in the same path /home/pi/.local/bin .
So you made it uniform with the normal install, great !

Will have a look at the menu later.

kiro

@Folly the dependencies are basically a couple of python modules, not much more now, If I remember correctly. Still I have to improve the install/update script so things like pip dies not get installed everytime and checking that the modules are already installed. Retroscraoer itself will install in /opt/retropie/supplementary as every other tool from within retropie_setuo

Folly

@kiro

Indeed, you should implement checks so things will not install every time.
Are you planning to do it with checksums or checking files/versions ?

Btw.
Is a pip install as dependency not an option ?
(as apt install from the function depends)

kiro

@Folly pip could be installed by apt, but I've had bad experiences with the pip install from the distribution, not always being the latest version. I ws planing to check if running pip would succeed and if not install it. And the. Checking if the dependencies are there (or even asking to install, since it's just a couple of modules and the installation is faulty quick) I will see what's the least cumbersome option.

Folly

@kiro

On second thought, I think it not a really big deal.
The update/install procedure is reasonably quick and fail-proof.
Perhaps it's best to keep it the way it is.

kiro

@Folly Could be, I'm going to be checking that today. On the other hand, I'm trying to improve the performance of the script when running under retropie-setup, latest test seems to be positive.

kiro

Hi @Folly ! I just finished adding the --mediadir flag, if you put a directory after it, it will store all media under it, it has two possibilities:

You start the directory by a '/' it will create an absolute path and add the system name (just to avoid clashes) so for example:

--mediadir /home/pi/RetroPie/medias/

will save medias in

/home/pi/RetroPie/medias/system_name/

The second option is not to start with '/' which will consider the path as relative, and save the media under the system directory, for example:

--mediadir emulationstation/medias/

will save medias in

/system_path/emulation_station/medias/

Also, I've improved (I hope) scraping speed.

Cheers!

Folly

@kiro

Great !
I wil test it asap.

Also had a quick look at the module-script config/tool menu.
I like the toggle setting in your menu and I did a small scrape with it.

kiro

@Folly Cool, let me know how it goes.. I have the impression that under retropie_setup it gets stuck sometimes, let me see if you experience the same behavior. I've also updated the retropie_setup retroscraper.sh script so it will not install pip again if already installed :-)