[SOFT] New Scraper in the works

Folly

I am experimenting with running retroscraper "online".

The advantage would be that the script is always up to date and that the basic script doesn't need to be stored anywhere.

I want to look if this is a good way of adding it to my script.
That way I only have to install the regular depends.

I already use a python script called gdrivedl in my module-script.
This is an example of running gdrivedl.py online using only the help option :

curl https://raw.githubusercontent.com/matthuisman/gdrivedl/master/gdrivedl.py | python3 - -h

So I wanted to see if I could do the same with your "retroscraper-rpie".
Basically, with the same help option and it actually works as expected :

curl https://raw.githubusercontent.com/zayamatias/retroscraper-rpie/main/retroscraper.py|python3 - -h

However your script uses more dependant scripts.
So when it comes to the actual scraping, it will miss these dependant scripts. (apicalls, checksums, scrapfunctions)

So I have a question :
Is there a way to make one script including these dependant scripts so it can be run online ?

kiro

@Folly said in [SOFT] New Scraper in the works:

curl https://raw.githubusercontent.com/zayamatias/retroscraper-rpie/main/retroscraper.py|python3 - -h

Hi Folly, I guess it would be possible to include these files in the main python file, but it will add complexity to the maintenance of the code (at least for me :-) ). Let me check if something can be done through the command line or any other alternative. will keep you posted.

EDIT: Just discovered the httpimport module which actually allows you to import dependencies from the web. I'm testing it right now. It may have an impact on the initialization of the script though.

kiro

@Folly , some update. Try this:

curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3

It has 2 consequences:

1-There's a new dependency (httpimport module, to be installed with pip)
2-Startup is slower since it grabs everything form the repository directly.

Let me know if it works for you.

Folly

@kiro

Thanks !
I will try it when I can and let you know how it goes.

Folly

@Folly

Ok, I tested it out and it seems the idea could work.

But I have a few problems to overcome.

I have added the dependency installs in the depends function of my module script .
However I have to find a way to skip it properly so it will not try to when it's not needed.
Or I have to separate so it can be installed in a menu option.

Was hoping that it would run ok the first time.
It did work in the terminal
But from the script it didn't work.
I had to change the cmd, in my module script, and run it with su $user -c because the dependencies are installed under the normal user and therefor the script has also be run with the normal user.
So that one is solved.

To my surprise the waiting time of getting the script and it's dependencies remotely doesn't take that much time
Of-course you have to wait a little while but it's acceptable in my opinion and the scraping speed it just as normal.

Sadly I ran into an issue.
The crc and the sha1sum are saved in the name in the gamelist.xml.
This is an example :

<name>Boulder Dash,149604,0C0F7FF29CBE1CAA5A22AA0CF291BF5560DDD358</name>

Do you have an idea what it could be ?

Edit :
For archiving and looking up purposes I will add here what code I added.

Getting the retroscraper-remote depends in my module-script :

    local pip=$( su $user -c "python3 -c 'import pip'" 2>&1 )
    local succ="ModuleNotFoundError"
    echo "$pip"
    if [[ $pip == *"$succ"* ]]; then
        su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py"
        su $user -c "python3 /tmp/get-pip.py"
    fi
    su $user -c "python3 -m pip install --user --upgrade pip wheel setuptools"
    su $user -c "python3 -m pip install --user googletrans==4.0.0rc1 Pillow==9.2.0 requests==2.21.0 httpimport==1.1.0"

Test command from the terminal :

curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3 - --systems deco_cassette --relativepaths --mediadir media/emulationstation --nobackup

Test command from the module-script :

su $user -c "curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3 - --systems deco_cassette --relativepaths --mediadir media/emulationstation --nobackup"

kiro

@Folly Argh! Sorry, I uploaded a test version (it's been a while since I didn't touch the code, sorry) I've updated it! Should be fine now..

Folly

@kiro

Nice, so there is probably no issue.
Will do test again when I can.

I hope it's the correct route for me.
Nevertheless it's already quite interesting that we managed to run it this way.

kiro

@Folly indeed, I never thought about that possibility to be honest. Let me know how it goes!

Folly

@kiro

Nice, it's working ;-)

kiro

@Folly Cool... I'm still in the process of importing all mess roms...it's a pain in the .... anyway, as soon as that is done, I'll start adding the images from your playlists..

Folly

@kiro

A pain ?
I can imagine.
Just be patient there is no hurry.

Hopefully we can manage that the outputs are be becoming a good alternative for the predefined gamelists,
because the scraping speed is much higher than downloading speed of the predefined ones.
So all in all that would really be great.

kiro

Hi @Folly! Finally all roms from mame-sl are in the backend and I've added the pictures found in your gamelists where missing, phew... hope this helps!

Supported systems with figures here: http://77.68.23.83/

Folly

@kiro

Very cool that you managed all of this.
I can only imagine how much work it must be.
Thumbs up ^^ :-)

I will do some scraping when I can ;-)

Folly

@kiro said in [SOFT] New Scraper in the works:

I've added the pictures found in your gamelists where missing

Which pictures are missing ?

kiro

@Folly I meant to say, when there were no pictures available on the backend, I imported those on your gamelists :-) Sorry for the confusion!

Folly

@kiro

Ok , I understand.

Looked at your page, I must say the list is very impressive !!!

Just a question :
How are the new systems detected and what is the accompanied rom-directory for detection ?
Or does the directory name make no difference in detecting ?

Edit :
Oh yea,
When it doesn't detect it by default it assumes it is "arcade/non-arcade(mame)", Right ?

kiro

@Folly So, systems are detected on base of common names, which for mame-sl these are the 'software list' names... for example "pv1000" for the casio-pv1000. This name is expected to be in the 'system' tag of the es_systems.xml, for example:

  <system>
    <name>gameandwatch</name>
    <fullname>Arcade</fullname>
    <path>/home/pi/RetroPie/roms/gameandwatch</path>
    <extension>.7z .cue .fba .iso .zip .7Z .CUE .FBA .ISO .ZIP</extension>
    <command>/opt/retropie/supplementary/runcommand/runcommand.sh 0 _SYS_ arcade %ROM%</command>
    <platform>arcade</platform>
    <theme>arcade</theme>
  </system>

Now, this being said, this is only necessary if you have roms which are not in the DB and want to match by maybe name or simlarities to another one in that system. Retroscraper will always use the checksum from your file and find a match accordingly, regardless of which system they actually belong to (you cannot have 2 file swith same checksums).

Hope this helos.

Folly

@kiro

Ok, good explanation.

Will let you know what my experiences are when scraping those mame-sl roms / systems.

Folly

@kiro

I just tried to scrape apogee, dragon64 and gamecom.
The systems seem to be recognized.
Sadly the files aren't recognized.

kiro

@Folly Hmm.. Interesting, for some reason the new additions have not been 'apified' if I may say..investigating...will get back to you asap...

ARGH! I updated with the test DB :-) fixing this, will be back soon...