[SOFT] New Scraper in the works

kiro

Thanks @mitu ... without --cli flag it will work only if you have a desktop (sdl2) available.... It is strange about the 'nothing happening' with the --cli... Are you trying on a machine that has the es_systems.cf in the usual place? (/etc/emulationstation)?

That it creates an empty cfg is normal upon first run.

Is your machine connected to the internet? I assume yes, but I have not seen anything hitting the backend, so it looks like it is not connecting?

I'll have a look...thanks again!

mitu

without --cli flag it will work only if you have a desktop (sdl2) available....

I see no error message about not having a desktop env.

It is strange about the 'nothing happening' with the --cli... Are you trying on a machine that has the es_systems.cf in the usual place? (/etc/emulationstation)?

Yes, I have a standard RetroPie EmulationStation installed.

Is your machine connected to the internet? I assume yes, but I have not seen anything hitting the backend, so it looks like it is not connecting?

Well, yes, otherwise I wouldn't have been able to download it.

To be honest, I don't like the fact that's trying to connect somewhere without my choosing any options. Which 'backend' does it connect to ? I don't see anything mentioned on the download page.

kiro

@mitu it connects to my server to verify that the version running is ok, and to download the images (if running on non cli mode) and translations . It is the same IP as the download server...but it is strange I do not see it hitting this server (I see in the logs if scraping is being accessed)... I'll try later again in my RPI, but it should at least hit the server to download the startup data...strange..

mitu

@kiro said in [SOFT] New Scraper in the works:

it connects to my server to verify that the version running is ok

I though it would use one of the various scraping sources - doesn't it use them for downloading the artwork/metadata ?

sleve_mcdichael

@kiro said in [SOFT] New Scraper in the works:

Are you trying on a machine that has the es_systems.cf in the usual place? (/etc/emulationstation)?

What if I'm not?

https://retropie.org.uk/docs/EmulationStation/#es_systemscfg-edits

kiro

@sleve_mcdichael it does look in all those possible directories, if not you can edit the config if running in cli mode or choose it if you're running in windowed mode.

kiro

@mitu nope, it uses its own server.

mitu

@kiro said in [SOFT] New Scraper in the works:

nope,

I see. Then this is a nope from me also.

kiro

If anyone is interested, I've released the source code here: https://github.com/zayamatias/retroscraper

Enjoy :-)

kiro

A new version is out, it allows you to keep your favorites and play count after scraping (among other bug fixes).

https://github.com/zayamatias/retroscraper

kiro

A new slimmed-down version of the scraper is available and should be easier to install and run, check it out here:

https://github.com/zayamatias/retroscraper-rpie

Thanks for your feedback!

Folly

@kiro

Hi,
Perhaps you knew already but I am the developer of this script.

So for me it would be nice to find a good solution for generating gamelists that can be shared from within the script or perhaps could even be scraped from within the script.

We already have some predefined gamelists with media that can be downloaded from within the script for the categories :
konamih, tigerh , etc. (many done by @DTEAM)

The script can install arcade categories like "shooter" or "pinball" too.
However, for these categories there aren't predefined gamelists with media yet.
Sadly these "categories" are not recognised by your scraper.
Basically not a big problem so I renamed them to arcade and scraped them and renamed them back.
Now I should have a proper gamelist+media, right.
Well It doesn't work that way because the gamelist.xml contains full paths to files so I have to rename the roms directory from /arcade/ to /shooter/ to get it working again.

Well, we had a different approach with our predefined gamelists.
For our predefined gamelists, have a look here :
https://drive.google.com/drive/folders/1f_jXMG0XMBdyOOBpz8CHM6AFj9vC1R6m
You will see that we use relative paths rather than full paths.
This solution makes it easier to copy to an other named folder or to a computer with an other username without editing the gamelist.xml.
So my question is, could you incorporate that solution ?

We also set the images/videos in the directory media/emulationstation/ .
Which emphases that the media is used by emulationstation.
So when running emulationstation only 1 media folder is seen instead of images/marquees/videos.
I would like you to think about this too.
For your folder it would mean :
media/emulationstation/images
media/emulationstation/marquees
media/emulationstation/videos

If you see something in both suggestions then we apply somewhat the same "standard" to the gamelists.
With this we could somehow join some forces.

What do you think ?
Let me know.

kiro

@Folly Hi, Sure it makes sense! I'm away this weekend but will put this as new features in the upcoming versions. I just need some time to understand exactly what's done.

I believe that we could actually have the same gamelists created from my scraper, without the need to actually have 'predefined' gamelists.

I'll definitely have a look.

By the way, my scraper takes into accounts arcade systems such as 'Capcom classics', 'konami classics' and so forth, this is an example of my es_system for some of these classics:

  <system>
    <name>atariclassics</name>
    <extension> .7z .cue .fba .iso .zip .7Z .CUE .FBA .ISO .ZIP</extension>
    <platform>arcade</platform>
    <theme>arcadeatariclassics</theme>
    <command>/opt/retropie/supplementary/runcommand/runcommand.sh 0 _SYS_ arcade %ROM%</command>
    <path>../roms/atariclassics</path>
    <fullname>Atari Classics</fullname>
  </system>

The scraper will match the name tag 'atariclassics' in this example.

Folly

@kiro

Ok, nice to hear you agree.
Would really be great if you could accomplish that.

So if I can somehow add <platform>arcade</platform> to a category then it should be recognised right ?

kiro

@Folly You do not need to add anything like 'platform'.... my scraper recognizes the roms based on their checksum, not their names, directories or anything. In principle, you could put all roms into a single directory and it would recognize them as long as their checksum is in the database.
If the checksum is not in the database then the <name> tag will help the scraper try to figure out the game, but it is not mandatory.

To answer your question about the <platform>, I'm not sure why this tag is there, it is not taken into consideration by the scraper :-).

Folly

@kiro

Somehow my categories are not recognised by your script so I can't select them in the systems menu.
This is an example of such category from the es_systems.cfg :

<system>
  <name>shooter</name>
  <fullname>shooter</fullname>
  <path>/home/pi/RetroPie/roms/shooter</path>
  <extension>.cmd .zip .7z  .CMD .ZIP .7Z </extension>
  <command>/opt/retropie/supplementary/runcommand/runcommand.sh 0 _SYS_ shooter %ROM%</command>
  <platform>shooter</platform>
  <theme>shooter</theme>
</system>

Indeed, changing it to <platform>arcade</platform> doesn't help also.

kiro

@Folly yes, it will not recognize such 'custom' categories...maybe I should think on way to imrpove this. As I said this should not matter if the checksums are found in the DB. Maybe I should think of an alternative here... Let me have a think this weekend

Folly

@kiro

If you can do all of that it would definitely be an improvement ;-)

Folly

@kiro

Hi,

Scanning on the arcade directory still gives me videos with the .png extension in the video directory, though some are good with .mp4.
I have .png files in the folders images and marquees, so that seems to be OK.

After this discovery I tried archimedes again.
This is good, no videos with .png extension.

So there still seems to be an issue with videos called .png in the videos directory for other systems.

kiro

@Folly thanks! The fact that there are still .PNG files in the video directory might be due to them still being there from previous scraps. Try using the "cleanmedia" option or --cleanmedia if running from command line. Let me know how it goes.