[SOFT] New Scraper in the works

Folly

Hi,
Perhaps you knew already but I am the developer of this script.

So for me it would be nice to find a good solution for generating gamelists that can be shared from within the script or perhaps could even be scraped from within the script.

We already have some predefined gamelists with media that can be downloaded from within the script for the categories :
konamih, tigerh , etc. (many done by @DTEAM)

The script can install arcade categories like "shooter" or "pinball" too.
However, for these categories there aren't predefined gamelists with media yet.
Sadly these "categories" are not recognised by your scraper.
Basically not a big problem so I renamed them to arcade and scraped them and renamed them back.
Now I should have a proper gamelist+media, right.
Well It doesn't work that way because the gamelist.xml contains full paths to files so I have to rename the roms directory from /arcade/ to /shooter/ to get it working again.

Well, we had a different approach with our predefined gamelists.
For our predefined gamelists, have a look here :
https://drive.google.com/drive/folders/1f_jXMG0XMBdyOOBpz8CHM6AFj9vC1R6m
You will see that we use relative paths rather than full paths.
This solution makes it easier to copy to an other named folder or to a computer with an other username without editing the gamelist.xml.
So my question is, could you incorporate that solution ?

We also set the images/videos in the directory media/emulationstation/ .
Which emphases that the media is used by emulationstation.
So when running emulationstation only 1 media folder is seen instead of images/marquees/videos.
I would like you to think about this too.
For your folder it would mean :
media/emulationstation/images
media/emulationstation/marquees
media/emulationstation/videos

If you see something in both suggestions then we apply somewhat the same "standard" to the gamelists.
With this we could somehow join some forces.

What do you think ?
Let me know.

kiro

@Folly Hi, Sure it makes sense! I'm away this weekend but will put this as new features in the upcoming versions. I just need some time to understand exactly what's done.

I believe that we could actually have the same gamelists created from my scraper, without the need to actually have 'predefined' gamelists.

I'll definitely have a look.

By the way, my scraper takes into accounts arcade systems such as 'Capcom classics', 'konami classics' and so forth, this is an example of my es_system for some of these classics:

  <system>
    <name>atariclassics</name>
    <extension> .7z .cue .fba .iso .zip .7Z .CUE .FBA .ISO .ZIP</extension>
    <platform>arcade</platform>
    <theme>arcadeatariclassics</theme>
    <command>/opt/retropie/supplementary/runcommand/runcommand.sh 0 _SYS_ arcade %ROM%</command>
    <path>../roms/atariclassics</path>
    <fullname>Atari Classics</fullname>
  </system>

The scraper will match the name tag 'atariclassics' in this example.

Folly

@kiro

Ok, nice to hear you agree.
Would really be great if you could accomplish that.

So if I can somehow add <platform>arcade</platform> to a category then it should be recognised right ?

kiro

@Folly You do not need to add anything like 'platform'.... my scraper recognizes the roms based on their checksum, not their names, directories or anything. In principle, you could put all roms into a single directory and it would recognize them as long as their checksum is in the database.
If the checksum is not in the database then the <name> tag will help the scraper try to figure out the game, but it is not mandatory.

To answer your question about the <platform>, I'm not sure why this tag is there, it is not taken into consideration by the scraper :-).

Folly

@kiro

Somehow my categories are not recognised by your script so I can't select them in the systems menu.
This is an example of such category from the es_systems.cfg :

<system>
  <name>shooter</name>
  <fullname>shooter</fullname>
  <path>/home/pi/RetroPie/roms/shooter</path>
  <extension>.cmd .zip .7z  .CMD .ZIP .7Z </extension>
  <command>/opt/retropie/supplementary/runcommand/runcommand.sh 0 _SYS_ shooter %ROM%</command>
  <platform>shooter</platform>
  <theme>shooter</theme>
</system>

Indeed, changing it to <platform>arcade</platform> doesn't help also.

kiro

@Folly yes, it will not recognize such 'custom' categories...maybe I should think on way to imrpove this. As I said this should not matter if the checksums are found in the DB. Maybe I should think of an alternative here... Let me have a think this weekend

Folly

@kiro

If you can do all of that it would definitely be an improvement ;-)

Folly

@kiro

Hi,

Scanning on the arcade directory still gives me videos with the .png extension in the video directory, though some are good with .mp4.
I have .png files in the folders images and marquees, so that seems to be OK.

After this discovery I tried archimedes again.
This is good, no videos with .png extension.

So there still seems to be an issue with videos called .png in the videos directory for other systems.

kiro

@Folly thanks! The fact that there are still .PNG files in the video directory might be due to them still being there from previous scraps. Try using the "cleanmedia" option or --cleanmedia if running from command line. Let me know how it goes.

Folly

@kiro

I removed all files manually.
Now I am also using your retroscraper-rpie which does the same.
I am seeing the files being added in the file-explorer.
So definitely an issue.

I also noticed that there are many videos that will not display, sound on the other hand is played though.
Could it be that some codecs are missing ?
I tried some files in VLC and gave me black screen but sound is there.

kiro

@Folly thanks, I'll have a look when I'm back home on Monday, I might need to revalidate the videos. Will also check about the PNG thing, definitely strange.

Folly

@kiro

If I can implement the setup somehow in my script then your retroscraper-rpie would probably be easy enough to add in my script.
I will have to think about it some time.

kiro

@Folly thanks! I actually do have a script which is running (installation goes fine as well as execution) based on scraper.sh script. My only concern right now is that output is buffered, so it seems (only seems) as if the scraper freezes.
But thanks for the offer, definitely we can talk about doing something together.

Folly

@kiro

I just did atari7800, although it also has the .png video issue, all videos and pictures are good.
I must say if something is recognised then recognition seems to be perfect.
I do not see that with other scraper methods.
I also notice that the scraping is very fast, much faster that other scrape methods.
Really nice !!!

Folly

@kiro

For your information.
I am testing your retroscraper-rpie now in my Debian11-x86_64 in VMWARE.
Looks like the .png problem for videos doesn't occur there.
Also the gamelist.xml contain the correct naming for the videos (.mp4)
So this problem could be specific for the RPI/raspbian/raspbian version.

ALL video seem to be playing correctly in emulationstation in my VM, although, for some videos, emulationstation seems to give errors like this :

[src/libmpg123/layer3.c:2184] error: part2_3_length (1408) too large for available bit count (1368)

Could it be that some videos are not that compatible or have some corruption ?

kiro

@Folly hi, videos are being fixed as we speak, it will need some time. PNG issue should be independent from platform (after all is running the same code) let me know if you encounter this problem again.

Folly

@kiro

I just scraped some on my RPI4 with both scrapers (normal and rpie)

For archimedes all videos are named *.mp4, so for this system that's good and all videos seem to play now.
However I discovered a new problem.
The last 2 games that are recognised and for which media is downloaded are not added to the gamelist.xml.
I checked older gamelists and they also miss these.

For arcade(shooter category),arcade(pinball category) and atari7800 the .png problem still remains.
Here I also checked If something was missing in atari7800.
The last 4 games that are recognised and for which media is downloaded are not added to the gamelist.xml.
For arcade(shooter category) the last 3 games were missing.
For arcade(pinball category) the last 3 games were missing.

Edit :
For the system with videos named as .png for some videos only sound is available.

kiro

@Folly hi, thanks for the testing, can you give me a game name for the 7800 for which videos are downloaded as PNG? I cannot reproduce the issue.

Just for clarity, the games that are not in the gamelisy, have the proper extension? (Recognized one)? I'm sure yes but better be safe 🙂

Folly

@kiro

Do you need some sha1sums ?
Here is one :

sha1sum 'Choplifter (1987) (Atari).a78'
6457dd40aeadb9733602c4ad3498780ebd7bf0d1  Choplifter (1987) (Atari).a78

Looks like these games are missing :

'Water Ski (1988) (Froggo).a78'
'Winter Games (1987) (Atari) [!].a78'
'Xenophobe (1989) (Atari) [!].a78'
'Xevious (1987) (Atari).a78'

This is the output of my videos folder for atari7800 :

'3D Asteroids (1987) (Atari) (Prototype) [a1].png'
'3D Asteroids (1987) (Atari) (Prototype).png'
'Ace of Aces (1988) (Atari).png'
'Alien Brigade (1990) (Atari) (CC2 Hack).png'
'Alien Brigade (1990) (Atari) [!].png'
'Asteroids (1987) (Atari).png'
'Ballblazer (1987) (Atari-Lucasfilm).png'
'Barnyard Blaster (1988) (Atari) [!].png'
'Basketbrawl (1990) (Atari).png'
'Beef Drop 7800 (Oct 30-Rev 08) (2004) (CC2 Hack).png'
'Centipede (1987) (Atari).png'
'Choplifter (1987) (Atari).png'
'Commando (1989) (Atari).png'
'Crack'\''ed (1988) (Atari).png'
'Crossbow (1988) (Atari) (CC2 Hack).png'
'Crossbow (1988) (Atari).png'
'Dark Chambers (1988) (Atari).png'
'Desert Falcon (1987) (Atari).mp4'
'Desert Falcon (2004) (Atari) (HSC Hack) [a1].mp4'
'Desert Falcon (2004) (Atari) (HSC Hack).mp4'
'Dig Dug (1987) (Atari).png'
'Donkey Kong (1988) (Atari).png'
'Donkey Kong Jr (1988) (Atari).png'
'Double Dragon (1989) (Activision) [!].mp4'
'F-18 Hornet (1988) (Absolute) [!].mp4'
'Fatal Run (1990) (Atari).png'
'Fight Night (1988) (Atari).png'
'Food Fight (1987) (Atari).png'
'Galaga (1987) (Atari).png'
'Gato (198x) (Atari) (Prototype) [a1].mp4'
'Gato (198x) (Atari) (Prototype) [a2].mp4'
'Gato (198x) (Atari) (Prototype).mp4'
'Hat Trick (1987) (Atari).png'
'Ikari Warriors (1989) (Atari).png'
'Impossible Mission (1987) (Atari) [a1].mp4'
'Impossible Mission (1987) (Atari) [f1].mp4'
'Impossible Mission (1987) (Atari) [!].mp4'
'Jinks (1989) (Atari).png'
'Joust (1987) (Atari).png'
'Karateka (1987) (Atari).png'
'Klax (1992) (Atari) (Prototype) [a1].mp4'
'Klax (1992) (Atari) (Prototype) [f1].mp4'
'Klax (1992) (Atari) (Prototype).mp4'
'Klax (2004) (Impossible Level 1 Hack).mp4'
'Klax (2004) (Impossible Level 2 Hack).mp4'
'Klax (2004) (Impossible Level 3 Hack).mp4'
'Kung Fu Master (1989) (Absolute) [!].png'
'Mario Brothers (1988) (Atari).png'
'Mat Mania Challenge (1989) (Atari).png'
'Mean 18 Golf (1988) (Atari) [!].png'
'Meltdown (1990) (Atari).png'
'Midnight Mutants (1990) (Atari).png'
'Motor Psycho (1990) (Atari).png'
'Ms. Pac-Man (1987) (Atari).png'
'Ninja Golf (1990) (Atari).png'
'One-on-One Basketball (1987) (Atari).png'
'Pete Rose Baseball (1989) (Absolute) [!].png'
'Pit Fighter (1992) (Atari) (Prototype) [a1].png'
'Pit Fighter (1992) (Atari) (Prototype) [a2].png'
'Pit Fighter (1992) (Atari) (Prototype).png'
'Planet Smashers (1990) (Atari).png'
'Pole Position II (1987) (Atari).png'
'Rampage (1989) (Activision) [!].mp4'
'RealSports Baseball (1988) (Atari).png'
'Rescue on Fractalus (1984) (Atari) (Prototype) (CC2 Hack).png'
'Rescue on Fractalus (1984) (Atari) (Prototype).png'
'Robotron 2084 (1987) (Atari).png'
'Scrapyard Dog (1990) (Atari).png'
'Sentinel (1988) (Atari) [!].png'
'Summer Games (1987) (Atari) [!].png'
'Super Huey UH-IX (1989) (Atari).png'
'Super Skateboardin'\'' (1989) (Absolute) [!].png'
'Tank Command (1988) (Froggo) [!].png'
'Title Match Pro Wrestling (1989) (Absolute) [!].png'
'Tomcat - The F-14 (1989) (Absolute) [!].png'
'Touchdown Football (1988) (Atari).png'
'Tower Toppler (1988) (Atari) [!].png'
'Tubes (2004) (Bruce Tomlin).png'
'Tubes (Nov 3) (2004) (Bruce Tomlin).png'
'Tubes (Nov 3-Signed) (2004) (Bruce Tomlin).png'
'Water Ski (1988) (Froggo).png'
'Winter Games (1987) (Atari) [!].png'
'Xenophobe (1989) (Atari) [!].png'
'Xevious (1987) (Atari).png'

Edit :
The gamelist.xml contain the video filenames mentioned above.
*.png or *.mp4

kiro

@Folly thanks, very helpful, will have a look tomorrow.