Where can I scrape Archimedes game pix and info?

kiro

@duglor hi! Try here https://github.com/zayamatias/retroscraper

duglor

@kiro Thanks!!!!

Folly

Hi, I am trying to install your retroscraper now.
I am running this on :
Raspberry pi OS (Debian11/Bullseye) 32 bit arm with desktop and python 3.9.2 .
I ran with 'python retroscraper.py' first but that gave me too much errors.
Now using 'python3 retroscraper.py' and more packages are installed though it complained about "Rust compiler" not installed.
Used this now to install rust :
https://www.rust-lang.org/tools/install
Seems it's missing in your readme.
Did a second run and packages are now compiled.
This process takes quite some time,
I was thinking it had freezed but it seems I am up and running now.

Great !

Will try to scrape archimedes.

kiro

@Folly Thanks for the testing! Yes, indeed I thought I got rid of the rust requirement but, alas, no. I've added it to the readme again.

Due to the fact that the dependencies need to be built, it takes some time in machines such as the RPI.

Thanks for your patience!

Folly

@kiro

I did archimedes running on the zip files from the extracted archimedes.zip (from mame-sl)

About 35/415 files are recognised.
Nice thing is that the gamelist.xml and the media is directly written it the roms/archimedes folder.
Though the folders images and marquees do not contain any files.
However the videos folder contain the videos but all files are named *.png.
In emulationstation the detected games have descriptions but no images or videos are displayed.
I also saw that the gamelist.xml is missing "newlines" so quite difficult to read or edit.

So there seems to be some issues.

kiro

@Folly Hmm that's really strange.... I've never seen this behavior (I'm constantly running the scraper for testing purposes)....Ahh I see for the videos... need to do something ... but the other directories is strange... I'll get back to you

Folly

@kiro

I used the default configuration. (nothing was changed)

EDIT:
I will remove this message later.
The "s" is missing from your post :

sudo apt install libxml2-utils

You can change it in your post, after that I will remove this EDIT.

kiro

@Folly Yes, I think there was an issue with the API cache, need to regenerate it,w ill take some time :-(

regarding the format of the file list, yes, I see what you mean. This is the way Python creates XMLs. You can make it more readable by using xmllint

sudo apt install libxml2-util

More info here

https://www.baeldung.com/linux/xmllint

I'll get back to you when the cache is regenerated. And thanks again for trying!

kiro

@Folly Hi ... issues with videos and images should be fixed... maybe marquees tend to be empty because there are fewer images available for that
Let me know how it goes now.

You need to download the latest version of course.

Thanks!

duglor

@kiro I think i'm a little dim to run your Archimedes scraper.

Is there an easier way for me?

Could you cook up something that runs on a retropie directly? ??

confused old man

Folly

@kiro

It pretty much working now after the update.
Nice thing the slow process of installing and compiling packages are skipped now ;-)

Although the bad formatted gamelist.xml will produce errors after closing emulationstation.
As you suggested I reformatted the gamelist.xml :

xmllint --format gamelist.xml > gamelist.xml.new

Later renaming gamelist.xml.new -> gamelist.xml again.
Now I have a good .xml and emulationstation gave me no errors after closing.
I would recommend to find a solution for this problem.

I will play with it some more.
Perhaps a good idea, for you, to make a new thread in "ideas and development" so we can discuss using/improving your scraper

Folly

@duglor

I could place the files on my google-drive for you.
Is that easy enough ?

kiro

@Folly Hi, It is strange that emulationstation is not closing properly with the unformatted XML, it is a proper xml after all, but I'll check on that.

BTW, the scraper works for every single system. There's a post here:

https://retropie.org.uk/forum/topic/32226/soft-new-scraper-in-the-works?_=1667982475849

Sure, google drive is ok if you want to send files.

I think I'll start working on a trimmed-down version specifically for retropie. The slow installation time is due to python libraries for GUI and SSH. (You can run it from windows and scrape a remote system for example), which is not necessarily needed on a retropie installation itself for example.

Again, Thanks! and please keep me posted if you keep on trying it, as I mentioned, it works for every single system, not only Archimedes :-)

Folly

@kiro

Just curious.
Is there a way to add stuff so more software is recognised ?

Folly

@kiro said in Where can I scrape Archimedes game pix and info?:

The slow installation time is due to python libraries for GUI and SSH.

Well, I think, if you know, that the process can take some time then it's basically NOT a problem.
As long as you don't recognise it as a freeze.

kiro

@Folly Hi, just fixed the formatting of the xml, so now it looks nice when editing (although it will not change the validity of the xml itself).

Regarding other systems, the scraper will read your emulation station systems file (usually located at /etc/emulationstation/es_systems.cfg)

As soon as you add more systems to your retropie via the retropie_setup script and ROMs, the scraper will automatically detect them.

Hope this helps!

Folly

@kiro

Aha a nice xml format, that's quick.
I will try it when I can !

I don't mean system but software.
Some archimedes files are not recognised, where does your script get the data and how can that be improved so more software of the archimedes is recognised.

Folly

@kiro said in Where can I scrape Archimedes game pix and info?:

@Folly Hi, just fixed the formatting of the xml, so now it looks nice when editing (although it will not change the validity of the xml itself).

I just tried and I got a pretty xml !
Though I get these errors from emulationstation :

mmal: mmal_vc_component_create: failed to create component 'vc.ril.hvs' (1:ENOMEM)
mmal: mmal_component_create_core: could not create component 'vc.ril.hvs' (1)
mmal: mmal_vc_component_create: failed to create component 'vc.ril.hvs' (1:ENOMEM)
mmal: mmal_component_create_core: could not create component 'vc.ril.hvs' (1)
mmal: mmal_vc_component_create: failed to create component 'vc.ril.hvs' (1:ENOMEM)
mmal: mmal_component_create_core: could not create component 'vc.ril.hvs' (1)
mmal: mmal_vc_component_create: failed to create component 'vc.ril.hvs' (1:ENOMEM)
mmal: mmal_component_create_core: could not create component 'vc.ril.hvs' (1)

I just tried also my "pretty" reformatted one which will give the same now.

Not sure why it reports these errors.

Edit :
I turned off the gamelist for archimedes, turns out this issue remains.
So it's probably not from your gamelist.xml after all.

kiro

@Folly Hi! Good to hear...

My scraper grabs information from various sources on the internet...but it also has a system of learning when something is missing and trying to update missing information whenever possible. The more people scraping the roms with my scraper the better this learning feature works. I also manually update the DB every so often, so keep your scraping coming :-)

duglor

This post is deleted!