[SOFT] New Scraper in the works

Clyde · 24 Feb 2022, 12:18

Take your time. I can't help you with the Windows version, but I may give the Linux version a try, although I'm not in any immediate need for scraping, and I'm used to and satisfied with Skyskraper. But cam certainly help you with some testing. 🙃

kiro · 24 Feb 2022, 13:47

@clyde Cool! Will let you know as soon as I update the Linux version!
Thanks!

arnaldchaves · 27 Feb 2022, 04:01

Perfect !

kiro · 3 Mar 2022, 08:28

Linux version is now available!!, please let me know if you test and see any issues (directory mapping is not yet working on any version, be aware)

Thanks!

Clyde · 4 Mar 2022, 14:05

@kiro Thanks, I will check it out. But please have patience, since I'm on vacation next week.

kiro · 4 Mar 2022, 14:35

@Clyde no worries, patience is one of the few things I've got plenty of 🤣🤣

kiro · 3 Apr 2022, 18:19

More news, I've added a 'Google translate' functionality, which lets you translate game descriptions to your language of choice (around 100 languages) ... give it a try and let me know your thoughts!
http://77.68.23.83/download.html

All versions (Linux/RPI/Windows) are up to date

kiro · 3 Aug 2022, 13:27

Some screenshots of scrapped and translated descriptions to different alphabets (Ukrainian, Chinese traditional, Greek & Japanese), all running in retropie (linux):

If the description in the desired language exists, it will use it, otherwise, it will translate it with google. Enjoy!

kiro · 21 Mar 2022, 16:29

A bit more of work this weekend, now you can download the game's bezels and you also have the option to put the default system bezel if the game's bezel is not available. This means that all games will show a bezel if run with a retroarch emulator. Also did some optimizations here and there and subdirectories are being scanned for ROMs.

If you want to help to translate the tool (it is already done by me in English, Spanish and French), drop me a message at retrsocraper[at]gmail.com and I'll set something up.

Get it here!

kiro · 2 Apr 2022, 13:40

I'm running out of ideas on what features I could add to the scraper... anyone has any ideas on what they would like to have in their ideal scraper? Thanks!

Clyde · 2 Apr 2022, 13:40

@kiro Hi, I haven't forgotten about you. I just didn't get around to occupy myself with it because of a "mild" COVID illness that nevertheless took me out for two weeks.

Just saying, stay tuned. 🙂

kiro · 4 Apr 2022, 07:25

@Clyde Hope you're ok! let me know if you have any issues testing the software! Thanks again!

kiro · 6 Apr 2022, 08:43

Hi @mitu, sorry to bother. How could I do to get my scraper into the retro-pie setup script? Is there any process that needs to be taken? or are you not accepting any more scrapers? Thanks!

mitu · 6 Apr 2022, 14:18

@kiro said in [SOFT] New Scraper in the works:

Is there any process that needs to be taken?

Any modifications to RetroPie are added via the Github project, at https://github.com/RetroPie/RetroPie-Setup, either directly (by project maintainers) or via Pull Requests.

are you not accepting any more scrapers? Thanks!

No, we're open to improvements. I would advise first to open topic in the forums with installation/operations instructions so users can test it and provide feedback, before submitting it to RetroPie-Setup. Also, while providing the source for your program is not mandatory, it would be useful for users to be able to compile on their own platform without relying on a distributed binary.

I tried downloading the program from your web page, but clicking on the download link opens a 'application/json' page filled with binary data - the download link is not working properly.

kiro · 6 Apr 2022, 14:26

Hi @mitu Thanks for the explanation, really appreciate it!!

I'll have a look on why you're getting that! Thanks! I've had others download without issues. Which browser/os are you using by the way?

I will provide a source for the scraper as soon as I feel it is ready to be distributed, still working on it.

mitu · 4 Jun 2022, 16:47

@kiro said in [SOFT] New Scraper in the works:

Which browser/os are you using by the way?

Firefox on Linux/Windows or plain old wget:

wget http://77.68.23.83/dist/retroscraper.rpi
--2022-04-06 15:45:37--  http://77.68.23.83/dist/retroscraper.rpi
Connecting to 77.68.23.83:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 194481976 (185M) [text/plain]
Saving to: ‘retroscraper.rpi’

retroscraper.rpi                           81%[==================================================================>                ] 150,53M  11,1MB/s    eta 4s

kiro · 4 Jun 2022, 17:35

@mitu and did you try executing the retroscraper.rpi file? (chmod 755 and ./) ?

Warning: if you execute with the --cli modifier it will start scraping directly!
It will not overwrite the gamelist.xml until each system is done. I need to implement a backup of gamelist.xml to prevent losing any manual changes that may have been done to them!

Thx!

mitu · 6 Apr 2022, 17:41

@kiro said in [SOFT] New Scraper in the works:

and did you try executing the retroscraper.rpi file ?

Yes, and I didn't get any message, just stands there doing something/nothing ?
I've tried with --cli and/or --help and nothing is displayed. I've just interrupted it from what it did, noticed it has created an - almost - empty retroscraper.cfg file.

kiro · 6 Apr 2022, 17:46

Thanks @mitu ... without --cli flag it will work only if you have a desktop (sdl2) available.... It is strange about the 'nothing happening' with the --cli... Are you trying on a machine that has the es_systems.cf in the usual place? (/etc/emulationstation)?

That it creates an empty cfg is normal upon first run.

Is your machine connected to the internet? I assume yes, but I have not seen anything hitting the backend, so it looks like it is not connecting?

I'll have a look...thanks again!

mitu · 4 Jun 2022, 18:51

without --cli flag it will work only if you have a desktop (sdl2) available....

I see no error message about not having a desktop env.

It is strange about the 'nothing happening' with the --cli... Are you trying on a machine that has the es_systems.cf in the usual place? (/etc/emulationstation)?

Yes, I have a standard RetroPie EmulationStation installed.

Is your machine connected to the internet? I assume yes, but I have not seen anything hitting the backend, so it looks like it is not connecting?

Well, yes, otherwise I wouldn't have been able to download it.

To be honest, I don't like the fact that's trying to connect somewhere without my choosing any options. Which 'backend' does it connect to ? I don't see anything mentioned on the download page.