[SOFT] New Scraper in the works

kiro

@Folly maybe have a look at the logs? I know this could happen due to permission issues, sometimes... it should ouptut something like :

CANNOT CREATE DIRECTORY [Your/directory] - ERROR [Explanation] - PLS VERIFY AND TRY AGAIN

And then exit ... so strange is not exiting...have you got he latest version?

Folly

@kiro said in [SOFT] New Scraper in the works:

so strange is not exiting...have you got he latest version?

It does exit now always, so no problem.

I checked the --mediadir option again with absolute path.
The issue is that when using a path with the folder, for example, RetroPie (R and P are uppercase) RetroPie isn't used/created but retropie is. (everything with lowercase)

I see you have fixed the other issues ;-)

kiro

@Folly About the folders, didn't quite understand, so if you put /RetroPie/ will it create /retropie/ or you put /retropie/ and it will not be recognized as /RetroPie/ ? The latter being totally ok as directories under unix/linux are case sensitive..

Folly

@kiro said in [SOFT] New Scraper in the works:

if you put /RetroPie/ will it create /retropie/

This.

kiro

@Folly Ok, then probably I'm putting everything lowercase where I should not, going to check right now..

Should be fixed and uploaded to git...

Folly

@kiro

I get this :

python3 /opt/retropie/supplementary/retroscraper/retroscraper.py --mediadir /home/pi/RetroPie/roms/upright --systems upright --cleanmedia
Starting scraping
Scanning Files
Scanning Directory
ERROR SYSTEM upright: NO ROMS IN DIRECTORY /home/pi/RetroPie/roms/upright/
Scan Done!
SCRAPPING ENDED --- Thank you for using retroscraper!!

kiro

@Folly said in [SOFT] New Scraper in the works:

NO ROMS IN DIRECTORY

Ok, should be fixed, interesting. To avoid scanning through images, I remove anything whose path is the same as the 'media' path, of course, if the mediapath is the same as the system path, well guess what.... I just added the system name to the media path so it should be ok now.

Folly

@kiro
Did a second update !
I works ok.

kiro

@Folly Thanks for that, would've never thought of doing that path combination!

Folly

@kiro

So you like it ?

kiro

@Folly I like the testing being done, usually when developing there are so many things you give by 'understood' that you tend to forget some of the possibilities a user may input...

Folly

@kiro

It's a good process,
It's more fun/productive when there is some interaction, don't you agree ?

Btw.
How do we add media that is not recognized right now for our project ?

kiro

@Folly Well, here's the difficult question... how do we do that.. ok there are 2 things to consider:

1- I'll have to manually add the images to the DB, that is not really a concern as long as I understand to which game each image corresponds, which should be fairly easy (thanks to filenames).

2-What do we do if there's already a screenshot available for the game in question, for example, Archimedes. (although I see that most of your images correspond rather to 'systems' than games? And some images seem to be fairly similar (game&watch for instance)

So, if you want to make a small test, pack all the all_in_1 images, which I see are missing from my backend, into a file and I'll try to put them in the right place, once done, we can try to see if scraping gets these images. makes sense?

Folly

@kiro
I see your point.
Perhaps we can start with an easy one, tigerrz (Tiger R-Zone) which has just 3 games.
I scraped it earlier with your scraper and, as far as I know there are no marquees, images and videos in your DB.

kiro

@Folly Let's try it :-)

Folly

@kiro

Here is the googledrive of DTEAM with predefined gamelists, which is used by our project :
https://drive.google.com/drive/folders/1f_jXMG0XMBdyOOBpz8CHM6AFj9vC1R6m

You can just download the file tigerrz.

I just tested my files with your scraper and these have no media and no description and full name etc :
rzbatfor.zip rzindy500.zip rztoshden.zip

Don't know if .7z files have media, etc in your DB.

kiro

@Folly thx!

But, when I try to download (I guess right click then download on folder), I get

Download failed...do I need permissions or am I not downloading what I'm supposed to be downloading?

Folly

@kiro

I just tested, and it works here so it should work for you too.
You should just click on the tigerrz and a zip file is created and downloaded.

I recently had the same problem working on an open network, though I think you are not on an open network now, right.

kiro

@Folly Not sure, I tried an incognito window and it seems to work... maybe because since I'm logged in with my Gmail account to chrome it somehow messes things up? not sure...Anyway, It's a bit on the late side so I'll be importing the images tomorrow first thing and will let you know as soon as I'm done. I see there's a screenshot, a marquee, and a video (almost for every game) correct?

Folly

@kiro

Aha, so it's working now.
Yes indeed, all should be there.

Good evening !