[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture

vbs

@hansolo77 said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

But you don't get ANYTHING until you donate at least 5 bucks, and that's only 1 thread. 10 bucks for 5. The thing is, when I looked into it using google translate, it looks like it's a monthly subscription, not a one time donation. It's a support feature, not a "thank you".

Nah thats definitely not true: When you contribute to the site then you get more threads for free (without paying). After I assigned about 40 ROMs to games I had 4 threads. By now I assigned like ~300 ROMs and I have 5 threads. So I am quite sure you don't have to pay monthly to get more threads (that would be really uncool).
Assigning like 20 ROMs does not take more than 15 minutes I guess.

vbs

@hansolo77
I found the rules what you have to do for more threads:

http://screenscraper.fr/webapi.php?alpha=0&numpage=0

Combien puis-je gagner de « Threads » ?
Votre nombre de Thread dépend de votre participation :
(ssuser/uploadsysteme + ssuser/uploadinfos + ssuser/romasso + ssuser/uploadmedia)
-	1 participations validées : 2 Threads maximum ouvert simultanément
-	2 à 49 participations validées : 3 Threads maximum ouvert simultanément
-	50 à 199 participations validées : 4 Threads maximum ouvert simultanément
-	200 à 499 participations validées : 5 Threads maximum ouvert simultanément
-	500 à 749 participations validées : 6 Threads maximum ouvert simultanément
-	750 à 999 participations validées : 7 Threads maximum ouvert simultanément
-	+ de 999 participations validées : 8 Threads maximum ouvert simultanément

Et si vous faite une participation financière vous gagnez des « Threads Bonus »
(ssuser/contribution) : 2 = 1 Thread Supplémentaire / 3 et + = 5 Threads Supplémentaires
-	De 5 à 9 euros : 1 Thread bonus 
-	Au-delà de 10 euros : 5 Threads bonus

Well, it seems I lied: you need 50 contributions for 4 threads (but only 2 for 3 threads).

hansolo77

How do you assign roms though? I have a hard time using that website because of the French.

Also it looks like just using RetroPie (mix) is working. Only problem now, my 1 hour scrape on Atari ST is now going to take 10 Days, and it only on Rom #8. :(

hansolo77

I think I might have figured out how to assign ROMs to entries in their database. Sometimes it says something like "For instance, can you associate this ROM to it's game name in the above list" and shows a list of games to pick from. I've done that a bunch of times. Then it asks for something like a description and website where I found it. I provided that for a bunch of games too. Each time, the page refreshes saying my input has been saved or something, but I don't seem to be getting any credit for it. (and yes, I'm logged in). Am I doing something wrong? Do I need to be actually in a the system, and not just entering data from the homepage?

mattrixk

@screech: Cheers for the awesome scraper, the Standard (3img) makes my system look great.

Now a question. I want to make my own MIX Image Profile template (so I can create a rockin new theme for myself), but I'm having problems.
(Sorry, this got a bit long)

Here's my process:

Go to the Mix folder that is created when you first run Universal_XML_Scraper64.exe.
Copy Standard (3img).zip.
Paste Standard (3img).zip into a test folder on my desktop.
Unzip Standard (3img).zip within the test folder.
Duplicate the Standard (3img) folder and rename the duplicate Standard (test).
Open the config.xml within the Standard (test) folder.
Change <Profil Name="Standard (3img)"> to <Profil Name="Standard (test)">.
Save config.xml.
Zip the Standard (test) folder as Standard (test).zip.
Copy Standard (test).zip back into the Mix folder.
Run Universal XML Scraper.

Now, that's a few steps, but it's pretty straight forward. I've just renamed an existing Image Profile that I know works. I would expect it to show up in the profile list with it's own name, but with all the details of the (3img) profile.

3img
"Standard (3img)" looks like this. Note: "Standard (test)" is also in the list

Unfortunately, this is what I end up being shown:
test
"Standard (test)" with missing Author and Description data. Note: the example images shown are just a leftover from the last option selected. If I chose "Arcade (moon)" and then went back to "Standard (test)", it would continue to show the "Arcade (moon)" example images.

I also get a bunch of errors showing up in the log:
PasteBin link

If I run the Scraper with this Image Profile selected, it doesn't generate an image in the downloaded_images folder. It does generate the gamelist.xml, except it is missing the <image> tag.

I've tried using the tutorial in the template, but it doesn't seem to match up with the config.xml file in the MIX Image Profile templates. (eg: It's missing any info on the <compression>, <root> and <game> tags. The <element> tags seem to be vaguely similar, but still quite different and the tutorial refers to a config.ini file.)

Any ideas what I might be doing wrong?

paradadf

@hansolo77 I've seen your contributions on SS and the they haven't got validated (yet?) because they are not neccesarly of good quality :S. Some descriptions have ??? signs everywhere, other are not good at all (relatively speaking), and so on. SS seeks for good quality data and some standards must be kept. The moderators try to correct all submited data/media but that is a lots of work. So please, try not to submit descriptions like "This is an action game developed by X, published by Y and released on 19XX.".

screech

@hansolo77
First of all : Everything is free, we are working (UXS and Screenscraper staff like retropie staff) only for you and for free. Donation is just rewarded by some few threads to thanks guys who help us to pay for a new dedicated server more powerfull, to increase ressources dedicated to users... (actually only one person, MarbleMad, pay for all technical infrastructure). You can use Software and database for free...

Grabbing datas is a hard job by the quantity of datas and medias needed to have a good database. And we aren't so much to work on it (about 10 guys working hard on it + some "small" contributor).
With more than 2.000.000 API request by day (more than 300 unique IP who scrape by day) we just want a little "help", not so much time.
Like @vbs say : 1 validated contribution and you have 2 threads, 2 validated contributions and you have 3 threads... If every User only contribute 2 times a day (less than 1 minute) we will have more than 600 new datas by day to have a great DB for all of us... And I can say we are really far from that...

Just a little translation to what vbs say :

0 contribution : 1 Thread MAX
1 validated contribution : 2 Thread MAX
2-49 validated contributions : 3 Thread MAX
50-199 validated contributions : 4 Thread MAX
200-499 validated contributions : 5 Thread MAX
500-749 validated contributions : 6 Thread MAX
750-999 validated contributions : 7 Thread MAX
more than 999 validated contributions : 8 Thread MAX

If you want to help us pay server

from 5 to 9€/$ : + 1 Bonus thread
more than 10€/$ : + 5 Bonus thread

It's just a gift and a "thank you thing" to those who helping us. Not a "I buy thread thing", DON'T BUY thread, donate only if you want (and can) donate...
We prefere you participate to have a big and nice DB by submitting new datas and medias. (that's why "award" are easier and bigger on DB participation)

So, now how it's work :
We don't want to have crap in the DB, so every submission is validated 1 by 1 by admin or moderator. It take a lot's of time to check every contribution but it's a "quality proof" of the DB.
When new submission are validated, you must wait near midnight (French hour) to gain your new threads (Server do is job calculating Stats).

Just check some of your submission :
Hyper Black Bass '95 on gameboy is not the same game as Black Bass: Lure Fishing So I can't validate your submission.
Soccer Manager on GBC is not the same game as Player Manager 2001 So I can't validate your submission.
Sorry we don't take synopsis like "A one player game published and developed by XXX in YYYY." It as no value (all info are already in the DB, synopsis we want are "real" synopsis)
....

Don't forget you can clic on the small flag (upper left corner) to change the website language.

@mattrixk
I need to update the Wiki (info are outdated and are for the V1 :S sorry)
Can you send me your new "zip" so I can check what's going on ?
As I saw in the log you pastebin, there is a problem with XML (I think a wrong tag or something like that)

hansolo77

Thanks for the feedback and information. I was also noticing a lot of the information I was providing was a little low on descriptions. I figured anything was better than nothing, and the site I found the descriptions at just had that information. I'll avoid posting in simple 1 sentence descriptions in the future. I agree, having a nice full database is the better way to go. As for the non-matching games.. I found through searching for those games that they are just renamed versions of the same game only in a different region. Since the list it provided me had the original region's version, I just linked it to that. I don't know how to ADD a new game to show it's an alternate region clone.

But yes, thanks again for the helpful feedback. I'll do better. :)

vbs

One question please: The database improves every day so what do I have to do to re-scrape my system? Is it sufficient to just scrape a system again and everything will be updated?
I know that the gamelist.xml will be recreated from scratch but what about the images? Will they be re-generated from the latest data or do I first have to delete them manually?

vbs

@vbs
Well, I justed tested and it seems the images get regenerated automatically. So no need to delete manually.

mattrixk

@screech said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

Can you send me your new "zip" so I can check what's going on ?

I've put it in my dropbox here.

there is a problem with XML (I think a wrong tag or something like that)

I made a copy of the existing Standard (3img) and literally the only thing I changed was the <Profil> name to match the folder name.

hansolo77

@vbs I was wondering this as well. There is an option to UPDATE, but according to the program it looks like it just adds new ROMs... it doesn't actually UPDATE the metadata/art. I think a feature that should be added would be some extra data in an xml or something that identifies what data was scraped. Then the next time you run the scrape, it compares what's online with what it already has. That will cut down on all the re-creating identical data, wasting a lot of time. Things you could get would be all the metadata fields (and have the "found.xml" file record with 1's and 0's (like gamename/publisher=1 then gamename/description=0 and it'll skip adding publisher data but get the description if it exists), and the artwork (if using MIX+3 or MIX+4, just have it again record 1's and 0's if it has it/doesn't). After comparing, it would get the new data, update the gamelist.xml as needed, and recreate any new artwork. I could probably write out all that in BASIC (the only programming language I know lol). So I can see this easily being implemented into the UXS program.

paradadf

I believe, without real understanding of how UXS works, that pulling data from the server doesn't that any considerable amount of time, but creating the mix images. I doubt that comparing anything with the db will be faster than just downloading the whole data.

hansolo77

My biggest slowdown when scraping is all the hash-checking it does. There should at least be a file that stores all the hash numbers so it doesn't have to re-hash every time.

paradadf

I don't know how big your roms are (what system), but hashing a regular (not cd based) rom doesn't take any longer than 0,1 s per file.

hansolo77

I'm having a really hard time with this now.
(refer to this post)
I think UXS has some how corrupted all my gamelist.xml files. Probably because yesterday I was having trouble getting the new version to work with the correct paths. I suspect it has multiple copies of them somewhere, and it's throwing everything out of whack.

But yea, my Amstrad CPC, Atari 800, and Atari St systems all took upwards of 4 hours each to hash, and they're no bigger than 1.2mb at the most (typical 3.5in floppy).

vbs

@hansolo77 said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:

I'm having a really hard time with this now.
(refer to this post)
I think UXS has some how corrupted all my gamelist.xml files. Probably because yesterday I was having trouble getting the new version to work with the correct paths. I suspect it has multiple copies of them somewhere, and it's throwing everything out of whack

Make sure to delete all the partially created gamelist files in all the Rom folders!

paradadf

I'd put the launcher on a new folder and try again. Do you scrape over the network, right? To be honest, I've never done than because the pi is too slow imo. I'm scraping ST right now on my pc with only one thread and with the 3img profil is taking about 30 min for 305 roms.
My results:
alt text

hansolo77

I need to either find an alternative rom set or go through and erase a lot of junk ROMs. My AtariST set has 3260 files. I got this set from my GameBase collection on my PC. It took 4 hours to scrape yesterday.

Rion

@hansolo77 I think it would be best to Scrape your collection on you PC and then copy over the images to your pi.

So make a copy of your roms folder on to the pc.

Or you could like i do run everything of my USB stick. Rom folder only that is.