[SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture
-
I've narrowed down the problem to being related to the
2.2.0.2version of the scraper. Appears to be some kind of memory leak. Funny thing is, when it crashes, it actually keepsscraper.exerunning in your task manager. I had 4 copies of it running earlier. Then I saw @screech mention in another thread about the version .3 coming with new MIX profiles, and that .2 was "really" buggy. So I gave it a try and was able to get a full scrape with2.2.01. Downgrade to the2.2.0.1version and you should be good.@screech - Can I make an observed suggestion? When you scrape ROMs that aren't in the server's database, and you have it set to "Add missing roms in XML", the gamelist.xml file creates game names that are not helpful at all.. meaning:
<name> (Clone, Beta, Demo, Proto, Trad, Hack, Unl)</name>I actually get a list with about 30 of these, and because of the parentheses these are at the top of all my lists inside EmulationStation. Can you change this so it creates the name as simply the ROMFILE instead?
-
I have just tried this scraper for the first time, and I'm having the following issues:
- I have 40 games in my FBA folder, but the program only recognizes 32.
- It then says that it's scraping all 32 with succes, but only 5-6 of these show up with images in ES.
- When scrolling through the remaining games, the system starts to become very slow, essentially crashes.
It seems there's PNG's made for all the recognized 32 games in the downloaded_images folder, though...
-
Hi guys, the 2.2.0.3 is out with correction :
2.2.0.3
Corrected :- Random Error making UXS crash. (Changing a part of MAIN <-> ENGINE communication system)
- Changing color progressbar is now only green (Maybe a raison of random crash)
SSH finally will work (sorry for that) - Apostrophes in filename now works... (Thanks Marblemad)
Added :
- ProgressBar when creating the XML File
(And of course correction of the 2.2.0.2 like the missing rom name ;) )
-
SSH kill of EmulationStation still doesn't work.
-
I haven't Retropie installed :( can someone give me a kill command to kill ES from SSH ?
I use "killall emulationstation" can someone test it from putty for exemple ?
It works on recalbox no reason it don't works on retropie :(
-
I just tried something else, and it is working... Apparently if you have it set to create the gamelist.xml locally (on the computer) then it doesn't kill EmulationStation. If you have it create on the pi, then it does. It wasn't even prompting before.
-
Yesterday I scraped some roms to test and I choice the video download in YES. For some systems, how FBA and Master Systems the videos run very well, but for SNES, for example, the videos don't run well on my Raspberry.
I think that the problem can be the power supply. When I surf at SNES roms, show every time a yellow ray on right top corner TV.
-
@screech Thanks for the update. I have done some testing and can confirm that SSH now works for me, and I still haven't had another crash either.
Wondering if you (or anyone else) can help me figure out what i'm doing wrong with my autoconfig path, though--I still can't get it to save images to my Pi, it just says it detected a problem with the autoconfiguration path. I tried resetting it to default but still doesn't work.
Here is a link to what it looks like:
https://ibb.co/bG6ZtFRight now it's just saving it to my C:\configs folder, which is fine I suppose (can still move it to the Pi manually via FTP) but I'm sure I'm doing something wrong.
Edit: I should say that I have not changed any default settings for rom, imge or XML location; roms are in RetroPie/roms and the gameslist and download_images folders are on my Pi in /opt/retropie/configs/all/emulationstation
-
Hi @screech, I know it was over 2 weeks ago that you gave me advice about scraping NeoGeo, but I finally got around to testing it properly. What I ended up doing was scraping NeoGeo normally, which found ~60 out of ~120 games, then I turned scrape mode to "update xml", left research mode on "crc+filename" and turned off "auto-select your system". Then when I went to scrape, I selected Mame instead of NeoGeo and it found the rest of the games.
-
Am I right in thinking that the XML scraper connects to screenscraper.fr on startup? I finally got all my problems sorted out last night and scraped 4 of my systems, then went to bed and when I get up this morning the program is REALLY slow to load and the picture of the system I choose to scrape isn't showing on the bottom of the window, it takes a long time to test the server and then won't scrape.
I can ping the site using cmd and that works perfect, but when I try to go there in a web browser like chrome it takes along time to load if it loads at all. I'm not sure if it's my connection or the site but I was more worried about the XML scraper itself. Obviously it has trouble scraping because of the slow connection, but would that make it slow to start as well or is that a program issue?
-
I tried to use this scraper tonight with a 4-Mix and 3 threads configured to work with no connection to the Pi3 (ie local folders only) and it took over two hours to do 100 games. At first, it was pretty fast, but after a few minutes it just slowed to a painful crawl.
I noticed in the logs that it was waiting a lot.
Also, there were a number of times where the game's Wheel Image wasn't applied and others where all I received was a corrupt image despite assets being available on the scraper site. I assume the application just timed out. I am in Florida.
Anyone have any suggestions or could it indeed just be the distance?
-
@RedDog said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
I tried to use this scraper tonight with a 4-Mix and 3 threads configured to work with no connection to the Pi3 (ie local folders only) and it took over two hours to do 100 games. At first, it was pretty fast, but after a few minutes it just slowed to a painful crawl.
I noticed in the logs that it was waiting a lot.
Also, there were a number of times where the game's Wheel Image wasn't applied and others where all I received was a corrupt image despite assets being available on the scraper site. I assume the application just timed out. I am in Florida.
Anyone have any suggestions or could it indeed just be the distance?
So, after watching the Windows Task Manager during a few scraping sessions, I found that Chrome was stealing a lot more RAM than I realized. After closing that down, the scraping improved, but there are still problems. When starting a session, the imaging is very quick at about one image a second, but after the first 15 or 20, it starts to slow down at an exponential rate. I found that after 10 minutes or so, it was better to cancel the process and start it again.
I also found what I assume is an issue with the scraper. I have my Mix set to the 4 image. If I set it to 'Picture Only' in the General Configuration, the scraper folders will show the program creating the 4-mix image, but then it only brings the background image to the destination folder and not the final 4-mix image.
Another issue I notices is that occassionally, the thread number in the General Configuration will change from my setting (3) to 1 or empty. When empty, it appears to act as if it were a 1.
Also, does anyone know why final images might be missing Wheel images if one exists on the screenscraper.fr site? Maybe an incompatibility with the image or a time out? For example: Gain Ground for megadrive. If you try to scrape a 4-mix, the resulting image will have a Box, a Cartridge, and Screenshot, but no Wheel image...and there is a Wheel image on the site.
-
Sup @RedDog? :) Fellow GameEx'er? I don't think the mix will create with a wheel image. The 4-mix is a combination of ScreenShot, Box, Cart, and the "marquee" game name at the right. If you have "Wheel" selected as an option, I believe it downloads that as a separate file like "filename-wheel.jpg".
@screech I think your xml generator is still a bit buggy. I'm going to try and scrape again, but I noticed tonight that the scraped gamelist.xml it created for Nintendo FDS created a bunch of unnecessary blank
<image/>lines. Please see here:
https://retropie.org.uk/forum/post/84974
Essentially, I fixed the problem I was having by removing those lines from my xml. But it shouldn't have created them in the first place. Can you check into it? :) -
@hansolo77 said in [SOFT] Universal XML Scraper V2 - Easy Scrape with High Quality picture:
Sup @RedDog? :) Fellow GameEx'er? I don't think the mix will create with a wheel image. The 4-mix is a combination of ScreenShot, Box, Cart, and the "marquee" game name at the right. If you have "Wheel" selected as an option, I believe it downloads that as a separate file like "filename-wheel.jpg".
Hey Han. Yup, it's me. Thanks for the tip on the Marquee.
-
@screech I should also mention that every scrape I have done for PSP has come up with no data and empty image files. This 15 items not found. Is it possible that the scraper isn't connecting up to the PSP area on the site?
Also, may I suggest that if in image is not found, that a placeholder image be copied over instead of an empty one, perhaps something that says "Image No Found". For some time, I assumed that some of the images were being corrupted when they were built, so I kept running them over and over hoping for a successful build. Then I finally realized that they were files that could not find a match.
-
I love the idea of the image not found idea. It would make it easier for people (cough, me) to get there own images off the net as well. Just rename your image to the created one and the xml file already points to it so you don't need to edit it.
My problem is It found all of my PSX games and got images but none of them show in ES at all. The games are in folders to keep them tidy so I took them out and the images still don't show. I ended up using the built in scraper to grab pictures and they work fine for now, so it's all good.
ALMOST all of my games are scraped now, ended up getting no-intro sets as they just work and used the built in scraper to grab my last few Amiga and Atari 2600 ones. Just missing a few that don't seem to be in any databases. I might try my hand at submitting some games to thegames.db to fill in the gaps
-
@RedDog: UXS only scrapes about 1/3 of my PSP titles.
@screech: I love your work mate, as always. I had an idea though: the built-in ES scraper gives you the option to verify which game is being scraped, or to pick one from a list. Do you think it would be viable to include an option like this in UXS?
My PSP and PSX systems both use .pbp files, so I imagine the scraper is going off file names. However, lots of the those file names have the colon
":"character, which windows won't let you put in filenames, so the scraper skips over them. It would be great if the scraper would pause and then give you a list to choose from. Even if it was just a list of every game in the system, rather than UXS trying to work out which games would be the most likely.Obviously this wouldn't be default behaviour, but maybe a checkbox in the General Settings. Maybe UXS would scrape all the ones it can automatically, then at the end it gives you the option to manually select the games it couldn't find on it's own.
-
@hansolo77
SSH : Normally, it works now (several confirmation from different user) so if it doesn't work on your computer, there is something else (try to use putty to test). And it's normal that when you scrape locally, it don't ask to kill es...
Blank <image/> : I need to make some test.. don't know why it do this (on for now, you are the only one with this issue I know)@brunocedup
Video quality : It's depending on what is on the DB... I haven't check every video ^^. But normally they are in the "original" resolution (SNES have a bigger resolution than Master system.. it may be a reason why)@ghostlywindmill
Problem with autoconf : Can you check you can access the different path from your samba Share with your computer ? it's like UXS can't access in these folder. You can try the alternate path (in rom folder) they're is a new option for that in advanced configuration menu.@mattrixk
Neogeo rom in Mame : That's not a good news :( it mean they're some neogeo rom in the Mame system on Screenscraper DB... I need to do something on that ;)
UXS improvement : Sorry, for now it's impossible. The API can match only on CRC or on filename... No other choice (for now)@A2ra3L
micro deconnection ? Don't know why it's "very long"... what system are you scraping (a CD one ? so it's about normal since the ISO/BIN are very big to hash).
One thing : At about midnight (french hour) the server do a lot's of stuff and can be very slow for about 10/20 minutes...
PSX without picture : When you scrape PSX, your Emulationstation were killed ? Can you check your psx gamelist.xml and the image picture ?.
And FYI, UXS don't use thegamdb but ScreenScraper DB ;) so if you enjoy UXS result and want to fill the missing you are welcome in Screenscraper Too ^^@RedDog
Same as above, maybe it's the "midnight" stuff that break your scrape ?
For wheels, we still have a lot's of work to associated these we have to the good country rom (exemple : a wheel is badly referenced as "Japan" and the rom is "US" so the wheel may not be downloaded, just check the fallback in general menu)
PSP : never tryed (I haven't PSP rom ^^) I need to make some test ;) (But for now, the PSP DB isn't so "big" on ScreenScraper, we need some contributor on this system ;) -
It started working again after a while. Must have been my connection or the website was busy. But the PSX games confused me. The es was killed and the xml file for the psx games showed the same filenames and locations as the png images so they should have worked. All my other systems worked fine. The strange thing is the psx images had doubled names. For example ape escapes png file was called ape escape_ape escape.png only the psx had this issue.
-
@RedDog
Same as above, maybe it's the "midnight" stuff that break your scrape ?
For wheels, we still have a lot's of work to associated these we have to the good country rom (exemple : a wheel is badly referenced as "Japan" and the rom is "US" so the wheel may not be downloaded, just check the fallback in general menu)
PSP : never tryed (I haven't PSP rom ^^) I need to make some test ;) (But for now, the PSP DB isn't so "big" on ScreenScraper, we need some contributor on this system ;)@screech I assume the 'midnight stuff' has to do with server maintenance or some such thing around midnight CEST. I am in Florida (USA) and we are 6 hours behind CEST. The thing is that for the most part, I have done the majority of my scrape tests in the later evening which puts the times after midnight CEST. I had been fighting with it for several days. However, last night around 2:00am to 3:00am CEST they worked fantasticly. I do not think I did anything to make a difference, but I had two 700 item sets finish with a 4-Mix at about 10 minutes each.
As for PSP, I have to disagree with you. While the PSP database may need some lovin, it does not explain why some of my titles that do exist on the web site will not pull. Let me give an example:
SSX on Tour.iso returns an empty image. When I look up the game on Screenscraper.fr:- The Name of the Game is "SSX on Tour".
- The Game Name (by Region) is "SSX on Tour" for USA.
- A Publisher, Developer, Rating, Synopsys (English), Screenshot, and Logo exists.
I don't think the matching criteria could get much better than this, yet I still get no Metadata or images.
I also wanted to say that you've got a really great little program here and it has a lot of potential for this hobby. I hope this all comes over as constructive feedback and not negative. :) Keep up the good work. I know a lot of people appreciate what you have put together.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.