Scraping Woes
-
Hi all,
Cabinet with Pi 3+ running great.
Now, I'd like to make it look nice with snapshots and maybe videos.
I first ran the internal scraper in ES. I got lots of textual metadata and things like marquees and cover art. Then I installed Skyscraper and ran against Mame and Intellivision - worked GREAT and got videos for IV and mame, but it only went through a few roms in mame arcade before quitting. So, I tried running again and later on another source db within Skyscraper (outside of ES of course). now, I've lost the videos (I've also manually imported a bunch of of snapshots with skyscraper and that worked). But now even though all the Skyscraper scrapes say green YES to videos there's no metadata video entry to point to the videos (I do generate the gameslist afterward). Also when I look in Skyscraper's video folder, I have LOTS of mp4 files, but their names are like GUIDs (dsfkldsflksdj;od....dfwfsgsdg.mp4).
Any ideas why continued scraping doesn't give me videos? (I've tried disabling the OMX player in ES). -
...but it only went through a few roms in mame arcade before quitting.
All of the scraping sources have monthly or daily limits. Some can be heightened by registering with their services. Read more about each of the source here.
...now, I've lost the videos
I'm pretty sure you haven't, they are still in the cache, unless you chose to purge the platform through the script.
But now even though all the Skyscraper scrapes say green YES to videos there's no metadata video entry to point to the videos
Do you have videos enabled in the script before generating the gamelists? Also remember that you have to select all platforms with space before starting the generation. If you did those things, you must've somehow disabled it somewhere else. If videos are enabled and you re-generate the gamelists, Skyscraper will export the videos to the expected folder and add a
<video>entry for them in the gamelist.xml. -
@muldjord said in Scraping Woes:
d you re-generate the gamelist
OK - thanks I 'm pretty sure I did - but I'll try again.
I need to do more reading since I don't quite get how Skyscraper generating the Gamelist.xml works with other scraped content that is already there (from other sources). i.e. does having Skyscraper generating the gameslist.xml overwrite/ignore other scraped content? It doesn't seem to since I have a mix now of flyers and snaps...hmm now that I think about it - I'm not sure ANY Skyscraper new content is getting in there...
Does it make sense those mp4 filenames being GUID-like?
-
Does it make sense those mp4 filenames being GUID-like?
Yes, those are the cached videos, and you shouldn't touch them. They will be copied automatically when you regenerate the gamelists.
hmm now that I think about it - I'm not sure ANY Skyscraper new content is getting in there...
Then that's probably your problem. I'm guessing you never actually generated the gamelists. Remember that you need to select the platforms with space.
I need to do more reading since I don't quite get how Skyscraper generating the Gamelist.xml works with other scraped content that is already there (from other sources). i.e. does having Skyscraper generating the gameslist.xml overwrite/ignore other scraped content?
Skyscraper will overwrite any data you have scraped with other scrapers, unless you import it into Skyscraper with the
esgamelistmodule before generating the gamelists with Skyscraper. -
OK - I figured ES was finding the other gamelist.xml file(s) before Skyscrapers, and yes that's what was happening.
I generated the game list and took note of where SkyScraper put it (which one). Then I renamed the xml file that was in the roms folder to something else so ES wouldn't find it - IT WORKED. I have videos!!!!
But the game list is by rom name and not the nice title...so I have to read on how to change that.
I figure it's possible I have to change a cfg and re-generate the gameslist.xml - so I'm hoping that's possible without having to rescrape everything...!
Thanks!
John -
@cousinsfan Yes, you can change any config and just re-generate the gamelist. You don't have to scrape everything again, that's the whole point of having the Skyscraper cache. :)
-
Awesome - thanks muldjord - forcefilename is the only thing I can think to change...would that work?
[EDIT] OK this is weird - I went to the basement to check on the machine (still running ES overnight) and now all the names are the right names - not the romnames - meaning....did it take time to "read" the names from the xml and this took time (overnight - but probably quicker of course)??
-
and now all the names are the right names - not the romnames - meaning....did it take time to "read" the names from the xml and this took time (overnight - but probably quicker of course)??
No, it's instant. Maybe you had two different gamelist.xml's and deleted one and didn't realize the other had the correct names?
forcefilename is the only thing I can think to change...would that work?
Yes. :)
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.