Scraping multi disc roms

HurricaneFan

@sano HA, I just tried this method. EmulationStation crashes to the command prompt with:

terminate called after throwing an instance of 'std::out_of_range'
what():  map::at
Aborted

Can't type anything at the command prompt after this happens. Had to hard reboot my pi.

Sano

You pressed select on the folder, then edit game metadata, and scrape ?
I'll try to reproduce.
@pjft are you aware of this ?

Edit : I wasn't able to reproduce the crash. Could you tell us more about your setup ?
(ES and retropie-setup versions, last update, ... see https://retropie.org.uk/forum/topic/3/read-this-first )

HurricaneFan

@sano I am running Retropie version 4.2.10, theme is Carbon.

I did exactly what you stated, select > edit metadata > Click Save and poof it crashes.

It's repeatable on my build.

pjft

Not aware. Might be easier to diagnose if you share details on the file formats, the games at hand and gamelist content for those systems you're scraping. When exactly does it crash though? Are you connected to the internet?

A short video would also help I imagine - it can be a crash in the scraper, in rendering, in saving...

Edit: what EmulationStation version, not that I'd imagine would make much difference?

mediamogul

@pjft

Just tried this on a folder for the Apple II system on my 4.2.10 setup using the Carbon theme and got the same result. This might also be as good an opportunity as any to mention a minor cosmetic issue in Emulation Station regarding the verbiage of scraping. Currently it reads "Scrape Metadata for this Game". Seeing as how scraping can be very useful for folders and visible shell scripts, would it not be better for it to read, "Scrape Metadata for this Item"?

pjft

@mediamogul same result as in crashing?

It could be renamed. To be honest I'm not sure quite familiar with the scraper - both in terms of code as well as in terms of what it's supposed to do. Meaning: I'm not sure it's supposed to work on folders (notwithstanding the fact that it may or may not effectively work).

mediamogul

@pjft said in Scraping multi disc roms:

same result as in crashing?

Yes indeed.

I'm not sure it's supposed to work on folders

It could have always been a happy accident, but until recently, it worked very well.

pjft

@mediamogul happy accidents are the best kind of accidents, though let's assume it's supposed to work.

Can you describe to me the folder structure, content of root and sub folders you're trying to scrape?

As I mentioned, a video may help.

I likely won't pick this up anytime soon, but can add it to the backlog.

Ideally/alternatively I'd certainly recommend creating a bug, for tracking, especially after we are able to narrow down the cause.

Thanks.

mediamogul

@pjft

Can you describe to me the folder structure, content of root and sub folders you're trying to scrape?

Sure. This particular example is about as basic as it gets. There's one folder and two items inside.

/The Oregon Trail/
Oregon Trail (Disk 1 of 2).dsk
Oregon Trail (Disk 1 of 2).dsk

Ideally/alternatively I'd certainly recommend creating a bug, for tracking, especially after we are able to narrow down the cause.

I'll go ahead and do that now.

HurricaneFan

Yeah, My scenario was I had a folder inside the SegaCD folder and 2 bin and 2 cue files in that folder. When I try to scrape the folder, it finds the data and image for the game. But when you hit the save icon. Hard crash to command line with no way of using your keyboard any more.

clueless

Hi, sorry to necro but this is the only relevant thread I found.
Did you ever find a solution to this problem? The same thing happens to me when trying to save metadata/scrape to folders. It's annoying because for some reason the folder is named wrong in ES and trying to change it manually kicks me out of ES when I click save, requiring me to pull the power plug as no input will work afterwards.

HurricaneFan

@clueless An issue was filed but its still a bug.

https://github.com/RetroPie/EmulationStation/issues/186

clueless

@hurricanefan thank you

mediamogul

Rather than clutter the Github issue with potential theories, I thought I might float one here first. This issue seems to coincide with the addition of the 'Favorites' feature, dating around the last few weeks of last July. I've noticed that you cannot add a folder as a favorite, but tonight I got curious to see what would happen if I manually added the 'favorite' MD tag to a folder.

Anticlimactically, ES launched without so much as an error, while simply ignoring the folder in the 'favorites' list. However, the folder was in fact showing as being a favorite in the MD inspector. I'm wondering if the fact that folders are not fully accounted for as being a valid option for the favorite MD tag is somehow related to the "out of range" instance that's being thrown.

pjft

@mediamogul Sure, I'll bite since you're bringing up the favorites feature here as potentially related. :)

It may certainly be related, I can't say otherwise. Thanks for trying to narrow it down. I do confirm that folders aren't designed to be treated any different if they're marked as favorites (though I'm not even sure there's a legit way to make that happen from within ES).

Would you be open to doing some good old-fashioned bisecting using git? I'm happy to help with it here.

Thanks!

mediamogul

@pjft said in Scraping multi disc roms:

Would you be open to doing some good old-fashioned bisecting using git?

Certainly. Git bisecting is something I've been very interested to learn more about. I've just looked over the man page and it seems reasonably straightforward. I'll read more about general use and application later today and see if I can't get to a jumping off point.

pjft

@mediamogul oh, great! Let me also share some pointers later as I'm sure I've written something about that in the past that may be helpful. :)

mediamogul

@pjft

That would be most helpful, thanks!

pjft

@mediamogul Not at all.

So, what bisect does is narrow down the change in the code (i.e. the git commit) that introduced the misbehavior, by means of a binary search - basically at each commit in the code history it asks you whether it's good or bad, and then depending on the answer it will select a new version for you to test that's halfway between the current one you just reviewed and the previous known one you reviewed.

It needs to start with two tags from you - a version that you know that it has the problem (let's say, a "bad" one - for this purpose it's the current version but we can probably go further back) and one that doesn't have it - a "good" one.

This post has the instructions I shared for someone else to bisect the code. They may be helpful here.

Still, here they are, adapted to this case. Some assumptions:

Your post was from July 20th 2017, so this was the last commit hash you would possibly have been able to been running at that date:
b28fa2c

The challenge will be to find a previous version where it was working. For the sake of the argument, try this one:
4cc0ced

Which was the last commit before I added metadata to folders, in April 2017.

I'd recommend checking out these two versions first and compiling, to make sure they are indeed bad - and good - as you'd need those two starting points to bisect. I believe that, after you clone the repository to your local drive, you can run:

git checkout <hash>

So, the first step would be to clone it to your Pi somewhere, with the recursive flag.

git clone --recursive --depth 1 https://github.com/RetroPie/EmulationStation

Then run

git checkout stable
git checkout <bad hash>
git bisect start
git bisect bad
git bisect good <good hash>

and then, in each iteration, run

cmake .
make
./emulationstation

and see if it works or not.

If it works, exit ES and run

git bisect good

if not,

git bisect bad

It will then checkout a new version, at which point you'd re-compile and test, using the commands mentioned above, and classify it as good or bad.

You'd do this until it'd tell you exactly the commit that broke that.

It may take a bit of time (depending on the number of steps it requires and how long it takes to re-compile the changes each time), but if you have the time and availability that will help really narrow down what might be the matter. If you don't, it's not a bother - it's understandable.

Sorry for the trouble, and thanks for taking the time to help improve this.

Let me know if this helps.

mediamogul

@pjft

Man that write up was above and beyond. I'm backing up all my metadata just i case and I'll give it a go afterwards. I'm amazed at how useful this process can be. I was reading that it can be used to isolate any changes that are unaccountable, even the odd performance boost. Very keen.

Sorry for the trouble, and thanks for taking the time to help improve this.

You can't make an omelette without breaking some eggs and the improvements you've made are well worth troubleshooting a few unintended side effects. I certainly don't mind taking the time, and if nothing conclusive comes from it, at least we can say it was explored. Thanks again for the instructions and your time in general.