Scraping multi disc roms

mediamogul

Can you describe to me the folder structure, content of root and sub folders you're trying to scrape?

Sure. This particular example is about as basic as it gets. There's one folder and two items inside.

/The Oregon Trail/
Oregon Trail (Disk 1 of 2).dsk
Oregon Trail (Disk 1 of 2).dsk

Ideally/alternatively I'd certainly recommend creating a bug, for tracking, especially after we are able to narrow down the cause.

I'll go ahead and do that now.

HurricaneFan

Yeah, My scenario was I had a folder inside the SegaCD folder and 2 bin and 2 cue files in that folder. When I try to scrape the folder, it finds the data and image for the game. But when you hit the save icon. Hard crash to command line with no way of using your keyboard any more.

clueless

Hi, sorry to necro but this is the only relevant thread I found.
Did you ever find a solution to this problem? The same thing happens to me when trying to save metadata/scrape to folders. It's annoying because for some reason the folder is named wrong in ES and trying to change it manually kicks me out of ES when I click save, requiring me to pull the power plug as no input will work afterwards.

HurricaneFan

@clueless An issue was filed but its still a bug.

https://github.com/RetroPie/EmulationStation/issues/186

clueless

@hurricanefan thank you

mediamogul

Rather than clutter the Github issue with potential theories, I thought I might float one here first. This issue seems to coincide with the addition of the 'Favorites' feature, dating around the last few weeks of last July. I've noticed that you cannot add a folder as a favorite, but tonight I got curious to see what would happen if I manually added the 'favorite' MD tag to a folder.

Anticlimactically, ES launched without so much as an error, while simply ignoring the folder in the 'favorites' list. However, the folder was in fact showing as being a favorite in the MD inspector. I'm wondering if the fact that folders are not fully accounted for as being a valid option for the favorite MD tag is somehow related to the "out of range" instance that's being thrown.

pjft

@mediamogul Sure, I'll bite since you're bringing up the favorites feature here as potentially related. :)

It may certainly be related, I can't say otherwise. Thanks for trying to narrow it down. I do confirm that folders aren't designed to be treated any different if they're marked as favorites (though I'm not even sure there's a legit way to make that happen from within ES).

Would you be open to doing some good old-fashioned bisecting using git? I'm happy to help with it here.

Thanks!

mediamogul

@pjft said in Scraping multi disc roms:

Would you be open to doing some good old-fashioned bisecting using git?

Certainly. Git bisecting is something I've been very interested to learn more about. I've just looked over the man page and it seems reasonably straightforward. I'll read more about general use and application later today and see if I can't get to a jumping off point.

pjft

@mediamogul oh, great! Let me also share some pointers later as I'm sure I've written something about that in the past that may be helpful. :)

mediamogul

@pjft

That would be most helpful, thanks!

pjft

@mediamogul Not at all.

So, what bisect does is narrow down the change in the code (i.e. the git commit) that introduced the misbehavior, by means of a binary search - basically at each commit in the code history it asks you whether it's good or bad, and then depending on the answer it will select a new version for you to test that's halfway between the current one you just reviewed and the previous known one you reviewed.

It needs to start with two tags from you - a version that you know that it has the problem (let's say, a "bad" one - for this purpose it's the current version but we can probably go further back) and one that doesn't have it - a "good" one.

This post has the instructions I shared for someone else to bisect the code. They may be helpful here.

Still, here they are, adapted to this case. Some assumptions:

Your post was from July 20th 2017, so this was the last commit hash you would possibly have been able to been running at that date:
b28fa2c

The challenge will be to find a previous version where it was working. For the sake of the argument, try this one:
4cc0ced

Which was the last commit before I added metadata to folders, in April 2017.

I'd recommend checking out these two versions first and compiling, to make sure they are indeed bad - and good - as you'd need those two starting points to bisect. I believe that, after you clone the repository to your local drive, you can run:

git checkout <hash>

So, the first step would be to clone it to your Pi somewhere, with the recursive flag.

git clone --recursive --depth 1 https://github.com/RetroPie/EmulationStation

Then run

git checkout stable
git checkout <bad hash>
git bisect start
git bisect bad
git bisect good <good hash>

and then, in each iteration, run

cmake .
make
./emulationstation

and see if it works or not.

If it works, exit ES and run

git bisect good

if not,

git bisect bad

It will then checkout a new version, at which point you'd re-compile and test, using the commands mentioned above, and classify it as good or bad.

You'd do this until it'd tell you exactly the commit that broke that.

It may take a bit of time (depending on the number of steps it requires and how long it takes to re-compile the changes each time), but if you have the time and availability that will help really narrow down what might be the matter. If you don't, it's not a bother - it's understandable.

Sorry for the trouble, and thanks for taking the time to help improve this.

Let me know if this helps.

mediamogul

@pjft

Man that write up was above and beyond. I'm backing up all my metadata just i case and I'll give it a go afterwards. I'm amazed at how useful this process can be. I was reading that it can be used to isolate any changes that are unaccountable, even the odd performance boost. Very keen.

Sorry for the trouble, and thanks for taking the time to help improve this.

You can't make an omelette without breaking some eggs and the improvements you've made are well worth troubleshooting a few unintended side effects. I certainly don't mind taking the time, and if nothing conclusive comes from it, at least we can say it was explored. Thanks again for the instructions and your time in general.

mediamogul

@pjft

I've cloned the repository using:

git clone --recursive --depth 1 https://github.com/RetroPie/EmulationStation

changed directories to ./EmulationStation/ and ran

git checkout 4cc0ced

However, I'm getting:

error: pathspec '4cc0ced' did not match any file(s) known to git.

The same thing goes for:

git checkout b28fa2c

mediamogul

@pjft

I don't know what the potential downsides may be, but removing --depth 1 when cloning seems to have done the trick. 4cc0ced is compiling now.

mediamogul

@pjft

Well, you were right. 4cc0ced didn't crash and b28fa2c did. However, if I'm reading it right, the result of the bisect seems like it can't possibly be an issue, as the commit appears to be nothing more that a version bump. The final word was:

b28fa2ce9f6a4f9b0d8ad3af7bf09fc008c80d59 is the first bad commit
commit b28fa2ce9f6a4f9b0d8ad3af7bf09fc008c80d59
Author: Jools Wills <redacted@redacted.org.uk>
Date:   Tue Jul 11 16:20:03 2017 +0100

    bump version to 2.4.1

:040000 040000 d7404be1e95b25ec121eece939d165655147a167 f0652c9eef80ad41f5b5ff4e40a16f3e0c78f088 M	es-app

pjft

@mediamogul thanks for going through this.

Hm, that is indeed weird.

To confirm, you did test b28fa2c for crashes? When you ran it, did it show "2.4.1" as the version in the menu?
Would you have the log for the entire bisect session?

I suppose, to confirm, you could always checkout b28fa2c and confirm that it crashes, and 227d5b8 (the previous commit) and confirm whether it doesn't crash.

I do agree that that commit is extremely likely not to be the culprit here, so we might have taken a wrong turn in the bisect process. Odds are that either:

b28fa2c doesn't crash but it's being marked by the bisect process as "bad" because it is the last commit he's working with.
The instructions I've provided might have misled bisect (sorry if that's the case!).
We might have mistakenly flagged one revision as "good" or "bad" by mistake in the process. I've had my fair share of those.

To confirm, what this suggests is that in your tests throughout all the bisect process, it never crashed. Is that what happened?

Thanks, and sorry for having you spend time on this.

mediamogul

@pjft

It seems the problem might have been in how the crash was being tested each time. I took for granted that scraping a folder would always reproduce the thrown exception. However, it appears that if the folder has already been scraped successfully, as it was by first testing 4cc0ced, each time after will also be successful, no matter what build you're using. Learning that, I started over from the beginning by first testing the two commits that were thought to be good and bad. After confirming the status of both, I'm now running the full bisect process again, making sure to wipe the metadata from my test folder each time. Hopefully, this will yield a more trustworthy result.

pjft

@mediamogul Oh, sorry about the trouble.

If you set "Save data on exit" to false, maybe you don't need to delete the MD each time?

Thank you.

mediamogul

@pjft said in Scraping multi disc roms:

Oh, sorry about the trouble.

Not your fault. All part of the cutting down the trees to see the forest.

set "Save data on exit"

Good call. You know, I haven't looked at that setting in so long, I forgot about it.

pjft

@mediamogul Well, I can't promise it'll work as intended for scraping, but it's worth testing once and see what to expect. I don't know if scraping has a separate saving process.