Scraping multi disc roms

mediamogul

I don't know what the potential downsides may be, but removing --depth 1 when cloning seems to have done the trick. 4cc0ced is compiling now.

mediamogul

@pjft

Well, you were right. 4cc0ced didn't crash and b28fa2c did. However, if I'm reading it right, the result of the bisect seems like it can't possibly be an issue, as the commit appears to be nothing more that a version bump. The final word was:

b28fa2ce9f6a4f9b0d8ad3af7bf09fc008c80d59 is the first bad commit
commit b28fa2ce9f6a4f9b0d8ad3af7bf09fc008c80d59
Author: Jools Wills <redacted@redacted.org.uk>
Date:   Tue Jul 11 16:20:03 2017 +0100

    bump version to 2.4.1

:040000 040000 d7404be1e95b25ec121eece939d165655147a167 f0652c9eef80ad41f5b5ff4e40a16f3e0c78f088 M	es-app

pjft

@mediamogul thanks for going through this.

Hm, that is indeed weird.

To confirm, you did test b28fa2c for crashes? When you ran it, did it show "2.4.1" as the version in the menu?
Would you have the log for the entire bisect session?

I suppose, to confirm, you could always checkout b28fa2c and confirm that it crashes, and 227d5b8 (the previous commit) and confirm whether it doesn't crash.

I do agree that that commit is extremely likely not to be the culprit here, so we might have taken a wrong turn in the bisect process. Odds are that either:

b28fa2c doesn't crash but it's being marked by the bisect process as "bad" because it is the last commit he's working with.
The instructions I've provided might have misled bisect (sorry if that's the case!).
We might have mistakenly flagged one revision as "good" or "bad" by mistake in the process. I've had my fair share of those.

To confirm, what this suggests is that in your tests throughout all the bisect process, it never crashed. Is that what happened?

Thanks, and sorry for having you spend time on this.

mediamogul

@pjft

It seems the problem might have been in how the crash was being tested each time. I took for granted that scraping a folder would always reproduce the thrown exception. However, it appears that if the folder has already been scraped successfully, as it was by first testing 4cc0ced, each time after will also be successful, no matter what build you're using. Learning that, I started over from the beginning by first testing the two commits that were thought to be good and bad. After confirming the status of both, I'm now running the full bisect process again, making sure to wipe the metadata from my test folder each time. Hopefully, this will yield a more trustworthy result.

pjft

@mediamogul Oh, sorry about the trouble.

If you set "Save data on exit" to false, maybe you don't need to delete the MD each time?

Thank you.

mediamogul

@pjft said in Scraping multi disc roms:

Oh, sorry about the trouble.

Not your fault. All part of the cutting down the trees to see the forest.

set "Save data on exit"

Good call. You know, I haven't looked at that setting in so long, I forgot about it.

pjft

@mediamogul Well, I can't promise it'll work as intended for scraping, but it's worth testing once and see what to expect. I don't know if scraping has a separate saving process.

mediamogul

@pjft

Bingo, bango, bongo! The preexisting metadata must have been the issue on the original bisect. This time the results have the first "bad" commit as being:

d0cdbf215958e01ffea585a1c6fd75c03b306c46 is the first bad commit
commit d0cdbf215958e01ffea585a1c6fd75c03b306c46
Author: pjft <redacted@gmail.com>
Date:   Mon Jun 12 17:38:59 2017 +0100

    Adding support for Virtual Systems
    - Refactoring System Environment data
    - Added Virtual System Manager class
    - Added "all", "favorites" and "last played" systems
    - Added GuiInfoPopup class for notifications
    - Added Favorites to metadata, as well as a shortcut to toggle favorites
    - Added warning if enabling systems but themes don't support it
    - Added "filter by favorites" per system
    - Adjusted "Go to Random Game" behavior to account for the fact that we now have an "All Games" system
    - Added "sort by system name" for the collections

:040000 040000 d5855f78ce5752cd936b89b1f979dd27eadb682e daa56330c7bbbf0e4a6fde6940db6f8f1e6dc91c M	es-app
:040000 040000 ce4e1e2612ef9020a97c41d2def9edd07e208d2d 5815f4bd6a1e41ec15440579955bd826393109ed M	es-core

pjft

@mediamogul thanks! So your intuition about being related to the favorites was correct. Well done.

I'll try to look into it then - my main concern is actually getting a consistent way to replicate the issue in order to then try to address it.

I'll see what I can do with it.

Would you be able to share exactly what steps you take and at what exact stage it will crash?

Thank you and congrats on your first bisect:)

HurricaneFan

@pjft My scenario was I had a folder inside the SegaCD folder and 2 bin and 2 cue files in that folder. When I try to scrape the folder, it finds the data and image for the game. But when you hit the save icon. Hard crash to command line with no way of using your keyboard any more.

This is using the built in EmulationStation scraper.

pjft

@hurricanefan ok. When exactly hit save for the specific game inside the folder, it crashes.

Is your error similar to the one above, the one for map()?

HurricaneFan

@pjft Yeah, the scrape finds the data, its the save button press that starts the save process and crashes to command line with that map() error.

mediamogul

@pjft

To make it as easy as possible, I believe the issue can be replicated when scraping any folder at all that doesn't already have a <folder></folder> entry attached to it in it's respective gamelist.xml file. Once 'Save' is selected after scraping, the issue occurs.

Thank you and congrats on your first bisect:)

Thank you for the detailed write up. It really saved me a lot of time.

pjft

@mediamogul so, I played a bit with the code as I don't have current access to the Pi, but this might do the trick.

Would you be so kind as to test whether this prevents the crash?

git clone --recursive https://github.com/pjft/EmulationStation ES-pjft
cd ES-pjft
git checkout scraper-crash
cmake .
make
./emulationstation

I believe I was able to replicate it and test the most common scenarios.

If it doesn't crash, could I ask you for a few tests, both on scraping folders and games?

Thanks.

EDIT: Actually, the change has been merged to the ES-dev branch, if it makes it easier for you to run.

HurricaneFan

@pjft Your build works for me. I tested scraping several individual games and then scraped a folder by itself. All of the test cases were good to me.

By the way thegamesdb scrape source is super slow today. But thats outside of all of our control.

Do you need a pi to test code on? I have an extra pi that isn't being used.

mediamogul

@hurricanefan said in Scraping multi disc roms:

Your build works for me.

Were you using the dev branch or the method above? I had already compiled based on above earlier and it crashed when scraping a folder. I 'm in and out today, but I'll compile the dev branch next time I'm back and see if it makes any difference.

pjft

@hurricanefan you're very kind, I appreciate the offer. It isn't necessary though:)

My previous dev pi is now being used for an arcade build I put together in the past few weeks, so I try to be careful not to break it much with my development shenanigans:)

In today's case I actually didn't have access to a pi at the time to test, as I was accessing it remotely, just that.

I was waiting for the Pi foundation birthday to see if they'd launch a new model, but it's not the case so I'm buying a new 3 for developing on.

Thank you for testing.

@mediamogul the dev branch and my branch are exactly the same as the PR has been accepted already. Actually, I also fixed something in regards to the collection paths that has been broken in the dev branch so that'll be the most advisable build to run things these days. My branch doesn't yet have that fix I also submitted today.

Edit: hm. You're suggesting you had built my branch and it crashed. If you could test it again let me know how it goes. It might have been something else I missed. Exact steps on how you get it to crash can help. But if you find that the dev branch works then that's what matters as that's what's final in the end.

Thanks both for flagging this and helping out! Have a great weekend.

HurricaneFan

@mediamogul I used the exact instructions pjft provided in his post.

mediamogul

@pjft, @hurricanefan

I'll give it another go later this evening.

Exact steps on how you get it to crash can help.

I highlighted a folder in ES without any prior metadata attached to it and selected the scraping option from the menu. The folder name was 'The Oregon Trail', so it discovered quite a few options. I selected one, which then left me at the menu to save. After selecting the 'save' option, the crash happens immediately.

pjft

@mediamogul from your description, that's the exact issue I addressed. Let me know. Best.