Scraping multi disc roms

mediamogul

Bingo, bango, bongo! The preexisting metadata must have been the issue on the original bisect. This time the results have the first "bad" commit as being:

d0cdbf215958e01ffea585a1c6fd75c03b306c46 is the first bad commit
commit d0cdbf215958e01ffea585a1c6fd75c03b306c46
Author: pjft <redacted@gmail.com>
Date:   Mon Jun 12 17:38:59 2017 +0100

    Adding support for Virtual Systems
    - Refactoring System Environment data
    - Added Virtual System Manager class
    - Added "all", "favorites" and "last played" systems
    - Added GuiInfoPopup class for notifications
    - Added Favorites to metadata, as well as a shortcut to toggle favorites
    - Added warning if enabling systems but themes don't support it
    - Added "filter by favorites" per system
    - Adjusted "Go to Random Game" behavior to account for the fact that we now have an "All Games" system
    - Added "sort by system name" for the collections

:040000 040000 d5855f78ce5752cd936b89b1f979dd27eadb682e daa56330c7bbbf0e4a6fde6940db6f8f1e6dc91c M	es-app
:040000 040000 ce4e1e2612ef9020a97c41d2def9edd07e208d2d 5815f4bd6a1e41ec15440579955bd826393109ed M	es-core

pjft

@mediamogul thanks! So your intuition about being related to the favorites was correct. Well done.

I'll try to look into it then - my main concern is actually getting a consistent way to replicate the issue in order to then try to address it.

I'll see what I can do with it.

Would you be able to share exactly what steps you take and at what exact stage it will crash?

Thank you and congrats on your first bisect:)

HurricaneFan

@pjft My scenario was I had a folder inside the SegaCD folder and 2 bin and 2 cue files in that folder. When I try to scrape the folder, it finds the data and image for the game. But when you hit the save icon. Hard crash to command line with no way of using your keyboard any more.

This is using the built in EmulationStation scraper.

pjft

@hurricanefan ok. When exactly hit save for the specific game inside the folder, it crashes.

Is your error similar to the one above, the one for map()?

HurricaneFan

@pjft Yeah, the scrape finds the data, its the save button press that starts the save process and crashes to command line with that map() error.

mediamogul

@pjft

To make it as easy as possible, I believe the issue can be replicated when scraping any folder at all that doesn't already have a <folder></folder> entry attached to it in it's respective gamelist.xml file. Once 'Save' is selected after scraping, the issue occurs.

Thank you and congrats on your first bisect:)

Thank you for the detailed write up. It really saved me a lot of time.

pjft

@mediamogul so, I played a bit with the code as I don't have current access to the Pi, but this might do the trick.

Would you be so kind as to test whether this prevents the crash?

git clone --recursive https://github.com/pjft/EmulationStation ES-pjft
cd ES-pjft
git checkout scraper-crash
cmake .
make
./emulationstation

I believe I was able to replicate it and test the most common scenarios.

If it doesn't crash, could I ask you for a few tests, both on scraping folders and games?

Thanks.

EDIT: Actually, the change has been merged to the ES-dev branch, if it makes it easier for you to run.

HurricaneFan

@pjft Your build works for me. I tested scraping several individual games and then scraped a folder by itself. All of the test cases were good to me.

By the way thegamesdb scrape source is super slow today. But thats outside of all of our control.

Do you need a pi to test code on? I have an extra pi that isn't being used.

mediamogul

@hurricanefan said in Scraping multi disc roms:

Your build works for me.

Were you using the dev branch or the method above? I had already compiled based on above earlier and it crashed when scraping a folder. I 'm in and out today, but I'll compile the dev branch next time I'm back and see if it makes any difference.

pjft

@hurricanefan you're very kind, I appreciate the offer. It isn't necessary though:)

My previous dev pi is now being used for an arcade build I put together in the past few weeks, so I try to be careful not to break it much with my development shenanigans:)

In today's case I actually didn't have access to a pi at the time to test, as I was accessing it remotely, just that.

I was waiting for the Pi foundation birthday to see if they'd launch a new model, but it's not the case so I'm buying a new 3 for developing on.

Thank you for testing.

@mediamogul the dev branch and my branch are exactly the same as the PR has been accepted already. Actually, I also fixed something in regards to the collection paths that has been broken in the dev branch so that'll be the most advisable build to run things these days. My branch doesn't yet have that fix I also submitted today.

Edit: hm. You're suggesting you had built my branch and it crashed. If you could test it again let me know how it goes. It might have been something else I missed. Exact steps on how you get it to crash can help. But if you find that the dev branch works then that's what matters as that's what's final in the end.

Thanks both for flagging this and helping out! Have a great weekend.

HurricaneFan

@mediamogul I used the exact instructions pjft provided in his post.

mediamogul

@pjft, @hurricanefan

I'll give it another go later this evening.

Exact steps on how you get it to crash can help.

I highlighted a folder in ES without any prior metadata attached to it and selected the scraping option from the menu. The folder name was 'The Oregon Trail', so it discovered quite a few options. I selected one, which then left me at the menu to save. After selecting the 'save' option, the crash happens immediately.

pjft

@mediamogul from your description, that's the exact issue I addressed. Let me know. Best.

mediamogul

@pjft

OK, I tried the one from your branch one more time and still got the crash. However, I then tried the dev branch and no crash at all. I tested it further by scraping multiple folders and game files. I also tested manual metadata additions and it still held up. I want to test a few more scenarios tomorrow such as scraping script files, but for now it's been a crazy long day and I'm looking to break out the Scotch and unwind a little. It really looks like you've nailed this thing down and it'll be great to have this feature back.

pjft

@mediamogul good to hear. Maybe I gave you the wrong instructions, so I wouldn't worry about my branch. Glad that the dev one works for you.

mediamogul

@pjft said in Scraping multi disc roms:

Maybe I gave you the wrong instructions

I'm sure the issue was on my end in some way. I'm just thrilled everything's set now. I plan on testing a few more scenarios around lunch, but I really think you got the crash licked.

mediamogul

@pjft

After testing several scenarios, your fix held up every time in both manually and automatically adding metadata to a directory. I also tested many individual game files for any possible issues that may have sprung up and everything seems to be working well there too. This was stellar work and is most appreciated. Also, thank you to @HurricaneFan for first bringing the issue to light and helping to troubleshoot.

pjft

@mediamogul not at all. A big part of the work was bisecting it, so thank you very much for all the help!

mediamogul

@pjft, @BuZz

It's been a while since this issue was fixed and it still doesn't seem to have made it into the main branch. Was there an issue of some sort, or has it just been lost in the mix? I haven't installed the development branch in a while, does it still exist there?

pjft

@mediamogul I wish I had a better answer here. I'm not really sure there's a proper graduation path from dev features to master. In fact, in hindsight, this breakdown was introduced when there was a lot of entropy because of multiple parallel developments that had an impact across the board - I recall power savings and some internal refactoring taking place at the same time.

Still, given that there's not a lot of meaty new development in ES these days, and the last meaningful changes took place quite a while back, it could make sense to consider a different approach so that new features are released faster?