Scraping multi disc roms

HurricaneFan

@pjft Yeah, the scrape finds the data, its the save button press that starts the save process and crashes to command line with that map() error.

mediamogul

@pjft

To make it as easy as possible, I believe the issue can be replicated when scraping any folder at all that doesn't already have a <folder></folder> entry attached to it in it's respective gamelist.xml file. Once 'Save' is selected after scraping, the issue occurs.

Thank you and congrats on your first bisect:)

Thank you for the detailed write up. It really saved me a lot of time.

pjft

@mediamogul so, I played a bit with the code as I don't have current access to the Pi, but this might do the trick.

Would you be so kind as to test whether this prevents the crash?

git clone --recursive https://github.com/pjft/EmulationStation ES-pjft
cd ES-pjft
git checkout scraper-crash
cmake .
make
./emulationstation

I believe I was able to replicate it and test the most common scenarios.

If it doesn't crash, could I ask you for a few tests, both on scraping folders and games?

Thanks.

EDIT: Actually, the change has been merged to the ES-dev branch, if it makes it easier for you to run.

HurricaneFan

@pjft Your build works for me. I tested scraping several individual games and then scraped a folder by itself. All of the test cases were good to me.

By the way thegamesdb scrape source is super slow today. But thats outside of all of our control.

Do you need a pi to test code on? I have an extra pi that isn't being used.

mediamogul

@hurricanefan said in Scraping multi disc roms:

Your build works for me.

Were you using the dev branch or the method above? I had already compiled based on above earlier and it crashed when scraping a folder. I 'm in and out today, but I'll compile the dev branch next time I'm back and see if it makes any difference.

pjft

@hurricanefan you're very kind, I appreciate the offer. It isn't necessary though:)

My previous dev pi is now being used for an arcade build I put together in the past few weeks, so I try to be careful not to break it much with my development shenanigans:)

In today's case I actually didn't have access to a pi at the time to test, as I was accessing it remotely, just that.

I was waiting for the Pi foundation birthday to see if they'd launch a new model, but it's not the case so I'm buying a new 3 for developing on.

Thank you for testing.

@mediamogul the dev branch and my branch are exactly the same as the PR has been accepted already. Actually, I also fixed something in regards to the collection paths that has been broken in the dev branch so that'll be the most advisable build to run things these days. My branch doesn't yet have that fix I also submitted today.

Edit: hm. You're suggesting you had built my branch and it crashed. If you could test it again let me know how it goes. It might have been something else I missed. Exact steps on how you get it to crash can help. But if you find that the dev branch works then that's what matters as that's what's final in the end.

Thanks both for flagging this and helping out! Have a great weekend.

HurricaneFan

@mediamogul I used the exact instructions pjft provided in his post.

mediamogul

@pjft, @hurricanefan

I'll give it another go later this evening.

Exact steps on how you get it to crash can help.

I highlighted a folder in ES without any prior metadata attached to it and selected the scraping option from the menu. The folder name was 'The Oregon Trail', so it discovered quite a few options. I selected one, which then left me at the menu to save. After selecting the 'save' option, the crash happens immediately.

pjft

@mediamogul from your description, that's the exact issue I addressed. Let me know. Best.

mediamogul

@pjft

OK, I tried the one from your branch one more time and still got the crash. However, I then tried the dev branch and no crash at all. I tested it further by scraping multiple folders and game files. I also tested manual metadata additions and it still held up. I want to test a few more scenarios tomorrow such as scraping script files, but for now it's been a crazy long day and I'm looking to break out the Scotch and unwind a little. It really looks like you've nailed this thing down and it'll be great to have this feature back.

pjft

@mediamogul good to hear. Maybe I gave you the wrong instructions, so I wouldn't worry about my branch. Glad that the dev one works for you.

mediamogul

@pjft said in Scraping multi disc roms:

Maybe I gave you the wrong instructions

I'm sure the issue was on my end in some way. I'm just thrilled everything's set now. I plan on testing a few more scenarios around lunch, but I really think you got the crash licked.

mediamogul

@pjft

After testing several scenarios, your fix held up every time in both manually and automatically adding metadata to a directory. I also tested many individual game files for any possible issues that may have sprung up and everything seems to be working well there too. This was stellar work and is most appreciated. Also, thank you to @HurricaneFan for first bringing the issue to light and helping to troubleshoot.

pjft

@mediamogul not at all. A big part of the work was bisecting it, so thank you very much for all the help!

mediamogul

@pjft, @BuZz

It's been a while since this issue was fixed and it still doesn't seem to have made it into the main branch. Was there an issue of some sort, or has it just been lost in the mix? I haven't installed the development branch in a while, does it still exist there?

pjft

@mediamogul I wish I had a better answer here. I'm not really sure there's a proper graduation path from dev features to master. In fact, in hindsight, this breakdown was introduced when there was a lot of entropy because of multiple parallel developments that had an impact across the board - I recall power savings and some internal refactoring taking place at the same time.

Still, given that there's not a lot of meaty new development in ES these days, and the last meaningful changes took place quite a while back, it could make sense to consider a different approach so that new features are released faster?

mediamogul

@pjft said in Scraping multi disc roms:

Still, given that there's not a lot of meaty new development in ES these days, and the last meaningful changes took place quite a while back, it could make sense to consider a different approach so that new features are released faster?

Sorry for the late response... Holidays, ho ho ho and all. I know I would be all for that personally. Especially when it comes to releasing quality of life bug fixes like this. Editing the gamelists by hand is no fun. I believe the 'Last Played' limit has also yet to make it out of the development branch.