Gamelist Addon - an assist tool to manually add new game entries

rilight

Thank you! This seems handy, I've been using the interface in EmulationStation to edit game entries but this seems like a nice way to add multiple entries at once without as much playing around with the controller.

Clyde

Very nice and gratifyingly for Linux. 😀 (No offense to the Windows users here, you usually have more tools to choose for our mutual retrogaming hobby. As for the Mac users – I feel for you. 😔 )

I like the idea of a merge function. How does it work exactly? E.g. does it merge only full game elements or also the different sub-elements of the same game element, like videos or descriptions? What does it do with elements that exist in both files?

And last, but not least, any chance that you'll host it on adevelopment repository like Github or Gitlab? I just it more convenient to keep them updated locally via the command line.

thelostsoul

@rilight Just making clear, this is a desktop tool for the pc and won't work on RetroPie. My recommendation is using SMB for reading and writing the gamelist.xml files back. And you still need to add the image files and such manually, it is not a scraper like too that does everything for you.
One question, how did you manage to add new game entries with a controller? I'm just curious about this workflow.

@Clyde Currently the operation is quiet simple. The merge function will only add a new game, if its not in the main base file already. I wanted to have an option to update the original entry either fully or even compare their sub elements individually. I am not sure how to deal with it and how to test if everything is fine. Because it gets very complex and I learned a lot new stuff in the last weeks. The comparison is done by the basename from path, which ignores the folder of it. This allows to detect same game regardless how the directory part of path is.

The script/tool is really for adding missing or new games. Btw it is hosted on Github and is actually my first project. But be aware the make script is not standardized, it is a custom Python script. Generall I don't know much how Github works and don't follow the standards. Just a disclaimer.

Initially it was a bit more ambitious. Hopefully I can get my head around to make the merge function more elaborate.

Clyde

@thelostsoul Well, it's enough for me to just have it on GitSomething so I can update my local copy with a single command. 🤓

One idea about doubles when merging: Maybe just add an option like "doubles: skip / prefer first / prefer second"?

thelostsoul

@Clyde Yes, I have something like this in mind too.
A small note here if you use from source. The source version would not run without a small command to generate Python code from the ui files.
I suppose you are not interested in all the other stuff the make.py file does, because it is not relevant for the Python only user. If you have Python 3 and PyQt5 + Designer installed, then you need to run ./make.py --ui once. That is why I have a separate download as a ready to run package without all the other files you don't need.
Edit: Now I understand how to work in Github. Each release page will include a download zip for the end user. I can get rid off to upload this to Google now (but the -bin and -bundle packages are still too big and that is why they are on Google Drive).

rilight

@thelostsoul said in Gamelist Addon - an assist tool to manually add new game entries:

@rilight Just making clear, this is a desktop tool for the pc and won't work on RetroPie. My recommendation is using SMB for reading and writing the gamelist.xml files back. And you still need to add the image files and such manually, it is not a scraper like too that does everything for you.

I already have the Pixel desktop set up on my RetroPie install, so with a keyboard and mouse I believe it'll run just fine from there. I set it up so I could run Skraper, do basic web browsing, and that kind of thing.

One question, how did you manage to add new game entries with a controller? I'm just curious about this workflow.

If you press select while on a game entry in a system menu, it brings up an options entry that has a metadata editor as part of it. I don't have my keyboard set up as a controller though, so it can be a bit awkward to switch back and forth to navigate between text fields.

thelostsoul

Its up again. I restructured and restarted the project, with a version numbering 0.1 (previously it was 1.0). Only the documentation and organization of the files are changed, the application still does the same.

thelostsoul

I have some questions.

I am almost finished with a merge option to ignore, replace or update entries. But I am not sure if such an update feature is useful at all.
Can I assume the "path" of a game is an unique id that can be found only once in a gamelist? Because that principle is the backbone of this application.

mitu

@thelostsoul said in Gamelist Addon - an assist tool to manually add new game entries:

Can I assume the "path" of a game is an unique id that can be found only once in a gamelist? Because that principle is the backbone of this application.

Generally, yes - that would be a correct assumption. Does your program check the file-path, to see if it exists ?

thelostsoul

@mitu No. Checking if files exist is not the job of the program. It is only for adding new game entries to an xml file and the comparison is made to make sure the game with same filename does not exist in it already (or when merging xml files). The user have to make sure to copy (or delete) all connected files. It is not meant to be a full rom manager.

Clyde

@thelostsoul said in Gamelist Addon - an assist tool to manually add new game entries:

I am almost finished with a merge option to ignore, replace or update entries. But I am not sure if such an update feature is useful at all.

I see at least one practical application: I tend to use Simple Arcade Multifilter to sort the roms into genre subdirectories to see which games belong to a certain genre – and which may be tagged with the wrong genre in the dat file. After I've sifted through the games, your tool may be useful to merge the several gameslist files into one.

Okay, that may be also possible with a simple for f in */; do cat "${f}gamelist.xml" >> gamelist.xml; done which does append every subdir's gamelist file to a new one in the current directory, but that wouldn't exclude doubles.

thelostsoul

@Clyde Your use case is exactly what I do, just combining both (or multiple) xml files into one. The default behavior is to lookout for "doubles", so it does not add the same entry again. This is what the merge functionality is designed for. I call this "ignore", just adding new unknown games.

My above question is leaning towards "update" functionalities, where individual tags like lastplayed or image would be updated individually. Because I do not see any benefit from such update functionality in a real world scenario. I probably give up on this (also can't make it work right).

Clyde

Given that I understand you correctly, I'm actually looking for a tool that could merge tags like lastplayed from an old gamelist to a newly scraped one, since I often rebuild the gamelist via Skyscraper, losing every change that Emulation Station made to the old gamelist.

Trying to write a bash script that does this is on my much too long to-do list for Retropie. 😉

thelostsoul

Version 0.2 is released. I hope the Update function works as you expect it would.

Update mode
Overall my attention mostly goes to the new update feature in the merge mode. When switched to the update mode via radio button, then in the following merge operation every single tag from both files are compared. The one from add content is then used, if its available. Otherwise it defaults to base content.

I catched a lot of bugs, but won't recommend it to use on production yet.

thelostsoul

Version 0.3 is released.

new dedicated settings for Merge Update mode, limit updates to selected Tags only
few missing Tags support added, previously they were ignored, namely sortname, thumbnail, favorite, hidden and kidgame
layout of main Add Game view redesigned, to reflect the new additions

The GUI does not look as simple and clean as before, but the changes was necessary. There was a few Tags missing, which are now supported. A new page for limiting the updates on individual Tags when using the Update Mode in Merge Gamelists is added. One use case scenario for this feature could be in example to transfer the lastplayed and playcount from one file to another. This could be needed when the main gamelist was rescaped for example.

Clyde

@thelostsoul That's exactly what I need, thank you so much! 😃

That said, a command line mode would be even better to automate things, like transferring lastplayed etc. after each new scraping. But that's a purely luxury wish, resp. a feature request for future times when you may get bored. 😉

The tool is already awesome as it is now.

thelostsoul

@Clyde I know what you mean, because it's exactly what I need too. :D

About the automation, yes I want that implemented in the main app too someday. Just in case you did miss it, there is a quick and dirty commandline version of merge functionality since the beginning: merge .py. Its located in the same folder as the main app (off course not included in the compiled version).

This is exactly for the case of automation, try ./merge.py --help. Using it without an output file will not modify anything. It uses the same backend for merging as the main app (modular design 🤗), but the custom tags functionality is not implemented yet, sorry bout that. And its error handling is poor too. When an error happens, it stops working and writes ERROR to terminal. That's all.

I also want to make the whole program better testable, but that requires a bit of internal reworking. I need testing more, although it seems to be solid right now. So before making more tests, I don't recommend using it in productive environment.

Clyde

@thelostsoul said in Gamelist Addon - an assist tool to manually add new game entries:

About the automation, yes I want that implemented in the main app too someday. Just in case you did miss it,

I actually did miss it. Thanks for the directions.