How do you edit the gamelist.xml manually?

thelostsoul

For starters: "what is the gamelist.xml?": gamelist.xml is the "database" file with all the information for the scraped games. Usually you don't need to edit it manually. Using a scraper tool will do all the needed work to add new games or update existing entries. However, sometimes games in special cases are not known to the scraper tool or database it uses. These cases are in example adding new romhacks. Then I have to edit the gamelist.xml and add all files manually. So, don't edit the file, if you don't know how to work with it.

My actual question is, which tools do you use, if any to edit the gamelist.xml properly? All the time until now I did it with a plain text editor and simple copy and paste. And everytime I use a scraper and let the file generate, all changes are gone. So I end up having a second file with all the <game> entries and put them back after the scraping process. Are there any common tools?

Impman66

I use programmers notepad. Which is free to download and use, and it allows you to open and edit a number of other stuff too.

Have a look

link to Programmers Notepad

thelostsoul

@Impman66 Oh, I wasn't asking for a notepad replacement. My question was more about if there are specialized tools to deal with the gamelist.xml format. So the answer I get from you is that you use an editor like me. Btw I am on Ubuntu and have multiple editors, so that is not the issue.

Actually I wrote a script and gui application (in Python, also good exercise to learn it) the last several days to help me with dealing directly with that specific XML format. My hope is, that there are others who made/use a more comprehensive tool.

Impman66

Apologies, I am just a beginner in all this, so wouldn't have a clue on what to recommend. I only said about the Programmers Notepad, as this has been invaluable to learning, by looking at others codes for a theme. Then using that to develop my own.

Good luck in your search

mitu

I think ARRM can help you edit the gamelist.xml file without resorting to direct edits.

thelostsoul

ARRM
@mitu Interesting app. I wasn't aware of it and it is quite impressive. Unfortunately I also can't test it, because its Windows only and I am using Linux. It could run under WINE, but don't know how well it works and I don't want install it for now (also quite big).

WeirdH

@thelostsoul said in How do you edit the gamelist.xml manually?:

All the time until now I did it with a plain text editor and simple copy and paste. And everytime I use a scraper and let the file generate, all changes are gone.

I don't know if this works the same for all scrapers, but I use the Steven Selph one: in the scraper menu I selected 'append' instead of 'overwrite' (or whatever the other option is called). This only scans games that aren't in the list already and leaves all other info alone.

mitu

I'm not sure if an editor is going to help much, but I see a few workarounds for this:

when editing your gamelist.xml to add hacks info, add a custom attribute to the game entry

<game mine=1>
<path>ROM_Path</path>
<name>Blah</name>
</game>

Before scraping, backup these entries to an additional file with xmlstarlet:

 xmlstarlet sel -t -m '//game[@mine=1]' -c . ~/roms/ngpc/gamelist.xml > ~/roms/ngpc/gamelist_mine.xml

When using skyscraper, instruct it to ignore any non-scraped entries and to no add them to the gamelist, with the skipped flag. After scraping is done, add the entries from your backup to your main gamelist.

(best option) add the necessary ROM metadata to Skyscraper's cache - either using the import module (if you need image/video) or via the cache editing options.

EDIT: forgot that skyscraper has an ES gamelist scraping module. As an addendum to the 2nd option presented above, you can scrape your games using it. The info you added to the games in the gamelist.xml files should be added to the cache, so it will help when re-generating the gamelist.xml again.

Bluescreen

Hi
I am trying to learn programming. I have created a small program. it works for me but it has a few bugs.

thelostsoul

@Bluescreen This looks great and is what I was looking for. Is this more of a private helper application that you developed for yourself?
@WeirdH When I learned and experimented with RetroPie and the scrapers (years back now) I experienced a few problems with append operation. Don't ask me what problems these was. From that point I decided to go with overwrite and two files, just in case.
@mitu Actually, the introduction of a custom attribute is a good idea to quickly find any custom or edited entry. For the way you suggest: if I have to copy the custom entries and save them to a custom xml and reinstert them after scraping... that is actually more work for me than what I do now. I could just cut out the first work of getting them from the main list. But I guess this is just a workflow thing and could be automated anyway, so thanks for the idea.
I never used Skyscraper. It looks promising and maybe I should change my default scraper, especially if it have the functionality I need.

So for now I need to evaluate what to do next and this can take ages. In the meantime if someone else have ideas or tools which helps you, you can post them here too.

Bluescreen

@thelostsoul Yes its a private helper application. i learn coding, and this is my learning projekt.

It´s works for the most. The program is slow from time to time and has a few small errors.
works very well for a learning project. I can send you the program if you want

-Features
-XML read write
-check whether files are available.
-ID and Info: opens the Screenscraper page with the ID of the respective game (cover, marquee, video etc)
-Bugs ;)

sorry my english is not the best.

Impman66

@Bluescreen said in How do you edit the gamelist.xml manually?:

@thelostsoul Yes its a private helper application. i learn coding, and this is my learning projekt.

It´s works for the most. The program is slow from time to time and has a few small errors.
works very well for a learning project. I can send you the program if you want

-Features
-XML read write
-check whether files are available.
-ID and Info: opens the Screenscraper page with the ID of the respective game (cover, marquee, video etc)
-Bugs ;)

sorry my english is not the best.

Your English is very good actually.
Looking at this I think there may be a few more people interested in your little "learning coding" project.
May be the mods can direct as to a way of getting it out there - with your permission of course.

Bluescreen

@Impman66 said in How do you edit the gamelist.xml manually?:

Your English is very good actually.
Looking at this I think there may be a few more people interested in your little "learning coding" project.
May be the mods can direct as to a way of getting it out there - with your permission of course.

I do not have a problem with that. the code only looks dirty;)
but i think most only want the program and not the code

Gamesman

@Bluescreen I would be interest in the code. The program looks nice and promising.

thelostsoul

@Bluescreen I guess it is a Windows only app, as you are learning right now. Does it run in Linux? I would want to have a little look at the application and if possible in the code, just curious.

Bluescreen

@thelostsoul
It´s a Windows only app. i don't know much around linux.

GamelistManager Download

password is: retropie

only the app with 4 test entries for the snes. I can also upload the 20gb (without video and roms) or the full 80gb (without roms)

@Gamesman i can upload the code or the complete projectfile

thelostsoul

I wasn't happy, so I made my own tool: https://retropie.org.uk/forum/topic/27466/gamelist-addon-an-assist-tool-to-manually-add-new-game-entries. Unlike the other tools, this is really just to add new games and not a general scraper or editing tool.

@Bluescreen, I had a quick look at the folder structure to see whats inside. Because its interesting. :-) About the 20gb, I am not really interested into the images itself (I suppose thats what the 20gb are).
What test entries do you have and what are you testing? I am absolutely not sure here what to test.

Bluescreen

@thelostsoul open the snes list via my program. the first 4 entries are with game information, cover video and marquee.
to see what it looks like

edit

my tool does exactly the same as yours. the only difference is video marquee and video display. It can't scrape. just edit the gamelist.xml.

I'm glad you programmed one yourself.

thelostsoul

@Bluescreen Unfortunately I can't test it, because as said before I am on Linux. I tried it on a very old Win10 virtual machine, but it does not work. Probably because of too old libraries.
The tool I have created do not edit in the sense of deleting stuff and it does not show a table. Its really just adding new games only, as this is exactly the only thing I need.

thelostsoul

Just saw this app related to this topic. Its not exactly what I am looking for, just want to link it here: RetroPi Media - gamelist.xml generator