Versatile C++ game scraper: Skyscraper

muldjord

@Used2BeRX
I think I need more information on what it is you are working on. A gamelist.xml won't be useful, at least not if it's the same format as the EmulationStation format. What I would like is for someone to define an xml format for game entries, that have nodes and attributes for everything that would make sense for a game.

What format is your source nes database in currently?

Used2BeRX

@muldjord The source files are TXT files.

Double Dragon
Platform: Nintendo Entertainment System
Region: USA
Media: Cartridge
Controller: NES Gamepad
Genre: Beat 'em Up
Gametype: Licensed
Release Year: 1988
Developer: Technos
Publisher: Tradewest
Players: 1 or 2 Alternating
_________________________

Set in a post-apocalyptic New York, Double Dragon is the story of Billy and Jimmy Lee, twin brothers trained in the fighting style of Sou-Setsu-Ken. Together, they manage a small martial arts training school, teaching their students in self-defense. One day, Billy's girlfriend, Marian, is kidnapped off the street by the "Black Warriors", a savage street gang led by a man named Willy. The Black Warriors demand the Lee brothers disclose their martial arts secrets in exchange for Marian's freedom. The Lee brothers set out on a rescue mission to crush the Black Warriors and save Marian.

Using whatever techniques they have at their disposal, from the basic punches and kicks to the invulnerable elbow strike, as well any weapon that comes into their hands, the Lee brothers must pursue the gang through the city slum, industrial area and the forest before reaching their hideout to confront the big boss, Willy.

Controls:

A Button: Punch
B Button: Kick
Start Button: Start game, Pause/Unpause
Select Button: Choose between game

Hints:

Vanishing Weapons
- When you get a weapon from someone it will vanish if you kill them so make sure to kill that person last.

Reviewer: Egosolus
http://www.consoleclassix.com/nes/double_dragon.html

meleu's script pulls all the relevant information and makes gamelist.xml entries from them. They were made for a system on the XBox that is much different than retropie, so depending on the category of rom there are quite a few info fields in these that will not be usable in EmulationStation.

muldjord

@used2berx said in Versatile C++ game scraper: Skyscraper:

Vanishing Weapons

When you get

Txt is easy to parse, that is not a problem. Any complete set of information for any platform would be of interest to me, since I could then take that and create a complete Skyscraper resource database from it. You would of course get all the credit for that database since it is your work it is based on.

How do you handle image and video files? Are there paths to them in the txt files?

Rion

@used2berx You could submit it to screenscraper.fr for example. Join the Irc channel when you submit large collection of files at once.

Used2BeRX

@muldjord I'm actually in the middle of a HUGE project updating the synopsis files and artwork files for the NES, as well as adding 150+ complete translations from romhacking and I'll be adding a lot of homebrew and other games I've found along the way as well. There is a lot of updates possible because the quality of stuff online has improved so much in the last decade.

I actually intend to make new videos too since ours were from 8-10 years ago and all in 480. The stuff on places like emu-movies is "ok", but there is no standards. Volume levels are all over the map, some videos are short while other videos are too long.

I want to make every video between 25-30 seconds long maximum, with an equal sound volume, and with about 15-20 seconds of gameplay wrapped up with 5-10 seconds of the title screen animation. This would be like the demo mode on arcade games. This dream might not be possible right now because of my very old computer that needs updating.

Obviously, this would take a ton of time beyond the thousands of hours I've recently put into emulation since the start of the year. I'm considering possibly looking into crowd funding so I can continue on with this work.

How do you handle image and video files? Are there paths to them in the txt files?

No. There is a specific folder structure for the various media that includes Box Front, Cart, Action, Title, 3D Boxart, videos, gamefaqs, manuals and various other media.

Meleu's script not only pulls all of the data from the synopsis files, but it searches these directories for any media that matches the file name of the game synopsis. Case sensitivity is ignored and it looks for proper extensions by filetype, ie: .jpg or .png for images and .mp4 or .wmv for movies.

If you look at the other thread, you can find the file structure being used. It makes sense, so if you get it for one system, you'd easily be able to figure out where media would go for all of the other systems as well.

I plan on doing writeups for all of this at some point, but I'm just not there yet.

EDIT:

Expect a link for a spreadsheet I've been making soon. It was originally a compatibility list for NES games on the XBox years back that I made that I'm using to update all sorts of stuff now. It has the dimensions of all of the new boxes and cartridge art, as well as a lot of other info.

@Rion Cool. I'll have to look into that. Any way of making that English without Google translate? I clicked on the US flag for "Région préférée" and that didn't do anything.

muldjord

Skyscraper 1.5.0 released!!! https://github.com/muldjord/skyscraper

And this is a whopper! Just look at that changelog!

MAJOR: Added support for local database resources
MAJOR: Added support for video scraping (currently supported in the 'screenscraper' scraping module)
MAJOR: Added 'localdb' scraping module
Added video tag in EmulationStation gamelist.xml output. Beware though, the Pi's are having a difficult time showing the videos properly.
Added several new command line options relevant to the new video and localdb features
Added cover, screenshot and video as part of the result output with "YES" or "NO" depending on whether they were found or not
Fixed a bug where image tag in gamelist.xml had wrong path when using non-default path
Now uses rom or filename (for .uae) sha1 for image filename, in case people have several roms with the same name under subdirs
Added 'players' scraping for 'mobygames' module and improved screenshot getter even more

The major new feature, of course, is the support for local game resource cache databases. One per platform. As you scrape more data, you will find it all under the default folder "./dbs/[platform]". It works completely seamless. The more sources you scrape from, the more complete results you'll get per platform.

And when you finally have results you really like, you can share your database with friends simply by copying the "./dbs/[platform]" folder and handing it to them. They can then use that database with the '-d' option and scrape using the '-s localdb' scraping module. A word of warning though. If your friend copies your database into his own database with the same name, his database will be overwritten! I will implement the '--mergedb' flag soon. But until then, just copy the database to any other location and set it with '-d' when scraping.

Another new feature is video support. As of RetroPie 4.2 EmulationStation supports video. Good news is that it works if you scrape with the '-s screenscraper' module. Bad news is that EmulationStation is having a hard time showing the videos properly. I recon it will get better over time if they keep developing it. If you are having issues with videos, simply disable them altogether with '--novideos'. Then rescrape with '-s localdb'.

This is by far the most elaborate release of Skyscraper to date. I've done a bunch of testing, haven't had any problems so far. But if you do run into trouble or have suggestions, pleeeease let me know. :)

Also, I'd really like to hear if you like these new features. I'm pretty darned excited about the prospects of sharing local db's. I plan to run my amiga setup through a bunch of the scraping modules. Then I'll have a 100% complete set of game details that I can then upload for all of you to use.

I'll get around to that soon aswell.

And now, I think, I've deserved to take the weekend off. I've literally worked on this project for 8 hours a day for the past several days. I love it! But it's starting to take its toll. :D

Have a great weekend everyone! And please comment and let me know what you think!

muldjord

@Used2BeRX
Ok, this is no problem. It is easy to find the files as long as the filenames match 100% and I'll just do .toLower on them.

Rion

@used2berx said in Versatile C++ game scraper: Skyscraper:

@Rion Cool. I'll have to look into that. Any way of making that English without Google translate? I clicked on the US flag for "Région préférée" and that didn't do anything.

Sorry no. As of now the translation is lacking on screenscraper.fr. But the people over there are friendly and helpful over irc.

Rion

@muldjord said in Versatile C++ game scraper: Skyscraper:

Skyscraper 1.5.0 released!!! https://github.com/muldjord/skyscraper

And this is a whopper! Just look at that changelog!

Big ass changelog!

This looks very promising! Bookmarking this now and following this thread.

muldjord

Skyscraper 1.6.0 release: github.com/muldjord/skyscraper
Changelog:

Now allows more resources of same type, as long as 'source' differs
Now allows user to set priorities for local resource sources
Fixed a bug that would nullify timestamp of local resources
Optimized LocalDb communication to improve scraping speed
Added README.md to dbs subfolder
Added priorities.xml.example file to dbs subfolder. Automatically copies this to new databases when they are created if none already exists.
Implemented '--cleandb' command line option that removes files with no resource entry
Implemented '--mergedb' command line option that merges two local databases together
Now no longer does sha1 for roms bigger than 50 MBs (Pi runs out of ram when reading them). Instead does sha1 on filename for those special cases.
Removed default platform when scraping. You are now forced to put in a valid platform with '-p [platform]'
Added more initial info when running Skyscraper
Added '--unattend' command line option
Added 'source' attribute to local database resources
Removed 'mobygames' descriptions from 'openretro' scraper. Now uses native descriptions.
Improved cover and screenshot scraping for 'openretro' module
Disabled filling in missing data when scraping from web sources. User is meant to use 'localdb' scraping module for this.
Implemented date formats to standardize output and better support EmulationStation requirements

Most prominent new additions are the '--cleandb' which cleans any local database folder from files that have no entry in the database and '--mergedb' which allows you to merge to databases together. Combine that with '--updatedb' if you want the source db's resources to take preference.

I've also added a 'source' attribute to any resource in the local databases, which means that you can now have several versions of the same type of resource for each rom in a local database. You can then prioritize them using a 'priorities.xml' file (find an example in './dbs'). A note on this: In the 1.5.0 release resources didn't have a 'source' attribute. When you use these resources with 1.6.0 it will autoadd a 'generic' source to those entries. I recommend deleting the 1.5.0 databases and start over. I know this is inconvenient, so I apologize for this. With that said, I feel the format of the database now does everything I want it to do, so I don't expect it to change again in the 1.x branch.

I've also improved some of the scraping modules quite a bit and fixed a few bugs (and probably created a few new ones :D)

Have fun! As always; comments and feedback are welcome!

incunabula

This tool is awesome! Thanks loads for all of the hard work and sharing it with the community!

UDb23

@muldjord said in Versatile C++ game scraper: Skyscraper:

I will look into vic20 later today. :)
EDIT: I've just had a look at the RetroPie platforms wiki page. I don't see vic20 in there. I know it's a Commodore branded machine, but how does the platform work in RetroPie? Where do the roms reside?

Commodore Vic20 is emulated by Vice; same as Commodore64.
So the System still is Commodore64 in Retropie ,but it can emulate also Vic20 (just different runcommand option)
Vic20 roms & images are therefore located inside the Commodore64 system folder.

Thanks!!

muldjord

@incunabula Thank you! Enjoy! :)

muldjord

@udb23 In that case, you should just be able to use the 'c64' platform which also looks for vic20 files. Just run it with "./Skyscraper -p c64". Have you tried that or am I missing something?

UDb23

@muldjord ok, will try it out over the weekend and let you know.

muldjord

@udb23
Awesome!

Used2BeRX

@muldjord I still have some more NES games to add and some more synopsis tweaks, but I'm going to be away from this work for a few weeks unfortunately. I won't forget about this though and you'll get the pack of updated NES synopsis.txt files when they're done. I've got this page bookmarked so I can catch up when I'm back to it.

muldjord

@Used2BeRX No hurries, I'll look into it when you have things ready. :)

incunabula

Is there a way to change the output of the <name> tag that is written to the gamelist.xml file such that the country information is not shown in ES? It looks like it is currently using the file name of the rom. For example:

Super Fun Game Deluxe (USA, Europe, Japan).zip

should be displayed in the ES game list as

Super Fun Game Deluxe

Is this already possible? I looked throught the readme and the command line switches but didn't see anything relevant. Thank you!

muldjord

@incunabula No, this is not possible at the moment. It is not actually using the file name. What it does is register any round and squarebracket "notes" as I call them. It then uses the web result title, and adds the notes back in.

I'll add a '--nobrackets' option in the next release that will disable the bracket tags. :) You will also be able to set 'brackets="false"' in the config.ini file under both [main] and [platform]. That should give you plenty of options for disabling it. :)

Also, stay tuned for attractmode support, also coming in the next release.