[Idea] Game Manual viewing in ES

SuperFromND

@jell I don't think there is a standardized format; I usually just see image files of each individual page, but I've heard of some people using PDF files instead. Personally I think the former would be better for the end user, since PDFs are a pain in the neck to deal with.

pjft

I think the idea has some merit, and I was thinking about this yesterday just for fun.

That being said, if we don't have a standard source for scraping the manuals, a scraper that supports that (so that users get them easily) and if there's no standard format for that content (single PDF file? Multiple image files?) then it's harder to generalize and implement.

SuperFromND

@pjft I spent a bit to think it over, and here's what I think:

As for file format, after looking into it, it turns out there's a lot of people that use PDF files for manuals, so I think using PDF would be the best choice here. (besides, it can't possibly be that hard to create a PDF out of a few PNG files.)

As for scraping? That's gonna be a bit tricky, as finding a website with a decent API and a hefty database of PDF manuals proves to be quite challenging. (the closest I came was with ReplacementDocs) However, videos aren't scraped by ES's scraper either (instead you have to use Steven's scraper, or just download and link the file manually), so I don't think that's a particularly huge deal.

Slimy

Don't really have anything to contribute, but I would like this to be implemented. It would be useful to have manual scans on the same machine as the games, whenever I wanted to read the story, remember the controls, or just look at the maps/pictures. Especially for a game like Startropics which asks you to input a number on a paper that came with the game.

detron

great idea, I asked about manuals about a year or so ago, but I had no idea how to implement it.
I see you put thought into it

pjft

@supercatfooz said in [Idea] Game Manual viewing in ES:

However, videos aren't scraped by ES's scraper either (instead you have to use Steven's scraper, or just download and link the file manually), so I don't think that's a particularly huge deal.

SSelph's scraper is still a scraper, meaning that there is a way for users to batch retrieve such information. I wasn't necessarily referring to the ES one.

It's not that it is hard to create a PDF out of a few PNG files, or the other way around, but what use case do you have in mind? Will each individual have to manually download and convert these to use them? I have a hard time imagining that that approach would become mainstream, as the reach of the feature will have a hard time justifying the effort.

I know we have the option of downloading videos as a one-off, but there are also databases such as EmuMovies that allow you to download an entire collection of videos that you can then map.

Regardless of the challenges, though, the idea has potential - I'm just voicing some practical concerns or questions. :)

darknior

If you release this addon, it will be a dream for me <3

@pjft said in [Idea] Game Manual viewing in ES:

I think the idea has some merit, and I was thinking about this yesterday just for fun.

That being said, if we don't have a standard source for scraping the manuals, a scraper that supports that (so that users get them easily) and if there's no standard format for that content (single PDF file? Multiple image files?) then it's harder to generalize and implement.

There are some PDF on ScreenScrapper ...

On xBox scene (EmuXtra) we transform many PDF manual to a .zip file with an .JPG image by page.
Because we can't read PDF and it take less memory to read images one by one with 64 Mo of RAM.
@Used2BeRX have all of them, and i have a part from his ResuectionX v1.0 ... if i can help.

On gamelist.xml, i think manual must be write like this :

<image>/home/pi/RetroPie/media/megadrive/box front/sonic1.zip</image>
<marquee>/home/pi/RetroPie/media/megadrive/logos/sonic1.zip</marquee>
<video>/home/pi/RetroPie/media/megadrive/videos/sonic1.zip</video>

<manual>/home/pi/RetroPie/media/megadrive/manuals/sonic1.txt</manual>
   or
<manual>/home/pi/RetroPie/media/megadrive/manuals/sonic1.zip</manual>  (images)
   or
<manual>/home/pi/RetroPie/media/megadrive/manuals/sonic1.pdf</manual>

Then you detect if the format, is TXT, PDF or ZIP file, and show it with the good reader.

There are many scan manuals over the net, we must show them and not only PDF.
Same for TXT files, for my part 50% of my files are TXT because i make one TXT for each TRAD or HACK i use to know what is it ... it will be a dream if i can see it on my PI. I also have TXT by games with GAMEFAQS trick or code ...

Exemple files :

Manual Images - Arkanoid
VGMap Images - Adventure Island 3
FAQ Walkthrough - 1943
Trans Infos - Ace Combat 3

Used2BeRX

@darknior Thanks for mentioning me here so I knew the topic was being discussed bud.

I'd love to see this as well. I'm not sure what my opinion is about what format it should be in (if we have to choose only one). Ideally, it would be something that supported multiple formats like @darknior mentioned above.

Things may be different these days, but I don't think that the memory constraints were the issue on the XBox. I think back then there was actually a problem coding any PDF reader to XBMC, inside or outside of the emulators. Adobe had a pretty tight hold on that format for years. Now I've seen 3rd party apps that support the image format, so it may be possible that some of these are open source and can be integrated easily or maybe there's even easier ways of doing it without source given how much more power we have under the hood with the Pi?

But yeah.... our images were originally PDF files, but I found a program to rip them into JPG images which were zipped into a file that matched the rom filename. On the XBox Madmab Edition emulators, you could select "View Game Manual" while playing a game and if there was a manual for the game you were playing it would show up on your screen. You could flip the page, move it around the screen easily and zoom in our out as well. Pretty cool stuff! :)

BTW... the script that @meleu has been constantly improving for me will actually make tags for any <manuals> that are present in the manuals folder. I requested that he add all of the additional fields in his code even though half of them aren't currently supported in EmulationStation. It was my hope that one day stuff like this would happen and we'd eventually see support for them. :)

SuperFromND

Alright, so until someone picks this idea up and begins to work on it, I've developed a workaround that should suffice until then. There's a few caveats to this, but it's the simplest way I could think of doing it. (also long post incoming, so get your scroll wheel prepped up for this.)

The Gist
What I did was I made a new "system" for manuals exclusively, with the following in emulators.cfg:

manual = "fbgs --timeout 3 --once --autozoom -c %ROM% > /dev/null"
default = "manual"

This will take whatever PDF file you selected in EmulationStation, break it down into .tiff files temporarily, then display them in order.

Command Breakdown
fbgs is installed via the ES Theme Gallery, so no sudo apt-get is required. (not sure if it's installed by default, but just in case, install the ES Theme Gallery in Retropie -> ES Themes if you haven't already done so). For those not in the know, fbgs is a program that converts PDF files into a series of .tiff files in a temporary directory, then calls fbi to open them in a sequence. fbi is an image viewing program that RetroPie uses for the ES Theme Gallery, hence why it's installed in the first place.

The first option, timeout 3, is a workaround to a caveat I'll talk about later. The option tells fbi to automatically move to the next image if no keyboard input has been recieved in 3 seconds. The next option, once, tells fbi to not loop the image sequence. The third option, autozoom, tells fbi automatically zoom the image to fit on your screen. The fourth option, -c, tells fbgs to extract the PDF in color, rather than the default monochrome.

%ROM% should be familiar if you've tackled with modifying emulator.cfg files before, but just in case: that's where the selected ROM's directory will be placed.

And finally, > /dev/null tells fbgs to write all of the output text to the special /dev/null directory, which ends up displaying no text on screen.

Alright, now that we've got this wall of text out of the way, let's add this into your RetroPie setup!

How To Add Manuals to ES
Note that, for this tutorial, I'll be using the command line built into RetroPie. Most of what I'm about to describe can be accomplished in GUI form with a program like WinSCP, if you prefer.

Step One: Create a new folder named "manual" in /home/pi/RetroPie/roms, using these two commands:

cd /home/pi/RetroPie/roms
sudo mkdir manual

Then create a folder also named manual in /opt/retropie/configs, like so:

cd /opt/retropie/configs
sudo mkdir manual

Step Two: Go into your newly created manual folder with:

cd ./manual

Open up the Nano text editor by typing:

sudo nano

Type in these two lines:

manual = "fbgs --timeout 3 --once --autozoom -c %ROM% > /dev/null"
default = "manual"

Now press Ctrl+X. Nano will ask if you want to save the file. Press Y, then type in emulators.cfg, then press Enter. Nano will automatically close itself, and your file will be saved. (If you want to make sure, just run ls in /opt/retropie/configs/manual)

Step Three: Now we need to add manuals to our list of systems in EmulationStation. Open es_systems.cfg in Nano with the command:

sudo nano /etc/emulationstation/es_systems.cfg

Scroll to the bottom of the document by holding Down, then press Up go one line above where it says </systemList>. Add in the following:

<system>
    <name>manual</name>
    <fullname>Game Manuals</fullname>
    <path>/home/pi/RetroPie/roms/manual</path>
    <extension>.pdf .PDF</extension>
    <command>/opt/retropie/supplementary/runcommand/runcommand.sh 0 _SYS_ manual %ROM%</command>
    <platform />
    <theme>manual</theme>
</system>

Press Ctrl-X, Y, then Enter to save the file.

Step Four: Add in your game manuals (in PDF format) to /home/pi/RetroPie/roms/manual.
Once you've added at least one, start up Emulationstation by typing in:

emulationstation

You should now have a "manuals" option near (or right next to, if you have Game Collections disabled) the RetroPie configuration option. (it most likely wont have any theme folder, so you're going to have to add that yourself. That's far too complex for me to dive into in this one post, but there are plenty of posts that tell you how to do it out there.)

If you did everything right, when you select your PDF in EmulationStation, it will boot into your PDF for viewing! However...

Caveats
...there's two main issues with this approach to game manuals.

You can only control fbi with a keyboard, the gamepad will not work at all.

This is why --timeout 3 is included in the command in emulators.cfg. My guess is that this has something to do with how fbgs calls fbi, versus how the ES Theme Gallery calls fbi directly.

Two lines of text are outputted by fbi that are never cleared.

This also probably has something to do with how fbgs calls fbi. I've heard that appending a clear or reset command could fix this, but that would happen before the two lines of text were drawn.

Addendum
I completely forgot a minor caveat, and that's that the manuals get added to the All Games and Last Played collections, and there isn't a way to prevent this short of disabling those collections altogether.

Also, if you happen to use the Pixel or MetaPixel themes, I've made a console.png and logo.png for use in themes, along with a launching.png

darknior

@supercatfooz It's really a good job, you found a solution. And i think you lost many time for it...

But it's not fine for me, go and search a manual ... in a special menu ... i will never do it, i know myself.
To use it, the manual must be a media of the game. Like his cover and his video. And stored in his gamelist.xml with other stuff.

The big adventage of the xBox was we can read the manual from the emulator, AND in game !
If you don't understand something, you pause the game and read it, or the walkthrough, or search for a cheat code lol
Here we will must SAVE game, EXIT game to ES, read the Manual, and LAUNCH again the game to continue ... If we must also go to a MANUAL ES menu, and search the game manual to read it, WOAW !!!

Another problem, like i write before many of my manual are TXT or JPG ... not PDF ;)

I prefer stand for a real ES update with manual support ... or not :p

Used2BeRX

@supercatfooz said in [Idea] Game Manual viewing in ES:

@jell I don't think there is a standardized format; I usually just see image files of each individual page, but I've heard of some people using PDF files instead. Personally I think the former would be better for the end user, since PDFs are a pain in the neck to deal with.

I have to agree with you here man.

I looked into this a bit since I saw this thread and found out a few things. There's nothing special about a PDF file except that it is a container with some encryption. Just having the manuals in this format means that you have to use specific image viewers to open them. (Not necessarily Adobe, but likely bloated and necessary either way).

There is no loss of quality when you "convert" the images from PDF to JPG. You're not really converting anything. You're just pulling them out of the container, kind of like unzipping an archive.

The only problem is, converting these images from PDF to JPG and renaming everything and zipping them back up can be a huge task. Ask me how I know.

That being said, I do have thousands of manuals spanning a dozen or so systems that I've already done this to years ago.

If you build it.... they will come. :)

Zigurana

@used2berx said in [Idea] Game Manual viewing in ES:

That being said, I do have thousands of manuals spanning a dozen or so systems that I've already done this to years ago.

When you say 'this', what exactly do you mean?
These manuals are a directory / zip of separate images? What is your naming convention?

Used2BeRX

@zigurana By "this", I mean that they've all been converted into JPG files that are contained in ZIP archives.

Looking at my NES collection which I am now focusing all of my work upgrading, I currently have 724 game manuals for the system and 10 manuals for the console and various peripherals.

The naming convention is for the RessurectionXtras 1.0 set that came out around 10 years ago and covered the roms and various extra media for well over a dozen consoles and handhelds and was designed for use on the XBox. (You were limited to only 42 characters including the extension).

All clones were removed from the set. US games were first, then EU, then Japan. If a game was very notable for having differences in regional releases it may have been added. There were also hundreds of translations, pirates, prototypes, unlicensed games and prototypes.

All media matched the game zipfile name. On the XBox we didn't need a crazy gamelist.xml file. We just needed to point to the directory of the various media and if it had a matching file name it worked with the game.

All that being said.... examples of the naming convention are as follows...

Anticipation.zip
Arch Rivals.zip
Archon.zip

Inside every one of these archives, the pages are named as follows:

01.jpg
02.jpg
03.jpg

In the very rare instance that there were more than 100 pages in a manual, they were written like this:

001.jpg
002.jpg
003.jpg

The NES manual collection I have is 734 archives weighing in at 3.18GB.

I'm not going to list all of them, but some other systems I have:

Atari 2600: 682 files, 677MB
SNES: 196 files, 431MB
GENESIS: 355 files, 881MB

I'm currently in the process of getting my hands on as many collections as I can right now since there seems to be a lot of them being taken down from various sources. I plan on upgrading all of these manuals on a per-system basis, but it will take a very long time. I've been working on the NES collection since the middle of May and I haven't even started working on the NES manuals yet.

Used2BeRX

If anybody has any advice for a good PNG to JPG converter that is free and won't degrade quality, I'm looking for one.

Any of the stuff I had years ago won't work on Windows 10 for sure. I don't want to go the piratebay route and deal with viruses either.

I've tried a few supposed "free" converterters this morning and they're BS. One has watermarks, one limits you to 50 pages and one just seems to hang and do nothing.

mitu

@used2berx Have you tried the ol' ImageMagick ? It's mostly command line, but I guess it's useful if you want to bulk/batch convert images.

Used2BeRX

@mitu OOPS....

I meant a PDF to PNG, or a PDF to JPG converter.

I believe I found what I was looking for. Have to do a few more tests. It was something called "PDF to Image" that just one dude made and he offers it for free, and it's portable so you don't have to even install it.

I was able to take a 26MB PDF of a magazine and turn it into over 2GB of PDF files using 600DPI. Way overkill and not what I'm looking for. lol

I'll have to try out the 300DPI and 72DPI settings and see what happens.

chipsnblip

@used2berx said in [Idea] Game Manual viewing in ES:

@mitu OOPS....

I meant a PDF to PNG, or a PDF to JPG converter.

I believe I found what I was looking for. Have to do a few more tests. It was something called "PDF to Image" that just one dude made and he offers it for free, and it's portable so you don't have to even install it.

I was able to take a 26MB PDF of a magazine and turn it into over 2GB of PDF files using 600DPI. Way overkill and not what I'm looking for. lol

I'll have to try out the 300DPI and 72DPI settings and see what happens.

calibre is a good one, if that doesn't work out