RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Take and Scrape Your Own Videos

    Scheduled Pinned Locked Moved Ideas and Development
    videosscraperruncommandemulationstatio
    29 Posts 8 Posters 7.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • herb_fargusH
      herb_fargus administrators @meleu
      last edited by

      @meleu it is possible ffmpeg will be in the stretch repositories, which should simplify the process a bit. I don't know the legalities of distribution but I vaguely remember ffmpeg being more restrictive

      If you read the documentation it will answer 99% of your questions: https://retropie.org.uk/docs/

      Also if you want a solution to your problems read this first: https://retropie.org.uk/forum/topic/3/read-this-first

      1 Reply Last reply Reply Quote 0
      • herb_fargusH
        herb_fargus administrators
        last edited by herb_fargus

        Ok. quick update. I am basing my approach off of method one on the take and scrape your own screenshots guide. This is all assuming that people have ffmpeg compiled and retroarch with ffmpeg support compiled as well. The idea is to have an onstart script that will create all the video configs, runcommand configs, and folders for scraping. To lay out my basic idea:

        • Create a videos folder in each rom folder so that it is portable ie when you transfer your roms your scraped data and videos go with you /home/pi/retropie/roms/<system>/videos I've added functions so the video file will be named based on the rom name (I also have a date variable that can be added if one is creating multiple videos)
        • Create the ffmpeg config /opt/retropie/configs/all/retroarch/videos/config.cfg uses iniset so its simple enough to modify any of the fields if you want to change codec configs
        • Have a function based on https://github.com/RetroPie/RetroPie-Setup/blob/master/scriptmodules/helpers.sh#L1125-L1181 that will generate recording configs for each core to be launched from the runcommand menu, though I wonder if it might be possible to see if there is some mechanism for starting/stopping recordings with a hotkey

        Once all the configs are generated by the script, again like the screenshot guide, we can then use sselph's scraper to add the video tags in our xml to match the newly created movies.

        Here's the preliminary script, guaranteed to be filled with bugs as its incomplete but I'd be happy to get any input for improvements.

        #!/usr/bin/env bash
        
        system="$1"
        rompath="$3"
        usbdir="/media/usb/RetroPie/roms/$system/videos"
        sddir="$HOME/RetroPie/roms/$system/videos"
        configdir="/opt/retropie/configs"
        vidconfigdir="$configdir/all/retroarch/videos" 
        
        rom_file="${rompath##*/}"
        rom="${rom_file%.*}"
        date="$(date +%Y-%m-%d-%H%M%S)"
        video="$rom.mkv"
        
        # Create videos folder in each respective rom folder
        #mkdir -p "$usbdir"
        if [[! -d "$sddir" ]]; then
            mkdir -p "$sddir"
        fi
        
        # Create video config file for ffmpeg $vidconfigdir/config.cfg
        
        source "/opt/retropie/lib/inifuncs.sh"
        
        if [[ ! -f "$vidconfigdir/config.cfg" ]]; then
         iniConfig " = " "\"" "$vidconfigdir/config.cfg"
            iniSet "format" "matroska"
            iniSet "threads" "3"
            iniSet "vcodec" "libx264rgb"
            iniSet "video_preset" "ultrafast"
            iniSet "video_tune" "animation"
            iniSet "pix_fmt" "bgr24"
            iniSet "video_qp" "0"
            iniSet "acodec" "flac"
        fi
        
        #if lr- copy lr and append video prefix to copied lr- and then append --record AND --recordconfig after .so, still needs tweaking
        if ! grep -q "recordconfig" "$configdir/$system/emulators.cfg"; then
            sed -i -e '/^lr-/ p' -e '/^lr/s=^=video-=' -e 's@.so@.so --record '$usbdir/"$video"' --recordconfig '$vidconfigdir/config.cfg'@g' "$configdir/$system/emulators.cfg"
        fi
        
        ###This is currently not complete, but this will add the correct launch perameters to retroarch
        # Create lines in emulators.cfg for the emulator you are using to enable video recording with --record and --recordconfig:
        #if [[ "$id" == lr-* && "$cmd" != "$emudir/retroarch/bin/retroarch"* ]]; then
        #    cmd="$emudir/retroarch/bin/retroarch -L $cmd --record $usbdir/$system/$video --recordconfig $configdir/config.cfg --config $md_conf_root/$system/retroarch.cfg %ROM%"
        #fi
        
        #--record $usbdir/$system/$video
        #--recordconfig $configdir/config.cfg
        

        I thought I would have options for and sd card directory or a usb directory but tbh if people have mounted their usb correctly the path should be the same...

        The main issues I see right now is it hates spaces in the --record filepath and possibly parentheses too. I also need to find a mechanism so that the rom name is dynamically changed each time a new game is loaded up.

        Sselphs scraper

        essentially same process as before with images:

        cd /home/pi/RetroPie/roms/nes
        /opt/retropie/supplementary/scraper/scraper -video_dir=videos -video_path=videos -video_suffix="" -add_not_found=true -download_images=false
        

        If you read the documentation it will answer 99% of your questions: https://retropie.org.uk/docs/

        Also if you want a solution to your problems read this first: https://retropie.org.uk/forum/topic/3/read-this-first

        meleuM 1 Reply Last reply Reply Quote 0
        • meleuM
          meleu @herb_fargus
          last edited by

          @herb_fargus I will have a chance to dive into it from Jun-14 on. I hope you don't lose momentum until then. :-)
          I'm sure we'll get it working!

          • Useful topics
          • joystick-selection tool
          • rpie-art tool
          • achievements I made
          1 Reply Last reply Reply Quote 0
          • herb_fargusH
            herb_fargus administrators
            last edited by

            sounds good. I just have to figure out the emulators.cfg, then it should hopefully be functional, then we can focus on optimisations. I don't imagine this will be something for the average user though

            If you read the documentation it will answer 99% of your questions: https://retropie.org.uk/docs/

            Also if you want a solution to your problems read this first: https://retropie.org.uk/forum/topic/3/read-this-first

            1 Reply Last reply Reply Quote 0
            • herb_fargusH
              herb_fargus administrators
              last edited by

              Well I'm a noob. My method of going about this is all wrong and incredibly overengineered. I'm going to give it a break for a bit so fresh eyes can look at it.

              I'm thinking the correct way to go about it would be to have a launch perameter in runcommand like we do for netplay or verbose logging except have it be for running a core with recording enabled.

              I imagine that once Raspbian updates to stretch with ffmpeg in the repository, this could be something included in retropie by default and much of my script would then be redundant (hopefully!)

              I was able to get sselphs scraper to scrape the videos correctly, but now the trouble is encoding it with a compatible codec that will play in emulationstation. Baby steps!

              If you read the documentation it will answer 99% of your questions: https://retropie.org.uk/docs/

              Also if you want a solution to your problems read this first: https://retropie.org.uk/forum/topic/3/read-this-first

              fellegF meleuM 2 Replies Last reply Reply Quote 1
              • fellegF
                felleg @herb_fargus
                last edited by felleg

                [created my own thread to ask for help about scraping videos]

                meleuM 1 Reply Last reply Reply Quote 0
                • meleuM
                  meleu @felleg
                  last edited by

                  @felleg please open another topic with your issue. It's unrelated to this topic (scrape user created video).

                  • Useful topics
                  • joystick-selection tool
                  • rpie-art tool
                  • achievements I made
                  1 Reply Last reply Reply Quote 0
                  • N.A.R.E.K.96N
                    N.A.R.E.K.96
                    last edited by

                    this will be great especially for making youtube videos

                    1 Reply Last reply Reply Quote 0
                    • meleuM
                      meleu @herb_fargus
                      last edited by

                      @herb_fargus said in Take and Scrape Your Own Videos:

                      I'm thinking the correct way to go about it would be to have a launch perameter in runcommand like we do for netplay or verbose logging except have it be for running a core with recording enabled.

                      Since the recording is not an official feature, I think it wouldn't be a good way to go. But I've got an idea: what about a runcommand-onmenu.sh?

                      A script to be launched only if user invoke runcommand menu and choose the option to launch it. I'll try to submit a PR and let you guys review the code. I can think other use cases for such feature...

                      • Useful topics
                      • joystick-selection tool
                      • rpie-art tool
                      • achievements I made
                      herb_fargusH 1 Reply Last reply Reply Quote 1
                      • herb_fargusH
                        herb_fargus administrators @meleu
                        last edited by

                        @meleu it may not be official now but if/when we update to stretch and ffmpeg is in the repos rather than requiring a source install then yes it may actually be viable to include by default but in the interim yes onmenu is a good solution (which I was unaware of) I await your solution

                        If you read the documentation it will answer 99% of your questions: https://retropie.org.uk/docs/

                        Also if you want a solution to your problems read this first: https://retropie.org.uk/forum/topic/3/read-this-first

                        meleuM 1 Reply Last reply Reply Quote 0
                        • meleuM
                          meleu @herb_fargus
                          last edited by

                          @herb_fargus said in Take and Scrape Your Own Videos:

                          onmenu is a good solution (which I was unaware of)

                          runcommand-onmenu.sh feature doesn't exist yet. I'll try to submit a PR for runcommand.sh implementing this later today.

                          • Useful topics
                          • joystick-selection tool
                          • rpie-art tool
                          • achievements I made
                          herb_fargusH 1 Reply Last reply Reply Quote 0
                          • herb_fargusH
                            herb_fargus administrators @meleu
                            last edited by herb_fargus

                            @meleu how would the on menu be any different than onstart unless there is some mechanism for modifying the dialog menu for runcommand?

                            Or are you saying it's an option in the runcommand menu to run a script?

                            If you read the documentation it will answer 99% of your questions: https://retropie.org.uk/docs/

                            Also if you want a solution to your problems read this first: https://retropie.org.uk/forum/topic/3/read-this-first

                            meleuM 1 Reply Last reply Reply Quote 0
                            • meleuM
                              meleu @herb_fargus
                              last edited by meleu

                              @herb_fargus runcommand-onmenu.sh would be a script to be launched only if user invoke runcommand menu and choose the option to launch it. runcommand-onstart.sh is always executed before the game is launched.

                              edit
                              by the way, I've submitted the PR and tagged you there: https://github.com/RetroPie/RetroPie-Setup/pull/2009

                              • Useful topics
                              • joystick-selection tool
                              • rpie-art tool
                              • achievements I made
                              herb_fargusH 1 Reply Last reply Reply Quote 1
                              • herb_fargusH
                                herb_fargus administrators @meleu
                                last edited by herb_fargus

                                @meleu Necropost AHOY!

                                • With the advent of raspbian stretch and subsequently ffmpeg integrated in the repos

                                • the runcommand onmenu functions integrated

                                • and Retroarch compiled with ffmpeg support:

                                I think I'm ready to give this another go. The integration is all pretty much handled as far as no more need for source compilations, now the trick is just configuring the onstart/onmenu scripts and optimal settings for ffmpeg to capture native res gameplay without degrading performance.

                                Runcommand OnMenu Options

                                As far as onstart/omenu scripts go, a few options:

                                • OnMenu that just launches the default config to the default video path with the saved video file being named after the rom.
                                • OnMenu that creates a videos folder for the system being launched, uses the default config, and saves the video in the created videos folder named after the rom.
                                • OnMenu with dialog options to select the saved video path and video config/container (mkv, mp4 etc), and maybe options to name based on rom, date, rom+date, etc.

                                Runcommand OnMenu

                                Per the docs, create a script called recordvideo.sh in `

                                opt/retropie/configs/all/runcommand-menu/`
                                

                                With the contents todo incomplete!

                                recordvideos.sh

                                #!/usr/bin/env bash
                                
                                system="$1"
                                emulator="$2"
                                rompath="$3"
                                launch="$4"
                                rom_file="${rompath##*/}"
                                rom="${rom_file%.*}"
                                mediadir="$HOME/RetroPie/roms/$system/images"
                                videofile="$mediadir/$rom-video.mkv"
                                
                                
                                # Create videos folder in system's rom folder
                                if [[ ! -d "$mediadir" ]]; then
                                    mkdir -p "$mediadir"
                                    dialog --msgbox "$mediadir created!" 20 60 >/dev/tty
                                fi
                                
                                # Launch to record gameplay on RetroArch Systems.
                                if [[ "$launch" =~ retroarch ]]; then
                                eval "$launch" "--record '$videofile'" 
                                exit 1
                                    else
                                    dialog --msgbox "Video recording is not supported for this system" 20 60 >/dev/tty
                                fi
                                

                                if you want to force ~30 second limits for your videos you can substitute the following code:

                                eval "$launch" "--record '$videofile'" & sleep 33
                                killall retroarch
                                exit 1
                                

                                Configs

                                The default path for recorded videos is:

                                /opt/retropie/configs/all/retroarch/records/
                                

                                If you add a custom config for ffmpeg the default path is:

                                /opt/retropie/configs/all/retroarch/records_config/
                                

                                Examples:

                                default.cfg:

                                vcodec = libx264rgb
                                pix_fmt = bgr24
                                acodec = flac
                                format = matroska
                                video_qp = 0
                                

                                optimised.cfg:

                                format = matroska
                                threads = 3
                                vcodec = libx264rgb
                                video_preset = ultrafast
                                video_tune = animation
                                pix_fmt = bgr24
                                video_qp = 0
                                acodec = flac
                                

                                mp4.cfg:

                                vcodec = libx264rgb
                                pix_fmt = bgr24
                                acodec = aac
                                format = mp4
                                video_qp = 0
                                

                                twitch.cfg:

                                vcodec = libx264
                                acodec = libmp3lame
                                pix_fmt = yuv420p
                                scale_factor = 2
                                threads = 3
                                video_crf = 25
                                video_preset = superfast
                                video_tune = animation
                                audio_global_quality = 75
                                sample_rate = 44100
                                format = flv
                                

                                If you read the documentation it will answer 99% of your questions: https://retropie.org.uk/docs/

                                Also if you want a solution to your problems read this first: https://retropie.org.uk/forum/topic/3/read-this-first

                                ClydeC V 2 Replies Last reply Reply Quote 6
                                • ClydeC
                                  Clyde @herb_fargus
                                  last edited by

                                  Hello @herb_fargus, I'm thinking about streaming my Retropie's screen to my Kubuntu desktop PC that's connected to my video projector, so that an audience of guests could watch the games. Can you kindly give me some hints how to set that up?

                                  herb_fargusH 1 Reply Last reply Reply Quote 0
                                  • herb_fargusH
                                    herb_fargus administrators @Clyde
                                    last edited by

                                    @clyde haven't the foggiest. Could try the twitch thing, or there is some experimental streaming thing that doesn't use a vnc server. Can't remember what it is, never tried it myself though I imagine there would be some latency

                                    If you read the documentation it will answer 99% of your questions: https://retropie.org.uk/docs/

                                    Also if you want a solution to your problems read this first: https://retropie.org.uk/forum/topic/3/read-this-first

                                    ClydeC 1 Reply Last reply Reply Quote 0
                                    • ClydeC
                                      Clyde @herb_fargus
                                      last edited by Clyde

                                      @herb_fargus Well, thanks anyway. I may look further into it when I have the time and leisure, since it isn't urgent at the moment. Latency wouldn't be a big issue as long as it's not more than a few seconds, since the audience wouldn't play but just watch. edit: On the players' side, there should be as little lag as possible, of course.

                                      herb_fargusH 1 Reply Last reply Reply Quote 0
                                      • herb_fargusH
                                        herb_fargus administrators @Clyde
                                        last edited by

                                        @clyde I started a post specific to the whole streaming thing and I think I came up\with a viable software solution, the uv4l raspidisp driver / webrtc server enables you to stream video with not a terrible latency that can be accessed from any web browser on the same network. Audio may be a bit of a trick to sort but I've tested it and it functions.

                                        Further discussion on this thread if interested.

                                        If you read the documentation it will answer 99% of your questions: https://retropie.org.uk/docs/

                                        Also if you want a solution to your problems read this first: https://retropie.org.uk/forum/topic/3/read-this-first

                                        ClydeC 1 Reply Last reply Reply Quote 1
                                        • ClydeC
                                          Clyde @herb_fargus
                                          last edited by

                                          @herb_fargus Wow, that's great. I'll check it out.

                                          1 Reply Last reply Reply Quote 0
                                          • V
                                            Valente @herb_fargus
                                            last edited by

                                            Great post. Unfortunately, I'm having an issue with the recording function on the RetroPie while recording gameplay. When launching an NES game, the emulator is really slow.

                                            Has anyone encountered this problem?

                                            Thanks,

                                            Valente

                                            V 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.