How to create a video upload platform using Ruby on Rails – Part 2

Welcome back to part 2 of How to create a video upload platform using Ruby on Rails. In Part 1, we worked on setting up the basic Ruby on Rails application that allowed us to upload, store, and playback video files from our app. In this next part, we are going to enhance that process some by adding some FFMPEG Transcoding. This way, we can get all videos that are uploaded converted into a common format that will make it easier to work with on the playback side.

So lets get started. The first thing we need to do is make sure we have ffmpeg installed. On mac this is pretty simple using brew. If you don’t have HomeBrew installed, please follow the directions at https://brew.sh/. If you are on another linux or windows system, there are many guides online to help set ffmpeg up for you. Once you have that installed, we can install the ffmpeg command:

brew install ffmpeg

I decided to install a lot of the extra ffmpeg options to make sure I had the best compatibility. Here is a more complete line

brew install ffmpeg --with-chromaprint --with-fdk-aac --with-libass --with-librsvg --with-libsoxr --with-libssh --with-tesseract --with-libvidstab --with-opencore-amr --with-openh264 --with-openjpeg --with-openssl --with-rtmpdump --with-rubberband --with-sdl2 --with-snappy --with-tools --with-webp --with-x265 --with-xz --with-zeromq --with-zimg

Once you have ffmpeg installed and running we need to add the carrierwave-video gem to our project:

# Gemfile
gem "carrierwave-video"

Next, we need to update our upload to include this new library. Open the app/uploaders/video_uploader.rb file and add the following include line to it:

class VideoUploader < CarrierWave::Uploader::Base
 include CarrierWave::Video # <== Add this line here
 
 ...

In order for Carrierwave to actually encode the video, we need to also add a line that instructs it to do so. Lets also adjust the resolution while we are at it.

class VideoUploader < CarrierWave::Uploader::Base
  include CarrierWave::Video

  process encode_video: [:mp4, resolution: "640x480"]

  ...

The last thing we need to do is change the file extension. If we upload an avi or mov file, we convert that to mp4 but with Carrierwave it will maintain the original extension. So add the following lines right after the process line added above

def full_filename(for_file)
  super.chomp(File.extname(super)) + '.mp4'
end

def filename
  original_filename.chomp(File.extname(original_filename)) + '.mp4'
end

Fire up your Rails app and upload a video just like we did in Part 1. You should see the following in your Rails console. Notice the line Running transcoding...

Started POST "/videos" for 127.0.0.1 at 2018-05-17 09:41:10 -0400
Processing by VideosController#create as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"pCgWw3lxulQ9qiGxfck5oqFXgQo0tqO7gG371YEbFaiiRMdAc98uPGQCvTYzn0/Kub/pQtQI9n95I+grmGvQ6w==", "video"=>{"title"=>"adf", "file"=>#<ActionDispatch::Http::UploadedFile:0x007fbb5f814be8 @tempfile=#<Tempfile:/var/folders/__/7ns9m_817l71zfd0xw5618n00000gn/T/RackMultipart20180517-60568-c6cwal.mov>, @original_filename="step.mov", @content_type="video/quicktime", @headers="Content-Disposition: form-data; name=\"video[file]\"; filename=\"step.mov\"\r\nContent-Type: video/quicktime\r\n">}, "commit"=>"Create Video"}
I, [2018-05-17T09:41:10.439284 #60568]  INFO -- : Running transcoding...
["/usr/local/bin/ffmpeg", "-y", "-i", "/Sites/VideoSite/tmp/1526564470-60568-0006-7430/step.mov", "-vcodec", "libx264", "-acodec", "aac", "-s", "640x480", "-r", "30", "-strict", "-2", "-map_metadata", "-1", "-aspect", "1.3333333333333333", "/Sites/VideoSite/tmp/1526564470-60568-0006-7430/tmpfile.mp4"]

I, [2018-05-17T09:41:17.640387 #60568]  INFO -- : Transcoding of /Sites/VideoSite/tmp/1526564470-60568-0006-7430/step.mov to /Sites/VideoSite/tmp/1526564470-60568-0006-7430/tmpfile.mp4 succeeded

   (0.3ms)  begin transaction
  SQL (0.3ms)  INSERT INTO "videos" ("title", "file", "created_at", "updated_at") VALUES (?, ?, ?, ?)  [["title", "adf"], ["file", "step.mp4"], ["created_at", "2018-05-17 13:41:17.643006"], ["updated_at", "2018-05-17 13:41:17.643006"]]
   (1.0ms)  commit transaction
Redirected to http://localhost:3000/videos/1
Completed 302 Found in 7260ms (ActiveRecord: 1.6ms)

 

There we go. We are now successfully encoding the video file to a common format using ffmpeg. In the next part we will look at moving this process to the background so that we don’t force the user to wait on the transcoding, and add some status indications of how the progress is going. Until next time!

How to create a video upload platform using Ruby on Rails – Part 1

Lets create a video upload platform using the popular Ruby on Rails web framework. I have never done a multi-part series before so I thought I would take a stab at it using a small project I did in my spare time. This series might go slow since I have limited time to write these, but hopefully it will be informative.

The small project I was working on was attempting to create a video upload platform using Ruby on Rails. The app would allow you to upload videos and would play them back using Adaptive Bitrate and HLS techniques to help adjust to changing bandwidth requirements. One other thing I added for fun, was a small Node.js server that handles and transcoded videos On-The-Fly when it could, or would fallback to the normal post transcoding process. This allowed the app to receive the upload and transcode at the same time, reducing the amount of time the user had to wait. Since uploading videos has a lot of idle time, why not use that to do some more work.

This walk through will hopefully try and reattempt the work I did with that project and hopefully someone can find fun with it.

We will assume you have a basic rails environment set up and can create new Rails apps:

rails new VideoSite

Let it do its stuff and then lets go into the project.

cd VideoSite

At this point, we have a freshly baked Ruby on Rails application. We need a place where we can upload some videos. We will use the popular Carrierwave gem for handling our uploads. Lets add that to our project and bundle the project:

# Gemfile

gem 'carrierwave'
bundle install

Next we need to create the Uploader using the Carrierwave generator.

rails generate uploader Video

Great. Next lets go ahead and create our controller/model/views for our videos pages and then migrate our database.

rails generate scaffold Video title:string file:string
rake db:migrate

Before we can start uploading videos however, we need to attach the uploader to our video model. Open up app/models/video.rb and change it as follows.

class Video < ApplicationRecord
  mount_uploader :file, VideoUploader
end

Now we can fire up our rails application and upload some videos! Start your rails server with rails s and then navigate to http://localhost:3000/videos/new and you should see our amazing video page.

In order to actually select a video file, we need to update the view to render a file_field instead of a text_field as well as allow multipart uploads. Lets open up the app/views/videos/_form.html.erb file and change it to the following.

<%= form_with(model: video, local: true, html: { multipart: true }) do |form| %>
  <% if video.errors.any? %>
    <div id="error_explanation">
      <h2><%= pluralize(video.errors.count, "error") %> prohibited this video from being saved:</h2>

      <ul>
      <% video.errors.full_messages.each do |message| %>
        <li><%= message %></li>
      <% end %>
      </ul>
    </div>
  <% end %>

  <div class="field">
    <%= form.label :title %>
    <%= form.text_field :title, id: :video_title %>
  </div>

  <div class="field">
    <%= form.label :file %>
    <%= form.file_field :file, id: :video_file %>
  </div>

  <div class="actions">
    <%= form.submit %>
  </div>
<% end %>

We should now be able to choose files and upload them!

Go ahead and select a video and click the Create Video Button. You should then see something like this.

This isn’t really all that fun since we cannot see the video, just the path to it. Lets fix that by using the HTML video tag. Open up the app/views/video/show.html.erb file and change it to the following and then reload the page and you should see your video playing.

<p id="notice"><%= notice %></p>

<p>
  <strong>Title:</strong>
  <%= @video.title %>
</p>

<p>
  <strong>File:</strong>
  <video controls width="640" height="480" src="<%= @video.file %>"></video>
</p>

<%= link_to 'Edit', edit_video_path(@video) %> |
<%= link_to 'Back', videos_path %>

Congratulation! You have the beginning of your very own video uploading service. If you try to load this in other browsers, you may run into problems however, since not all browsers are capable of playing all formats all the time. Another issue is that some web servers will send the entire video file down to the client, which we do not want. It would be a lot nicer if only the part of the video we request is what we receive, continuously. In the next part, we will continue to expand on our simple video uploading by adding some fancy transcoding techniques and use some front end tools that will help us render on all browsers, including mobile and stream smaller chunks of video at a time.

Hope this was fun. Take a look at Part II next!

How to create a Sidekiq service application without Rails

Sometimes you want to create a small Sidekiq service to process requests, without all the extra bloat (and memory usage) from a Rails framework. It turns out that creating such service is not all that hard to do. Create your application using the folders and files below as your starting point.

Gemfile

# Gemfile
source 'https://rubygems.org'
ruby "2.5.0"
git_source(:github) do |repo_name|
  repo_name = "#{repo_name}/#{repo_name}" unless repo_name.include?("/")
  "https://github.com/#{repo_name}.git"
end

gem "sidekiq"

config/sidekiq.yml

---
:concurrency: <%= ENV["CONCURRENCY"] || 25 %>
:queues:
 - default

config/application.rb

# config/application.rb

ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", __FILE__)

require "bundler/setup" if File.exist?(ENV["BUNDLE_GEMFILE"])
Bundler.require

ENV["RACK_ENV"] ||= "development"

module Example
 class Application
   def self.root
     @root ||= File.dirname(File.expand_path('..', __FILE__))
   end

   def self.env
     @env ||= ENV["RAILS_ENV"] || ENV["RACK_ENV"] || ENV["ENV"] || "development"
   end

   def self.logger
     @logger ||= Logger.new(STDOUT)
   end
 end
end

require_relative "initializers/sidekiq"

config/initializers/sidekiq.rb

# config/initializers/sidekiq.rb

Sidekiq.configure_server do |config|
  config.redis = { url: ENV["REDIS_URL"] }
end

app/workers/my_worker.rb

# app/workers/my_worker.rb

class MyWorker
  include Sidekiq::Worker

  def perform(*args)
    # Do work here
  end
end

Once you have created those files, then you can run the following on the command line:

bundle exec sidekiq -e ${RACK_ENV:-development} -r ./config/application.rb -C ./config/sidekiq.yml

If you want to enqueue jobs from another application using Sidekiq, its as easy as:

Sidekiq::Client.enqueue_to "default", MyWorker, "param_1" => "1", "param_2" => "2"

When you have all this working, then you can start adding things like your database connections and other gems and code to the worker. Let me know if this helps anyone or if you run into any problems!

Installing mysql2 Gem error targeting Mac OS X 10.5 or later

I recently tried to install the mysql2 gem and was presented with the following error. I was on El Capitan 10.11.6 and using ruby 1.9.3-p125.

Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.
/Users/…/.rvm/rubies/ruby-1.9.3-p125/bin/ruby extconf.rb
checking for rb_thread_blocking_region()… yes
checking for rb_wait_for_single_fd()… yes
checking for mysql.h… yes
checking for errmsg.h… yes
checking for mysqld_error.h… yes
creating Makefile

make

compiling client.c
couldn’t understand kern.osversion `15.6.0’
compiling mysql2_ext.c
couldn’t understand kern.osversion `15.6.0’
compiling result.c
couldn’t understand kern.osversion `15.6.0’
linking shared-object mysql2/mysql2.bundle
couldn’t understand kern.osversion `15.6.0’
ld: -rpath can only be used when targeting Mac OS X 10.5 or later
collect2: ld returned 1 exit status
make: *** [mysql2.bundle] Error 1

Gem files will remain installed in /Users/…/vendor/bundle/ruby/1.9.1/gems/mysql2-0.3.11 for inspection.
Results logged to /Users/…/vendor/bundle/ruby/1.9.1/gems/mysql2-0.3.11/ext/mysql2/gem_make.out

An error occurred while installing mysql2 (0.3.11), and Bundler cannot continue.
Make sure that `gem install mysql2 -v ‘0.3.11’` succeeds before bundling.

After some searching I found out the solution was to install the gem by setting the Mac Deployment Target as so:

MACOSX_DEPLOYMENT_TARGET=10.9 gem install mysql2

Hope this helps someone else in the future!

Open Source Application Performance App

Its been a while since I have posted anything and I thought I would share with you a small project I have started to work on. Its called app_perf and its a little application that is intended to do application performance monitoring. Right now I am only supporting Ruby via the agent gem, but other languages can easily be added to post metrics to this as well. I encourage anyone interested to clone the project, submit any PR’s and help out making this a lot more full featured. Here is the Github link, https://github.com/randy-girard/app_perf

Here are some screen shots:

Let me know what you all think!

Zero Downtime Delayed Jobs

Here is a script I created that will do Zero Downtime Delayed Job restarts. This script will spin up a new set of delayed jobs workers and send a signal to the currently running workers to shut down after they are finished working on current work. The current script/delayed_job restart would only wait up to some timeout value and then would kill the job, potentially losing any work that it was working on. I store this in a file called delayed_job_restart.sh. Make sure you

chmod +x delayed_job_restart.sh

Feedback is welcome!

##############
CONFIGURATION
##############

# Set the rails environment.
RAILS_ENV=development

# This is the location of your rails application. (Rails.root)
APP_DIR=/path/to/app

# Whatever queues you want to handle.
QUEUES=queue1,queue2

# Number of delayed job instances.
NUMJOBS=6

########################
DONT EDIT BELOW HERE
UNLESS YOU UNDERSTAND IT
########################
# This is the file that will hold the location of
# the new round of delayed job pids.
NEW_PID_DIR_FILE=$APP_DIR/pids/delayed_job_pid_dir.new

# This is the directory where the new delayed job pids will live. We
# create a random directory.
NEW_PID_DIR=$APP_DIR/pids/delayed_job/`cat /dev/urandom | env LC_CTYPE=C tr -cd ‘a-f0-9’ | head -c 32`

# Make the directory and create the
# pid file.
mkdir -p $NEW_PID_DIR
touch $NEW_PID_DIR_FILE

# Put the pid dir into the pid file.
echo $NEW_PID_DIR > $NEW_PID_DIR_FILE

echo “Starting new delayed job processes…”
cd $APP_DIR
RAILS_ENV=$RAILS_ENV bundle exec script/delayed_job start –queues=$QUEUES –pid-dir=`cat $NEW_PID_DIR_FILE` -n $NUMJOBS
echo “Done.”

# This is the current pid dir.
PID_DIR_FILE=$APP_DIR/pids/delayed_job_pid_dir

# If the pid dir exists, let tell them to shut down
# when they are done.
if [ -f $PID_DIR_FILE ]; then
echo “Sending signal to stop old delayed job processes…”
PID_DIR=`cat $PID_DIR_FILE`
for f in $PID_DIR/*.pid
  do
    PID=`cat $f`
    echo ” TERM to $PID”
    kill -TERM $PID
  done
  echo “Done.”
  echo “Removing old PID directory.”
  rm -rf $PID_DIR
fi

# Since we have a set of new jobs running,
# lets move the pid dir to the “current” dir.
mv $NEW_PID_DIR_FILE $PID_DIR_FILE

Faster Rails Tests

Anyone that has worked on a large rails application knows how long it takes for tests to run. I have this same problem as well. I use the build in testing framework packaged with Rails. One thing I noticed was that the Rails environment seemed to be loading between running unit, functional and integration tests. I felt this was unnecessary if I wanted to run the whole test suite, say for CI purposes, so I created a custom rake task that I use:

# lib/tasks/test.rake
require "rake/testtask"

Rake::TestTask.new do |t|
  t.libs << "lib"
  t.libs << "test"
  t.name = "test:ci"
  t.warning = false
  t.verbose = false
  t.test_files = FileList[
    "test/unit/**/*_test.rb",
    "test/functional/**/*_test.rb",
    "test/integration/**/*_test.rb"
  ]
end

This allows me to run all my tests in a single application boot and shave a small amount of time off the full run:

bundle exec rake test:ci

Validating uniqueness of nested model on create

I recently ran into the issue where I setup a uniqueness validator on a field, with a scope on the parent’s ID. This was fine for updating with an existing parent, but if I ever created two or more new fields, the validation would fail because there was no ID on the parent model yet for the child to validate against. This is apparently still an open issue with rails, as found by this url: https://rails.lighthouseapp.com/projects/8994/tickets/2160-nested_attributes-validates_uniqueness_of-fails. There is a solution in that link, but I have came up with a similar method, with less code that seems to work fine and passes all my tests. I have also added to my solution the ability to add an error to the individual nested items that duplicate. This is how I fixed this (note that I make a lot of assumptions with code, just showing the example):

class Child < ActiveRecord::Base
  belongs_to :parent

  validates :value, :uniqueness => { :scope => :parent_id }
end

class Parent < ActiveRecord::Base
  has_many :children
  accepted_nested_attributes_for :children

  validate :uniqueness_of_children
 
  private
  
  def uniqueness_of_children
    hash = {}
    
    children.each do |child|
      if hash[child.value]
        # This line is needed to form the parent to error out,
        # otherwise the save would still happen
        if errors[:"children.value"].blank?
          errors.add(:"children.value", "duplicate error")
        end
     
        # This line adds the error to the child to view in your fields_for
        child.errors.add(:value, "has already been taken")
      end
    
      hash[child.value] = true
    end
  end
end

Let me know if you try this out and it works for you, or if it needs any improvements. Thanks!

How to use Delayed Job to handle your Carrierwave processing

This tutorial builds on my previous post about how to add FFMPEG processing to Carrierwave. Here I will show you my attempt at being able to utilize Delayed::Job to do the heavy lifting of processing when uploading files using Carrierwave. Remember, this could probably use some improvement, but it is a great starting point. So lets begin. The first thing you will need to do is add Delayed::Job to your application:

# Gemfile
gem "delayed_job"

Next you need to create the migration and migrate the database:

rails generate delayed_job
rake db:migrate

Now we get to the good part. Lets create a module to include into Carrierwave that will support holding off on doing the processing until Delayed::Job gets around to it:

# lib/carrier_wave/delayed_job.rb
module CarrierWave
  module Delayed
    module Job
      module ActiveRecordInterface
        def delay_carrierwave
          @delay_carrierwave ||= true
        end

        def delay_carrierwave=(delay)
          @delay_carrierwave = delay
        end

        def perform
          asset_name = self.class.uploader_options.keys.first
          self.send(asset_name).versions.each_pair do |key, value|
            value.process_without_delay!
          end
        end

        private

        def enqueue
          ::Delayed::Job.enqueue self
        end
      end

      def self.included(base)
        base.extend ClassMethods
      end

      module ClassMethods
        def self.extended(base)
          base.send(:include, InstanceMethods)
          base.alias_method_chain :process!, :delay
          ::ActiveRecord::Base.send(
            :include,
            CarrierWave::Delayed::Job::ActiveRecordInterface
          )
        end

        module InstanceMethods
          def process_with_delay!(new_file)
            process_without_delay!(new_file) unless model.delay_carrierwave
          end
        end
      end
    end
  end
end

Awesome! Now we need to tie this into our Uploader:

# app/uploaders/asset_uploader.rb
require File.join(Rails.root, "lib", "carrier_wave", "ffmpeg")
require File.join(Rails.root, "lib", "carrier_wave", "delayed_job") # New

class AssetUploader < CarrierWave::Uploader::Base
  include CarrierWave::Delayed::Job # New
  include CarrierWave::FFMPEG

  # Choose what kind of storage to use for this
  uploader: storage :file

  # Override the directory where uploaded files will be stored.
  # This is a sensible default for uploaders that are meant to be mounted:
  def store_dir
    "#{Rails.root}/uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
  end

  # Add a version, utilizing our processor
  version :bitrate_128k do
    process :resample => "128k"
  end
end

The last thing we have to do is update our model to queue up delayed job:

# app/models/asset.rb
class Asset < ActiveRecord::Base
  mount_uploader :asset, AssetUploader
  
  after_save :enqueue # New
end

There you have it. Now when you create a new Asset, associate a file, and save it, it shouldn’t run the processes, but instead create a Delayed::Job record. Then Delayed::Job should pick it up and run the processors on it. This may not be perfect, but at least its a start! Thanks for reading!

Create FFMPEG processor for Carrierwave in Rails 3

I have had the pleasure of working with the carrierwave gem recently (as opposed to paperclip), and I must say, I am quite the fan. Once major thing I missed however, was the available list of custom user plugins for it, unlike paperclip. I believe this is mostly due to how new and recent carrierwave is. That being said, I put together a simple example of a FFMPEG process that will allow me to resample the bitrate of a file. This should lay the ground work for other features as well. This example is using Rails 3, but should be easily adaptable for 2. Also, make sure you already have FFMPEG installed and running properly. So lets get started: First things first…we need to add the appropriate gems to our Gemfile:

# Gemfile
gem "carrierwave"
gem "streamio-ffmpeg"

Next is the meat and potatoes of this..the actual FFMPEG process for carrierwave. I choose to keep my plugin files in the directory lib/carrierwave. Make sure you have this path included in your application.rb file if you are using rails 3. Here is the code:

# lib/carrierwave/ffmpeg.rb
require "streamio-ffmpeg"
module CarrierWave
  module FFMPEG
    module ClassMethods
      def resample( bitrate )
        process :resample => bitrate
      end
    end
 
    def resample( bitrate )
      directory = File.dirname( current_path )
      tmpfile = File.join( directory, "tmpfile" )
      File.move( current_path, tmpfile )
      file = ::FFMPEG::Movie.new(tmpfile)
      file.transcode( current_path, :audio_bitrate => bitrate)
      File.delete( tmpfile )
    end
  end
end

Good. Now that we have the plugin coded up, we need to include it into our uploader. I already have one mounted to my Asset model. Here is what my AssetUploader now looks like:

# app/uploaders/asset_uploader.rb
require File.join(Rails.root, "lib", "carrier_wave", "ffmpeg")

class AssetUploader < CarrierWave::Uploader::Base
  include CarrierWave::FFMPEG # <= include the plugin
  
  # Choose what kind of storage to use for this uploader:
  storage :file

  # Override the directory where uploaded files will be stored.
  # This is a sensible default for uploaders that are meant to be mounted:
  def store_dir
    "#{Rails.root}/uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
  end
 
  # Add a version, utilizing our processor
  version :bitrate_128k do
    process :resample => "128k"
  end
end

There! Now whenever you add a new file, it should fire off the processor and create a new version. I hope this help anyone still up in the air about how to put together their own plugin/process for carrierwave. Next I will demonstrate how to incorporate Delayed::Job to move these intensive tasks to the background!

1 2 3 5