Red Artisan

Provisioning a brand new server or VPS slice can be quite tricky, tedious and time consuming, particularly if done manually with changing software versions and configurations.

In the Rails world, most of us are using virtual private servers which are instantiated from base operating system images, it takes only a few minutes to create a slice, however installing the rest of your server's stack, be it Rails, Merb or another framework is where the work begins. Provisioning in this sense, is installing all software required post operating system install.

Sprinkle is a new prototype tool that you can use to provision your servers/slices. Its declarative policy/state based approach for specifying how a remote system should be provisioned with intelligent logic to support dependencies, multiple installer types and remote installation is really compelling.

Several free and commercially available tools already exist to help automate the installation of software however most fall into two styles of design:

1 - Task based, where the tool issues a list of commands to run on the remote system, either remotely via a network connection or smart client.
2 - Policy/state based, where the tool determines what needs to be run on the remote system by examining its current and final state.

Task based solutions are usually quite easy and fast to get up and running, but can be problematic as the user has to define all of the commands manually (not to mention get them right with testing). Policy/state based solutions have much more intelligence about how to modify and adapt the remote system, but often require specialized software to run remotely.

Sprinkle is a prototype tool I've been working on recently in this space that merges both concepts together, using a Ruby domain specific language to declaratively describe the state of the remote system. Using Sprinkle, provisioning your brand new remote server or slice can be automated using pre-defined and/or customized scripts from a single machine at your fingertips.

Sprinkle reads a script that defines a set of packages, a set of policies that define what packages should be installed on what roles of target machines, and a deployment section that defines the delivery mechanism for communicating with remote machines, and any default settings.

Packages can have relationships between each other to support dependencies. Virtual packages are also supported allowing you to define a role that a package (or multiple) fulfills, with the user or Sprinkle selecting which concrete package should be used at runtime.

Packages can also support arbitrary installer types, allowing you to install packages from source, gems, apt, or any other installer you'd like to employ. Installer types know what commands need to be issued to install packages, so all that needs to be specified in a script is the installer type and metadata about the package itself.

In essence, Sprinkle is about defining a domain specific meta-language for describing and processing the installation of software.

Example Sprinkle Script

Here's an example Sprinkle deployment script, annotated to explain each section:

# Annotated Example Sprinkle Rails deployment script
#
# This is an example Sprinkle script configured to install Rails from Gems, Apache, Ruby and
# Sphinx from source, and MySQL from APT on an Ubuntu system.
#
# Installation is configured to run via capistrano (and an accompanying deploy.rb recipe script).
# Source based packages are downloaded and built into /usr/local on the remote system.
#
# A sprinkle script is separated into 3 different sections. Packages, policies and deployment:


# Packages (separate files for brevity)
#
#  Defines the world of packages as we know it. Each package has a name and
#  set of metadata including its installer type (eg. apt, source, gem, etc). Packages can have
#  relationships to each other via dependencies.

require 'packages/essential'
require 'packages/rails'
require 'packages/database'
require 'packages/server'
require 'packages/search'


# Policies
#
#  Names a group of packages (optionally with versions) that apply to a particular set of roles:
#
#   Associates the rails policy to the application servers. Contains rails, and surrounding
#   packages. Note, appserver, database, webserver and search are all virtual packages defined above.
#   If there's only one implementation of a virtual package, it's selected automatically, otherwise
#   the user is requested to select which one to use.

policy :rails, :roles => :app do
  requires :rails, :version => '2.0.2'
  requires :appserver
  requires :database
  requires :webserver
  requires :search
end


# Deployment
#
#  Defines script wide settings such as a delivery mechanism for executing commands on the target
#  system (eg. capistrano), and installer defaults (eg. build locations, etc):
#
#   Configures sprinkle to use capistrano for delivery of commands to the remote machines (via
#   the named 'deploy' recipe). Also configures 'source' installer defaults to put package gear
#   in /usr/local

deployment do

  # mechanism for deployment
  delivery :capistrano do
    recipes 'deploy'
  end

  # source based package installer defaults
  source do
    prefix   '/usr/local'
    archives '/usr/local/sources'
    builds   '/usr/local/build'
  end

end

# End of script, given the above information, Spinkle will apply the defined policy on all roles using the
# deployment settings specified.

Given such a script, Sprinkle will apply the defined policy rails on the target machines identified by the role app, where the policy rails is composed of packages for the rails gem itself, an application server, webserver, search daemon, and the ruby runtime.

Currently, Sprinkle uses Capistrano internally for communicating with remote systems, however this is pluggable as well, allowing for just about any concievable delivery mechanism in the future. The deployment section above identifies Capistrano as the delivery mechanism, specifying a local deploy.rb script that defines what roles are available, and what machines are defined within those roles.

This particular script breaks the package section up into multiple files, here are some of the actual package definitions (complete example available here):

package :ruby do
  description 'Ruby Virtual Machine'
  version '1.8.6'
  source "ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-#{version}-p111.tar.gz" # implicit :style => :gnu
  requires :ruby_dependencies
end

package :rubygems do
  description 'Ruby Gems Package Management System'
  version '1.0.1'
  source "http://rubyforge.org/frs/download.php/29548/rubygems-#{version}.tgz" do
    custom_install 'ruby setup.rb'
  end
  requires :ruby
end

package :rails do
  description 'Ruby on Rails'
  gem 'rails'
  version '2.0.2'
end

package :sphinx, :provides => :search do
  description 'MySQL full text search engine'
  version '0.9.8-rc2'
  source "http://www.sphinxsearch.com/downloads/sphinx-#{version}.tar.gz"
  requires :mysql_dev
end

package :apache, :provides => :webserver do
  description 'Apache 2 HTTP Server'
  version '2.2.6'
  source "http://apache.wildit.net.au/httpd/httpd-#{version}.tar.bz2" do
    enable %w( mods-shared=all proxy proxy-balancer proxy-http rewrite cache headers ssl deflate so )
    prefix "/opt/local/apache2-#{version}"
    post :install, 'install -m 755 support/apachectl /etc/init.d/apache2', 'update-rc.d -f apache2 defaults'
  end
  requires :apache_dependencies
end

Each package includes a description, optional version, optional list of dependencies and an installer type (also optional allowing for meta-packages).

Source installers are particularly intelligent and will download, configure and install source archives from a remote location directly on the target machine. They assume GNU style source archives by default (ie. tar.gz/tar.bz2 compressed archives, configure script and make, make install style semantics), however are completely customziable to support any arbitrary build style (rubygems for example does this above), with pre and post commands.

The Apache installer for example, specifies a few extra source installer options such as a set of --enable options, an alternate installation prefix and a series of post installation commands to be executed.

With this example configuration, lets take a look at actually using Sprinkle to provision a remote server.

Usage

Sprinkle supports several command line options:

Usage
=====

$> sprinkle [options]

Options are:

  -s, --script=PATH                Path to a sprinkle script to run
  -t, --test                       Process but don't perform any actions
  -v, --verbose                    Verbose output
  -c, --cloud                      Show powder cloud, ie. package hierarchy and installation order
  -h, --help                       Show this help message.

where you can name the script to be procesed, enable testing mode or verbose output, and/or examine the cloud of packages and operations that will be performed.

Viewing the powder cloud!

Sprinkle calculates all operations to be performed on remote servers upfront which is nice, as it allows you to inspect what modifications will be made to the system before any are actually performed. Lets inspect the powder (ie. package) cloud for the above script:

$> sprinkle -c -t -s rails.rb
--> Cloud hierarchy for policy rails

Policy rails requires package rails
        Package rails requires rubygems
                Package rubygems requires build_essential
                Package rubygems requires ruby
                        Package ruby requires build_essential
                        Package ruby requires ruby_dependencies

Policy rails requires package appserver
Selecting mongrel_cluster for virtual package appserver
        Package mongrel_cluster requires rubygems
                Package rubygems requires build_essential
                Package rubygems requires ruby
                        Package ruby requires build_essential
                        Package ruby requires ruby_dependencies
        Package mongrel_cluster requires mongrel
                Package mongrel requires rubygems
                        Package rubygems requires build_essential
                        Package rubygems requires ruby
                                Package ruby requires build_essential
                                Package ruby requires ruby_dependencies

Policy rails requires package database
Selecting mysql for virtual package database

Policy rails requires package webserver
Selecting apache for virtual package webserver
        Package apache requires build_essential
        Package apache requires apache_dependencies

Policy rails requires package search
Selecting sphinx for virtual package search
        Package sphinx requires build_essential
        Package sphinx requires mysql_dev

--> Normalized installation order for all packages: build_essential, ruby_dependencies, ruby, rubygems, rails, mongrel, mongrel_cluster, mysql, apache_dependencies, apache, mysql_dev, sphinx

-c indicates that Sprinkle should print the powder cloud (ie. the output above)
-t indicates that we're operating in test mode, so we won't actually perform any remote commands
-s identifies the Sprinkle script that should be processed

Above we can see that the policy rails required packages rails, appserver, database, webserver and search.

Note that all of these packages bar rails are actually virtual packages, so Sprinkle has selected an appropriate implementation of each virtual package automatically based on the supplied package definitions. If more than one package provided an implementation of a virtual package, then the user would be given the opportunity to select which one they prefer.

Under each package is a textual representation of that package's dependency tree, including all sub-dependencies, etc. Dependencies are packages that need to be installed first before a higher level package can be installed.

You'll notice that several packages have the same dependencies, eg. both rails and mongrel require ruby, which has its own dependencies as well. Sprinkle will install all packages in reverse dependency order so that lower level dependencies are installed before higher level packages, and it will also normalize the final package list to remove duplicates so that packages aren't installed multiple times unnecessarily. This is the final line in the output above which lists the actual packages to be installed and order of installation.

Provisioning a remote system

To actually provision a remote server we simply remove the testing (and if desired cloud) flags from the command issued above and Sprinkle will process the configuration and provision the remote system. Note for the moment, you'll need to ensure that your SSH keys are appropriately installed on the remote server under a user that has enough privileges to install software (generally the root user):

$> sprinkle -s rails.rb
--> Installing build_essential for roles: app
--> Installing ruby_dependencies for roles: app
--> Installing ruby for roles: app
--> Installing rubygems for roles: app
--> Installing rails for roles: app
--> Installing mongrel for roles: app
--> Installing mongrel_cluster for roles: app
--> Installing mysql for roles: app
--> Installing apache_dependencies for roles: app
--> Installing apache for roles: app
--> Installing mysql_dev for roles: app
--> Installing sphinx for roles: app

(its also possible to put the #!/usr/bin/env sprinkle -c line at the top of a sprinkle script and make it executable).

After the command is finished, all of the requested software will have been applied on your target system.

If you'd like to see more action printed as commands are run, specify the --verbose (-v) flag. Internally, Capistrano tasks are dynamically defined and executed at runtime for each package's installation, using a Capistrano configuration file to identify the actual roles and hostnames associated with those roles to communicate with. The verbose option will display Capistrano activity in addition to the usual Sprinkle output.

An extra benefit of leveraging Capistrano is that you can actually provision multiple servers/slices simultaneously and in parallel if desired.

I want!

If you're interested in downloading and experimenting with Sprinkle, you can clone and/or watch the project at GitHub, or download it from GitHub's gem server using:

$> sudo gem install crafterm-sprinkle --source http://gems.github.com/

The official Rubyforge gem server will also be updated over the coming days as well. If you download the source, you can create a gem package for installation by:

$> rake package
$> sudo gem install -l pkg/sprinkle-0.1.0

There are also specs with a decent amount of coverage over the code base that you can run as well:

$> rake spec

Prerequsites

Installing the Sprinkle gem will also install all pre-requsite gems such as activesupport, highline and capistrano. The only other pre-requisite is that you have SSH connectivity to the remote system you wish to provision, preferably with SSH keys in place to prevent passwords being asked for.

Finally

Sprinkle is a young project and while operational still in development, with several limitations. Currently, only Ubuntu/Debian has been tested as a target deployment platform, operating system abstraction and other platforms will be tested and supported in the future, along with several new features that are in the pipeline.

I'm most certainly open to ideas, suggestions and thoughts about how Sprinkle can be improved and generally made better for the community, and I really welcome any bug reports, patches and suggestions. Please feel free to contact me with any comments at all.

Special Thanks!

Several people have been really helpful during the development of Sprinkle. In particular I'd like to thank Ben Schwarz and Pete Yandell for their initial feedback and help after my first demos. I'd also really like to thank Matthew and Jared from Slicehost for their help and support as well.

In a previous article I described how to use Core Image as the backend image processor for Attachment Fu in your Rails applications. In that particular article we looked at supporting image scaling and thumbnails to be compatible with the other Attachment Fu backends such as RMagick and ImageScience.

With Core Image available however, we have an entire range of post processing filters available to use at our fingertips. In this article we’ll step through a few of these additional filter options that you can use to post process your images with.

Here’s a few examples of what we can do with Core Image post file upload. All of the following examples use the following input image taken in Berlin while at RailsConf EU (also used in the performance measurements article):

Greyscale or Sepia

A scaled version of the source image is cool, but how about an automatic greyscale or sepia version of the image:

Greyscale

Sepia

Code Fragment

module RedArtisan
  module CoreImage
    module Filters
      module Color

        def greyscale(color = nil, intensity = 1.00)
          create_core_image_context(@original.extent.size.width, @original.extent.size.height)

          color = OSX::CIColor.colorWithString("1.0 1.0 1.0 1.0") unless color

          @original.color_monochrome :inputColor => color, :inputIntensity => intensity do |greyscale|
            @target = greyscale
          end
        end

        def sepia(intensity = 1.00)
          create_core_image_context(@original.extent.size.width, @original.extent.size.height)

          @original.sepia_tone :inputIntensity => intensity do |sepia|
            @target = sepia
          end
        end

      end
    end
  end
end

Exposure and Noise Control

Another option for us is to automatically adjust exposure and noise parameters upon upload to brighten images up, or remove unwanted noise from lower quality images:

1 F-Stop

2 F-Stops

Noise Removal

Code Fragment

module RedArtisan
  module CoreImage
    module Filters
      module Quality

        def reduce_noise(level = 0.02, sharpness = 0.4)
          create_core_image_context(@original.extent.size.width, @original.extent.size.height)

          @original.noise_reduction :inputNoiseLevel => level, :inputSharpness => sharpness do |noise_reduced|
            @target = noise_reduced
          end
        end

        def adjust_exposure(input_ev = 0.5)
          create_core_image_context(@original.extent.size.width, @original.extent.size.height)

          @original.exposure_adjust :inputEV => input_ev do |adjusted|
            @target = adjusted
          end          
        end

      end
    end
  end
end

Watermarking

Sometimes we’d like to automatically add a watermark to our images, either with a single watermark image, or as a tiled watermark image:

Single Watermark

Tiled Watermark

Code Fragment

module RedArtisan
  module CoreImage
    module Filters
      module Watermark

        def watermark(watermark_image, tile = false, strength = 0.1)
          create_core_image_context(@original.extent.size.width, @original.extent.size.height)

          if watermark_image.respond_to? :to_str
            watermark_image = OSX::CIImage.from(watermark_image.to_str)
          end

          if tile
            tile_transform = OSX::NSAffineTransform.transform
            tile_transform.scaleXBy_yBy 1.0, 1.0

            watermark_image.affine_tile :inputTransform => tile_transform do |tiled|
              tiled.crop :inputRectangle => vector(0, 0, @original.extent.size.width, @original.extent.size.height) do |tiled_watermark|
                watermark_image = tiled_watermark
              end
            end
          end

          @original.dissolve_transition :inputTargetImage => watermark_image, :inputTime => strength do |watermarked|
            @target = watermarked
          end
        end

      end
    end
  end
end

Funky Effects

We can also use cool and funky effects used in applications like Photobooth, here’s an example using the edge colouring algorithm:

Edges

Core Fragment

module RedArtisan
  module CoreImage
    module Filters
      module Effects

        def edges(intensity = 1.00)
          create_core_image_context(@original.extent.size.width, @original.extent.size.height)

          @original.edges :inputIntensity => intensity do |edged|
            @target = edged
          end
        end

      end
    end
  end
end

The sign artwork works particularly well with this algorithm.

Core Image Processor

All of the code above is also available as a usable image processor via git.

Some examples of using the processor:

require 'red_artisan/core_image/processor'

# generate some test output images for various effects

processor = RedArtisan::CoreImage::Processor.new('berlin.jpg')

grey = processor.greyscale
grey.save 'results/berlin-grey.jpg'

sepia = processor.sepia
sepia.save 'results/berlin-sepia.jpg'

watermarked = processor.watermark('watermark_image.png')
watermarked.save 'results/berlin-watermarked.jpg'

watermarked = processor.watermark('watermark_image.png', true)
watermarked.save 'results/berlin-watermarked-tiled.jpg'

noise_reduced = processor.reduce_noise
noise_reduced.save 'results/berlin-noise-reduced.jpg'

exposure_adjusted = processor.adjust_exposure
exposure_adjusted.save 'results/berlin-exposure-adjusted-half-stop.jpg'

exposure_adjusted = processor.adjust_exposure(2.0)
exposure_adjusted.save 'results/berlin-exposure-adjusted-two-stops.jpg'

edge = processor.edges
edge.save 'results/berlin-edge.jpg'

Summary

The above shows us only a fraction of what can be done with the 100+ filters Core Image provides by default. There’s many other filters that let you create all sorts of effects with single and multiple images combined. Enjoy!

Recently I’ve been doing lots of work with databases full of existing data and often have needed to extract data out to into YAML files to use either as bootstrap data (particularly images coming from attachment_fu that need to be preloaded after deployment) or as fixtures (yep, sometimes the data is just too complex to mock).

To ease the process I’ve created a script that I use inside of TextMate, which lets me convert a CSV exported file from CocoaMySQL to a YAML file, suitable for the db:bootstrap rake task or fixture data, all at the close distance of a keyboard shortcut.

First, to get the data out of the database, open up CocoaMySQL, select your table and view its content, and go to ‘File -> Export -> Table Content Result -> Text File’.

Save the exported contents.

Then you’ll need the following script:

#!/usr/bin/env ruby
#
# Format CSV output of cocoamysql into a fixture file
#
# "id","name","mime_type","extensions","icon_url"
# "1","unknown","unknown/unknown","||","/images/icon/file_unknown.gif"
# "2","image/tiff","image/tiff","|tiff|tif|","/images/icon/blank.png"

require 'csv'

class String
  def unquote
    self.gsub(/^"|"$/, '')
  end
end

# first line contains the field names
line = gets
fields = line.split('","').collect {|f| f.unquote.chomp}

CSV::Reader.parse(STDIN) do |row|
  fixture = "#{row[1].downcase}_#{row[0]}:\n"
  fields.each_with_index do |field, i|
    fixture += "  #{field}: #{row[i]}\n"
  end

  puts fixture; puts
end

Which we’ll add as a new ‘Command’ inside of a TextMate bundle.

To do this, open up TextMate, and select the ‘Edit Commands’ menu item

Then, in a custom bundle add a new command (+ button along the bottom of the dialog) and paste the contents of the script above.

Select ‘Entire Document’ as the input source, and ‘New Document’ as the output, this will ensure that you don’t overwrite your CSV file incase you require it later.

Then we need to reload the bundles so that TextMate knows about your new command. To do this, select the ‘Reload Bundles’ menu item

And we’re all set to go. Open up your CSV file, and convert it to a YAML file either via the keyboard shortcut defined in your Command (mine was ctrl-shift-cmd-F) or the menu item from your custom bundle and voila, a converted YAML document will appear which you can save directly to db/bootstrap or your fixtures directory. Enjoy.

RailsConf has been an absolute blast, with many very cool sessions about all kinds of topics. There’s also been quite a few late nights in Berlin with Rails peeps after hours as well :)

David’s keynote was great, particularly the discussion of how Rails has progressed from its early development stages to the form it’s in today, the introduction of several powerful factors such as REST have really solidified the concepts of how Rails works and can be so powerful.

Edge Rails will be released as a 2.0 preview release soon. In Portland, RailsConf US, we saw several of these (Edge Rails) features demonstrated and I’ve been using them in several projects since. It will be great when it’s officially released.

Jason Hoffman gave an interesting talk about scaling, and in particular how Rails is a small part to the larger picture when examining how to get the most out of your hardware and architecture. There was one slide in particular where Jason listed perhaps 35 different things that need to be examined when scaling up your application, of which only perhaps 3 of them were Rails application based.

I also enjoyed Craig McClanahan and Nick Siegers talk, particular the discussion of using ActiveResource to allow your applications to communicate between each other. I’ve used ActiveResource in the past, so it was good to hear about others using it in similar situations as well.

Evan Phoenix also gave a fantastic talk about Rubinius, which is the alternate Ruby VM implementation that I’ve also been contributing to over the past few months. In particular, some of the statistics Evan showed were great, and be sure to take a look at them when he publishes his slides :)

Geoffrey Grosenbach gave a top talk about the various ways of DRYing up your CSS. I particularly liked the discussion of using Sass to do so. I’ve looked at Haml before and love it, it’s great to see those ideas now available in CSS.

Many of the presentations are already available online as well.

So that’s it for this year, next year RailsConf US will be in Portland again, and I’m also looking forward to seeing which city the next RailsConf EU will be located in. The 26 hour flight from Australia was certainly worth it, although I’m glad I came over for more than a week to take advantage of the trip.

Last night I met up with the local Frankfurt Rails User Group developers. It was an awesome night, with lots of fun and laughter had by all. I always find it interesting to meet new people and hear what they’ve been doing with Ruby and Rails.

I also gave a few presentations about Rubinius and RSpec, the slides are available for anyone interested:

Rubinius
RSpec
RSpec handout

Many thanks to Mariano for hosting the event, and also a special thanks to Claudia for catering! :)

Today I’m traveling up to Berlin for the next part of my trip, RailsConf Europe. looking forward to meeting up with so many fellow Rails developers in the coming week!