Desktop Screen Capture via Ruby Cocoa!

posted by crafterm, 12 January 2008

During the week a good friend of mine laid down the challenge to work out how to programatically create a screenshot of your Mac OSX desktop. The following article steps through the process of performing this, adding some charm to the operation to create a full desktop snapshot tool using Ruby Cocoa.

Before stepping into the code, I’ll first present a small extension module that all code snippets rely upon. It’s essentially a small extension to the OSX::CIImage class to simplify loading and saving of image files, and conversions between Core Image and Quartz Image objects (which we’ll need later).

extensions.rb

module OSX
  class CIImage      
    def save(target, format = OSX::NSJPEGFileType, properties = nil)
      bitmapRep = OSX::NSBitmapImageRep.alloc.initWithCIImage(self)
      blob = bitmapRep.representationUsingType_properties(format, properties)
      blob.writeToFile_atomically(target, false)
    end

    def cgimage
      OSX::NSBitmapImageRep.alloc.initWithCIImage(self).CGImage()
    end

    def self.from(filepath)
      raise Errno::ENOENT, "No such file or directory - #{filepath}" unless File.exists?(filepath)
      OSX::CIImage.imageWithContentsOfURL(OSX::NSURL.fileURLWithPath(filepath))
    end
  end
end

Capturing the Desktop

With our extensions code in place, we can use the new CGWindow API, recently added to Mac OSX 10.5, to snapshot our desktop:

require 'osx/cocoa'
require 'extensions'

class Screen
  def self.capture
    screenshot = OSX::CGWindowListCreateImage(OSX::CGRectInfinite, OSX::KCGWindowListOptionOnScreenOnly, OSX::KCGNullWindowID, OSX::KCGWindowImageDefault)
    OSX::CIImage.imageWithCGImage(screenshot)
  end
end

Screen.capture.save('desktop.jpg')

This works really well and is only a few lines of code. It’s quite fast, in fact the Apple documentation mentions “For capturing pixels, the CGWindow API should demonstrate performance that is equal or better than the techniques used by the OpenGLScreenSnapshot and OpenGLScreenCapture samples”. The only thing is that the user has no feedback that an actual screenshot was taken, other than the creation of the target image file.

So, lets build upon this and add some feedback.

Capturing the Desktop with a Fade operation

What we’ll do to make it obvious that a screenshot is being taken, is fade the desktop out to a black colour, take the screenshot of the original desktop content, and fade the desktop back to it’s original state. The effect is similar to what OSX does while changing screen resolutions when you attach an external display or a projector to your Mac.

Implementation wise, we’ll add a fade operation to our Screen class, that will accept a block of code to perform in between fading the display out and back in again. The relevant Cocoa operations are documented in the Quartz Display Services guide. Essentially we need to invoke CGAcquireDisplayFadeReservation() to obtain a fade reservation token, after which we can invoke CGDisplayFade() to fade the display to a solid colour and back. Once we’re done fading, we can release the fade reservation token with CGReleaseDisplayFadeReservation() (or allow it to time out).

require 'osx/cocoa'
require 'extensions'

class Screen

  def self.capture
    fade do
      screenshot = OSX::CGWindowListCreateImage(OSX::CGRectInfinite, OSX::KCGWindowListOptionOnScreenOnly, OSX::KCGNullWindowID, OSX::KCGWindowImageDefault)
      OSX::CIImage.imageWithCGImage(screenshot)
    end
  end

  private

    def self.fade
      err, token = OSX::CGAcquireDisplayFadeReservation(OSX::KCGMaxDisplayReservationInterval)

      if err == OSX::KCGErrorSuccess
        begin
          OSX::CGDisplayFade(token, 0.3, OSX::KCGDisplayBlendNormal, OSX::KCGDisplayBlendSolidColor, 0, 0, 0, true)
          return yield if block_given?
        ensure
          OSX::CGDisplayFade(token, 0.3, OSX::KCGDisplayBlendSolidColor, OSX::KCGDisplayBlendNormal, 0, 0, 0, false)
          OSX::CGReleaseDisplayFadeReservation(token)
        end
      end
    end
end

Screen.capture.save('desktop.jpg')

Capturing the Desktop with a Fade and Snap Photo Picture

The fade looks really nice, but since other applications can also perform the same effect, lets embed a small graphic in between the fade of a camera to really show that we’re taking a picture of the desktop.

To do this, we need to capture the desktop in between the fade operation, and draw directly onto the display, without creating a window or any other graphical decorations. Capturing and drawing directly onto a display are discussed in the Quartz documentation online also.

Essentially, we need to use CGDisplayCapture(display) to capture a specified display. While it’s captured we have exclusive access to the display, and no other application will interfere with it. Then, we can use CGDisplayGetDrawingContext(display) to obtain a drawing context, and CGContextDrawImage() to draw an image directly to the display. Once we’re done showing our picture, we can then release our capture of the display using CGDisplayRelease(display).

require 'osx/cocoa'
require 'extensions'

class Screen

  def self.capture
    fade do
      screenshot = OSX::CGWindowListCreateImage(OSX::CGRectInfinite, OSX::KCGWindowListOptionOnScreenOnly, OSX::KCGNullWindowID, OSX::KCGWindowImageDefault)
      OSX::CIImage.imageWithCGImage(screenshot)
    end
  end

  private

    def self.fade
      err, token = OSX::CGAcquireDisplayFadeReservation(OSX::KCGMaxDisplayReservationInterval)

      if err == OSX::KCGErrorSuccess
        begin
          OSX::CGDisplayFade(token, 0.3, OSX::KCGDisplayBlendNormal, OSX::KCGDisplayBlendSolidColor, 0, 0, 0, true)

          snap(token)

          return yield if block_given?
        ensure
          OSX::CGDisplayFade(token, 0.3, OSX::KCGDisplayBlendSolidColor, OSX::KCGDisplayBlendNormal, 0, 0, 0, false)
          OSX::CGReleaseDisplayFadeReservation(token)
        end
      end
    end

    def self.snap(token)
      display = OSX::CGMainDisplayID()

      if OSX::CGDisplayCapture(display) == OSX::KCGErrorSuccess
        begin
          ctx = OSX::CGDisplayGetDrawingContext(display)
          if ctx
            pic = OSX::CIImage.from('nikon.jpg')

            OSX::CGDisplayFade(token, 0.0, OSX::KCGDisplayBlendSolidColor, OSX::KCGDisplayBlendNormal, 0, 0, 0, true)

            display_width, display_height = OSX::CGDisplayPixelsWide(display), OSX::CGDisplayPixelsHigh(display)
            pic_width, pic_height = pic.extent.size.width, pic.extent.size.height
            position_x, position_y = (display_width - pic_width) / 2.0, (display_height - pic_height) / 2.0

            OSX::CGContextDrawImage(ctx, OSX::NSRectFromString("#{position_x} #{position_y} #{pic_width} #{pic_height}"), pic.cgimage)

            sleep(0.7)

            OSX::CGDisplayFade(token, 0.0, OSX::KCGDisplayBlendNormal, OSX::KCGDisplayBlendSolidColor, 0, 0, 0, true)
          end
        ensure
          OSX::CGDisplayRelease(display)
        end
      end
    end
end

Screen.capture.save('desktop.jpg')

Summary

First we created a simple capture class that used the new CGWindow API in Mac OSX 10.5, then we built upon that adding a fade effect around the actual screen capture. Next we drew an image directly on the display in between the fade, to make it even more obvious that we were taking a screenshot.

There we have it, a really useful tool for capturing screenshots of your desktop programatically.

Special Thanks!

  1. Thanks Lachlan for the challenge mate! :)
  2. Thanks Pete for your help with the Cocoa desktop fade semantics.
  3. Thanks DSevilla for the camera image used above.

Cool Effects with Core Image!

posted by crafterm, 31 December 2007

In a previous article I described how to use Core Image as the backend image processor for Attachment Fu in your Rails applications. In that particular article we looked at supporting image scaling and thumbnails to be compatible with the other Attachment Fu backends such as RMagick and ImageScience.

With Core Image available however, we have an entire range of post processing filters available to use at our fingertips. In this article we’ll step through a few of these additional filter options that you can use to post process your images with.

Here’s a few examples of what we can do with Core Image post file upload. All of the following examples use the following input image taken in Berlin while at RailsConf EU (also used in the performance measurements article):

Greyscale or Sepia

A scaled version of the source image is cool, but how about an automatic greyscale or sepia version of the image:

Greyscale

Sepia

Code Fragment

module RedArtisan
  module CoreImage
    module Filters
      module Color

        def greyscale(color = nil, intensity = 1.00)
          create_core_image_context(@original.extent.size.width, @original.extent.size.height)

          color = OSX::CIColor.colorWithString("1.0 1.0 1.0 1.0") unless color

          @original.color_monochrome :inputColor => color, :inputIntensity => intensity do |greyscale|
            @target = greyscale
          end
        end

        def sepia(intensity = 1.00)
          create_core_image_context(@original.extent.size.width, @original.extent.size.height)

          @original.sepia_tone :inputIntensity => intensity do |sepia|
            @target = sepia
          end
        end

      end
    end
  end
end

Exposure and Noise Control

Another option for us is to automatically adjust exposure and noise parameters upon upload to brighten images up, or remove unwanted noise from lower quality images:

1 F-Stop

2 F-Stops

Noise Removal

Code Fragment

module RedArtisan
  module CoreImage
    module Filters
      module Quality

        def reduce_noise(level = 0.02, sharpness = 0.4)
          create_core_image_context(@original.extent.size.width, @original.extent.size.height)

          @original.noise_reduction :inputNoiseLevel => level, :inputSharpness => sharpness do |noise_reduced|
            @target = noise_reduced
          end
        end

        def adjust_exposure(input_ev = 0.5)
          create_core_image_context(@original.extent.size.width, @original.extent.size.height)

          @original.exposure_adjust :inputEV => input_ev do |adjusted|
            @target = adjusted
          end          
        end

      end
    end
  end
end

Watermarking

Sometimes we’d like to automatically add a watermark to our images, either with a single watermark image, or as a tiled watermark image:

Single Watermark

Tiled Watermark

Code Fragment

module RedArtisan
  module CoreImage
    module Filters
      module Watermark

        def watermark(watermark_image, tile = false, strength = 0.1)
          create_core_image_context(@original.extent.size.width, @original.extent.size.height)

          if watermark_image.respond_to? :to_str
            watermark_image = OSX::CIImage.from(watermark_image.to_str)
          end

          if tile
            tile_transform = OSX::NSAffineTransform.transform
            tile_transform.scaleXBy_yBy 1.0, 1.0

            watermark_image.affine_tile :inputTransform => tile_transform do |tiled|
              tiled.crop :inputRectangle => vector(0, 0, @original.extent.size.width, @original.extent.size.height) do |tiled_watermark|
                watermark_image = tiled_watermark
              end
            end
          end

          @original.dissolve_transition :inputTargetImage => watermark_image, :inputTime => strength do |watermarked|
            @target = watermarked
          end
        end

      end
    end
  end
end

Funky Effects

We can also use cool and funky effects used in applications like Photobooth, here’s an example using the edge colouring algorithm:

Edges

Core Fragment

module RedArtisan
  module CoreImage
    module Filters
      module Effects

        def edges(intensity = 1.00)
          create_core_image_context(@original.extent.size.width, @original.extent.size.height)

          @original.edges :inputIntensity => intensity do |edged|
            @target = edged
          end
        end

      end
    end
  end
end

The sign artwork works particularly well with this algorithm.

Core Image Processor

All of the code above is also available as a usable image processor via git.

Some examples of using the processor:

require 'red_artisan/core_image/processor'

# generate some test output images for various effects

processor = RedArtisan::CoreImage::Processor.new('berlin.jpg')

grey = processor.greyscale
grey.save 'results/berlin-grey.jpg'

sepia = processor.sepia
sepia.save 'results/berlin-sepia.jpg'

watermarked = processor.watermark('watermark_image.png')
watermarked.save 'results/berlin-watermarked.jpg'

watermarked = processor.watermark('watermark_image.png', true)
watermarked.save 'results/berlin-watermarked-tiled.jpg'

noise_reduced = processor.reduce_noise
noise_reduced.save 'results/berlin-noise-reduced.jpg'

exposure_adjusted = processor.adjust_exposure
exposure_adjusted.save 'results/berlin-exposure-adjusted-half-stop.jpg'

exposure_adjusted = processor.adjust_exposure(2.0)
exposure_adjusted.save 'results/berlin-exposure-adjusted-two-stops.jpg'

edge = processor.edges
edge.save 'results/berlin-edge.jpg'

Summary

The above shows us only a fraction of what can be done with the 100+ filters Core Image provides by default. There’s many other filters that let you create all sorts of effects with single and multiple images combined. Enjoy!

Core Image Performance

posted by crafterm, 16 December 2007

Following my previous post, a few people have asked me how well Core Image performs in comparison to RMagick and ImageScience when used as part of Attachment Fu. To answer these questions, I’ve spent a bit of time collecting some performance results.

Please note that as with all performance testing, results will vary according to many parameters such as hardware, processing power, and even the input image content itself, as you’ll see below.

Naturally Core Image should be faster since it will run hardware accelerated on my MacBook Pro, so please take these results more as an indication of library characteristics rather than final timings. The tests I performed also mirrored how each library was being invoked by Attachment Fu, and beyond performance there’s also several other tangible benefits to Core Image such as deployment and access via RubyCocoa to keep in mind as well.

The first test performs 5 successive resizes of a 1mb image under each library. The input image is a complex photo (smaller version above) taken in Berlin of parts of the Berlin Wall, including a high colour range and lots of shapes and sizes:

Both RMagick and ImageScience yield consistent results as expected. Core Image’s resize values are interesting as successive resizes are twice as fast than the first, which I speculate would include an initial Core Image to OpenGL compilation phase of the resize operation being performed.

Similar characteristics can be observed with a slightly larger image size of around 2.5mb:

In both cases, Core Image performs really well, particularly on consecutive runs.

The actual image content being resized also affects results as well – the following test uses a blank image of identical dimensions to the 2.5mb Berlin image:

Finally, the following graph shows the results of resizing 6 versions of the Berlin image with each library, at increasing size & dimensions. The results plotted for each measurement are calculated over the average of 5 resizes of each image.

At small image sizes, we can see the difference in resize time to be essentially negligible, however as image size increase up to 500kb, the hardware acceleration of Core Image starts to shine, particularly around the 2.5mb – 5mb range and beyond.

There does seem to be a limit to performance gains, on Friday Ben and I also made a few experiments with really large image sizes (TIFF files around 90mb), and noticed that Core Image can be slower than Image Science under these conditions:

At this stage I can only speculate as to why this occurs, and it deserves further inspection.

Another interesting aspect to look at is the actual on disk size of the resized images. The following is a directory listing of the results after the 2.5mb image (2200x1467 resize to 640x480):

-rw-r--r--  1 crafterm  staff  153492 16 Dec 22:30 core_image-complex_resized.jpg
-rw-r--r--  1 crafterm  staff  288455 16 Dec 22:30 image_science-complex_resized.jpg
-rw-r--r--  1 crafterm  staff  356827 16 Dec 22:30 rmagick-complex_resized.jpg

Core Image’s default compression range is quite impressive, and while I’m sure the other libraries compression factors can be fine tuned I suspect it would also affect their relative performance measurements.

All measurements shown in the graphs are in seconds, and the test machine was my MacBook Pro, 15" Intel Core 2 Duo, 2.16 Ghz with an ATI Radeon X1600 video card with 128mb vram. I’d certainly be keen to see how other types of hardware perform, so please feel free to run the test code on your system and send me the resultant graphs, and I’ll add them to the project.

All test code and input images (except for the 90mb example) are available in a git repository online.