OnRails.org

One of the great “new” features of Rails (as of 2.3) is accepts_nested_attributes_for, allowing you to build cross-model CRUD forms without “cruddying” your controller. There are some great examples out there about how to do this, but I’d like to walk thorough a particular use case — managing the “join” records in a has_many :through relationship.

Consider the following database schema:

class Villain < ActiveRecord::Base
  has_many :gifts
  has_many :super_powers, :through => :gifts
end

class Gift < ActiveRecord::Base
  belongs_to :villain
  belongs_to :super_power

  validates_uniqueness_of :super_power, :scope => :villain_id
end

class SuperPower < ActiveRecord::Base
  has_many :gifts
  has_many :villains, :through => :gifts
end

In our dataset, there are a relatively small number of super powers which we wish to present as a list of checkboxes on the villain management form. Checking/unchecking the boxes will manage the gift records for that villain, effectively managing the list of super powers available to the baddy.

To get started, we need to add accepts_nested_attributes_for :gifts to the Villain class — piece of cake. To complete the implementation, we need to change the params hash that our form generates. Let’s review the cases that we need to support and the associated params hash format needed to implement the correct functionality.

The first case is a super power record that is not currently associated with the villain. Here, the UI should display an unchecked checkbox. If we check it and submit the form, a gift record should be created linking the villain with the super power, making this bad guy that much badder. Here is an example of the params hash we should be sending to accomplish this:

  {
    'villain' => {
      'name' => 'Lex Luthor',
      ...
      'gifts_attributes' => {
        1 => { 'super_power_id' => 5 },
        2 => { 'super_power_id' => 7 },
        ...
      }
    }
  }

The alternate case is a super power this villain already possesses. In this instance, the UI should display a checked checkbox, and if we uncheck it, the existing gift record should be deleted, diminishing the villain’s capacity for evil. And our params hash needs to look like:

  {
    'villain' => {
      'name' => 'Two-Face',
      ...
      'gifts_attributes' => {
        1 => { 'id' => 101, '_delete' => true },
        ...
      }
    }
  }

Note that the keys for the gifts_attributes hash are arbitrary; we can use any scheme to generate unique keys for the hash.

So how can we craft a form that sends the params hash that Rails wants to see? Here’s my implementation:

  <%- SuperPower.all.each_with_index do |super_power, index| -%>
    <label>
      <%- if gift = @villain.gifts.find_by_super_power_id(super_power.id) -%>
        <%= hidden_field_tag "villain[gifts_attributes][#{ index }][id]", gift.id %>
        <%= check_box_tag "villain[gifts_attributes][#{ index }][_delete]", false, true %>
        <%= hidden_field_tag "villain[gifts_attributes][#{ index }][_delete]", true %>
      <%- else -%>
        <%= check_box_tag "villain[gifts_attributes][#{ index }][super_power_id]", super_power.id %>
      <%- end -%>

      <%= super_power.name %>
    </label><br />
  <%- end -%>

If the gift is detected, the villain has the super power, and we handle our second case from above, using the checkbox / hidden field hack Rails employs in the check_box helper method to make sure a value is sent whether or not the checkbox is checked. The else block handles the other case, setting up our params hash to create the gift if the checkbox is checked.

This works, but we probably don’t want to copy and paste that code everywhere we use this pattern. How can we reuse this in a DRY fashion? Here’s a helper method that encapsulates the logic:

  def has_join_relationship(model, join_collection_name, related_item, collection_index, options={})
    returning "" do |output|
      relationship_name = options[:relationship_name] || related_item.class.table_name.singularize + "_id"
      tag_prefix = "#{ model.class.class_name.underscore }[#{ join_collection_name }_attributes][#{ collection_index }]"

      if join_item = model.send(join_collection_name).find(:first, :conditions => { relationship_name => related_item.id })
        output << hidden_field_tag("#{ tag_prefix }[id]", related_item.id)
        output << check_box_tag("#{ tag_prefix }[_delete]", false, true)
        output << hidden_field_tag("#{ tag_prefix }[_delete]", true)
      else
        output << check_box_tag("#{ tag_prefix }[#{ relationship_name }]", related_item.id, false)
      end
    end
  end

Drop that in a helper, and then your form code becomes:

  <%- SuperPower.all.each_with_index do |super_power, index| -%>
    <label>
      <%= has_join_relationship(@villain, :gifts, super_power, index) %>
      <%= super_power.name %>
    </label><br />
  <%- end -%>

Much nicer … although I’m not sold on the name has_join_relationship. Any suggestions?

I’ve been loving Rails BDD with Cucumber for the past year or so — it helps me focus on the next required step to build a feature in my application, and better focus equals better development velocity. However, one thing I found tedious in starting with Cucumber was defining route matchers in paths.rb.

The stock path_to method is implemented as a case statement, allowing you to add a case for each path you want to recognize. This works, but leaves you feeling a bit un-DRY since you’re basically duplicating information in your routes.rb file.

Here’s a quick hack to paths.rb that lets you leverage your existing routes:

change

    else
      raise "Can't find mapping from \"#{page_name}\" to a path.\n" +
        "Now, go and add a mapping in #{__FILE__}"
    end

to

    else
      begin
        page_name =~ /the (.*) page/
        path_components = $1.split(/\s+/)
        self.send(path_components.push('path').join('_').to_sym)
      rescue Object => e
        raise "Can't find mapping from \"#{page_name}\" to a path.\n" +
          "Now, go and add a mapping in #{__FILE__}"
      end
    end

In your Cucumber steps, you can now use any named route that does not require a parameter. For example, users_path would become “the users page”, and new_product_path would become “the new product page”. As you add new resources, at least the index and new options should work out of the box — no further edits to paths.rb required!

UPDATE:

w00t!

This is now baked into cucumber-rails!

After the release of 10.5, I published an article about building RMagick from source on Leopard. I won’t rehash the why, you can read the original article for that. My clean install necessitated updating the RMagick script, so here’s what worked for me to install from source on Snow Leopard! For the impatient, here’s the download link: rmagick-build.sh

First, we start with installing wget, as it seems to be a bit more clever than curl about dealing with mirrors, etc. Then, we compile and install each prerequisite package. Finally, we install the gem.

All the links in the script worked for me, but, depending on your location, network, conditions, etc, your mileage may vary. Enjoy!

#!/bin/sh

# install wget, which is cleverer than curl
curl -O http://ftp.gnu.org/gnu/wget/wget-1.11.tar.gz
tar zxvf wget-1.11.tar.gz 
cd wget-1.11
./configure --prefix=/usr/local
make
sudo make install
cd /usr/local/src

# prerequisite packages
wget http://nongnu.askapache.com/freetype/freetype-2.3.9.tar.gz
tar zxvf freetype-2.3.9.tar.gz
cd freetype-2.3.9
./configure --prefix=/usr/local
make
sudo make install
cd /usr/local/src

wget http://superb-west.dl.sourceforge.net/sourceforge/libpng/libpng-1.2.39.tar.gz
tar zxvf libpng-1.2.39.tar.gz
cd libpng-1.2.39
./configure --prefix=/usr/local
make
sudo make install
cd /usr/local/src

wget ftp://ftp.uu.net/graphics/jpeg/jpegsrc.v6b.tar.gz
tar xzvf jpegsrc.v6b.tar.gz
cd jpeg-6b
ln -s `which glibtool` ./libtool
export MACOSX_DEPLOYMENT_TARGET=10.6
./configure --enable-shared --prefix=/usr/local
make
sudo make install
cd /usr/local/src

wget ftp://ftp.remotesensing.org/libtiff/tiff-3.9.1.tar.gz
tar xzvf tiff-3.9.1.tar.gz
cd tiff-3.9.1
./configure --prefix=/usr/local
make
sudo make install
cd /usr/local/src

wget http://superb-west.dl.sourceforge.net/sourceforge/wvware/libwmf-0.2.8.4.tar.gz
tar xzvf libwmf-0.2.8.4.tar.gz
cd libwmf-0.2.8.4
make clean
./configure
make
sudo make install
cd /usr/local/src

wget http://www.littlecms.com/lcms-1.17.tar.gz
tar xzvf lcms-1.17.tar.gz
cd lcms-1.17
make clean
./configure
make
sudo make install
cd /usr/local/src

wget ftp://mirror.cs.wisc.edu/pub/mirrors/ghost/GPL/gs870/ghostscript-8.70.tar.gz
tar zxvf ghostscript-8.70.tar.gz
cd ghostscript-8.70
./configure  --prefix=/usr/local
make
sudo make install
cd /usr/local/src

wget ftp://mirror.cs.wisc.edu/pub/mirrors/ghost/GPL/gs860/ghostscript-fonts-std-8.11.tar.gz
tar zxvf ghostscript-fonts-std-8.11.tar.gz
sudo mv fonts /usr/local/share/ghostscript

# Image Magick
wget ftp://ftp.fifi.org/pub/ImageMagick/ImageMagick.tar.gz
tar xzvf ImageMagick.tar.gz
cd `ls | grep ImageMagick-`
export CPPFLAGS=-I/usr/local/include
export LDFLAGS=-L/usr/local/lib
./configure --prefix=/usr/local --disable-static --with-modules --without-perl --without-magick-plus-plus --with-quantum-depth=8 --with-gs-font-dir=/usr/local/share/ghostscript/fonts --disable-openmp
make
sudo make install
cd /usr/local/src

# RMagick
sudo gem install rmagick

UPDATE There is a bug with libgomp that breaks the convert utility (See comments below). the --disable-openmp configure option has been added to the script to fix this.

UPDATE 2 A new patchlevel of ImageMagick has been released that supersedes the original one referenced in this script, and the original has been removed from the server. Thanks to Sebastian for this update that will grab the latest release.

I’ve just finished my first day of RailsConf 09. Things have gotten started well — I’ve met several interesting people, found some interesting people to follow on twitter. I’ll try to post as much as I can in the way of reviews for those of you attending by blog.

Starting with the DHH Keynote. David’s talk was entitled “Rails 3 and the Real Secret to High Productivity”.

He started off by reviewing some of the so-called “mortal wounds” Rails has experienced:

• It was deemed as not enterprise-ready, just a lot of hucksters trying to generate interest in Ruby to sell their own books, etc.
• it encountered an “attack of the clones” in which several other languages/frameworks tried to duplicate (what they felt was) the one “key” feature that was the root cause of Rails’ explosion in success and popularity
• some early adopters switched back (Derek Sivers, CD Baby)
• some early adopters encountered serious problems (Twitter), which may or may not have been Rails-specific issues, but were perceived as such
• some people complained about “feature bloat” (Array#fourteenth, etc.)

The takeaway from this review: There is no one best tool for every job — it’s okay if not everyone chooses Rails. Rails can’t be distilled down to a single feature, it must be viewed as a sum of its components. Emotions or reactions can be quite strong, but are usually not all that important/long-lasting.

Next, he outlined some central ideas of the Rails 3 philosophy:

• Lock up all the unicorns: This is not a complete rewrite. Many people might look at Rails 3 and be disappointed, expecting more radical changes.
• No “Sacred Cows” allowed: Everything is on the table for possible change in Rails 3. That being said, it’s important to note that any changes causing backward compatibility issues will need a very good justification to be accepted/implemented.
• “Have it your way”: The default application generated under Rails 3 will rock, but you can ask for things to be done a different way and override the defaults. This seems like a natural extension of the templates now available in Rails 2.

He also covered some of the progress made to date in Rails 3. It sounds like things are not going as fast as was hoped, but there is code available now in GitHub — just no officially tagged release.

Plans for Rails 3 include:

• a new router, with the ability to route requests by subdomains, user-agent strings, etc. also able to route to other rack machinery
• XSS protection, in which any text output will be html-escaped by default, requiring you to explicitly request raw output if/when you need it.
• JS will be unobtrusive and framework-agnostic. This is one of the more exciting plans for me. tags will be generated with the HTML 5 validating data-* attributes, which can be used as unobtrusive javascript hooks:

HTML

  <a href="/person/1" data-remote="true" data-method="delete">Delete This Person</a>

JavaScript

  $(document.body).observe("click", function(event) { 
    var element = event.findElement("a['data-remote']"); 
    if (element) { 
      var method = element.readAttribute("data-method") || "get"; 
      new Ajax.Request(element.readAttribute("href"), { method: method }); 
      event.stop(); 
    } 
  });

• more agnosticism in ORM choice and generators
• more refactoring of various “crufty” areas of the framework, speed related improvements

Then, near the end, he got to the “secret to high productivity” part. In a nutshell — don’t view the requirements as carved in stone. If you are able to make (slight) changes to the requirements as originally communicated by the business, it may serve their purposes just as well as the original requirements but be drastically easier to implement. As cool as Rails is, no framework is going to match the performance boost this can bring.

Don’t Mock Yourself Out – David Chelimsky

I went to David Chelimsky’s talk on Mocks, but it was a bit too introductory-level. He basically explained good reasons to use mocks — from the title, I was expecting ways to determine if your mocks are becoming too heavy-handed, preventing you from actually testing any of your implementation!

Here are the highlights:

David Started with a review of the difference between mocks and stubs — basically, mock objects care about which methods are or aren’t called during the test, stubs are “dumb” containers to give canned responses to methods. See Martin Fowler’s explanation for more information.

Here are examples of when stubs might be useful:

• isolation from non-determinism: when dealing with any randomly generated values, your tests face brittleness from non-determinism. Similar for timestamps, date ranges, etc.
• isolation from external dependencies: Anything that connects to an external resource (e.g. network/db) can benefit from (smart) mocking. You should still test the “real” operation, but for adding permutations/edge case tests, mocks can speed up your test suite by multiple orders of magnitude.
• polymorphic collaborators : (strategies / mixins/plugins) calling a method on the “main” object ends up delegating to a method somewhere else

Here are examples of when mocks might be useful:

• testing side effects (e.g. something gets logged)
• testing caching / memoization — ensure an expensive method gets called exactly once.
• interface discovery : “outside-in” development : build a mock of something before it exists, use to discover the “ideal” interface, how you will be using this in the real world.

At this point, we started getting to Rails-specific content. David pointed out that if you are doing traditional Unit/Functional/Integration tests, it’s not DRY-some parts of your app (e.g. model validations) are being tested multiple times across your test suite. He recommends to split tests instead into business-facing / developer-facing specs, even using different frameworks where appropriate-for example, RSpec stories are more suited for business-facing specs, since they are close to the natural language of the domain experts. For lower-level, developer-facing tests (e.g. database-level), it probably doesn’t make sense to invest time writing stories—the developer is the domain expert.

A couple of helpful tips — if you care about which template gets rendered, but not necessarily the content (which could change!), you can stub the call to render, and set an expectation that it receives the correct partial name when called. When doing Rails functional testing, you need to set up valid/invalid data to make sure both sides of create/update actions are executed. Instead, you could stub the new/save/etc methods on the object to force the branch you want to test.

Upcoming “stubble” project helps with this.

UI Fundamentals For Programmers – Ryan Singer

Next, I saw Ryan Singer (37 Signals) talk about UI design concepts for developers. This was a high-level talk to help developers with the approach to take to UI/design, not a how-to, recipe style talk.

First of all, Ryan advocates taking a BDD, outside-in approach — including the UI. The UI is the closest thing to the user, so start there and work your way in, implementing backend functionality as you need.

Next, overcome your natural tendency (as a developer) to be terse — you should add ample text to guide your end user through the flow of your site/screens. Don’t rely on implicit meaning in forms/layouts as a guide, be explicit and explain what is expected of the user.

You should form a model (conceptual model, not ActiveRecord model) of the domain problem that makes sense to the customer and is implementable (feasable). Recommended reading: Domain-Driven Design by Eric Evans. (also Tufte for information display in general)

Designers tend to think in “Screens” as the atomic unit of work. This plays nicely into REST/Resource conventions, mapping e.g. displaying a particular object to the “show” action of your resource controller. For multiple related objects, break the screen into multiple areas that each represent a resource.

Be careful with contrast-make sure that you emphasize the most important elements on the page. Watch colors, sizes, etc. Don’t present a visual “buffet line” — aim more for a “gourmet meal”. Less is more.

Every action a user can take on your site has three distinct parts, a beginning, a middle, and an end. Think about where the user is when they need/want to take the action (where they are linked from). For the middle, collect the data for the action, then consider what the user will want/need to do next.

Some rules of thumb: helpers should not generate blocks of HTML (although generating an HTML element is probably ok — think image_tag); Organize your CSS/JS around REST conventions for your application; Treat ERb partials that generate HTML just as you would an image tag; Use helpers to reveal intention.

Smacking Git Around – Scott Chacon

Scott’s talk (aka the Git Firehose) was just as awesome (and fast-paced) as last year. Much too quick to take good notes, but I was able to capture a couple of useful tips & tricks:

First, on the concept of reachability: every commit that is an ancestor of your current commit. Git uses this in the dot-dot syntax for git-log and subtracts the first from the second. So git log origin/master..master gives you everything in your local master not already pushed to the remote repo. Or, after a fetch, reversing it (git log master..origin/master) gives you all commits you just pulled down that haven’t been merged into local master. Handy! A couple of alternate syntaxes: git log branch_a --not branch_b, or git log branch_a ^branch_b. This is also extensible to more than two branches: git log origin/master master ^experiment would give you all commits on origin/master and/or master that are not in experiment.

Second, he illustrated a common problem with diffing between branches in Git. If you imagine a file existing in two branches, in which lines have been added on both branches, and you ask git for a diff, most often you want to know what has been added to the other branch that you don’t have. By default, though, the question git is answering is “how do I make this file look like the other file?”, which includes deleting the lines you have added on the current branch — probably not what you want. In this instance, you can use the triple-dot syntax: git diff master...topic (asking git “what would happen if I merged the two branches?”)

He talked about subtree merging as an alternative to submodules. Basically, the process is:

1. add a remote to the project you want to add as a subtree and fetch:
   

   git remote add rails_remote git@github.com:myfork/rails
   git fetch rails_remote

2. check out a local branch of the other project:
   

   git checkout -b rails_branch rails_remote/master

3. return to your project and use git-read-tree to add the subtree, then add and commit:
   

   git checkout master
   git read-tree --prefix=vendor/rails -u rails_branch
   git add vendor/rails
   git commit

This keeps your ability to make changes to the remote project and submit them back upstream:

1. make change(s) to in remote project and commit
2. go to remote project’s local branch
   

   git checkout rails_branch

3. merge your changes in:
   

   git merge -s subtree --no-commit --squash master

This obviates the need for braid/piston! See http://tinyurl.com/braidgit for more info.

Also mentioned was patch staging — simple, but incredibly useful! Say you’ve made multiple changes to the same file, and you want to commit some, but not all. With patch staging (git add -p), you can select the hunks you want to stage for commit — awesome!

More useful info, including how to give Git the ability to diff binary files (kinda, but it can work). Slides are posted at http://tinyurl.com/gitrailsconf09

Quality Code with Cucumber – Alsak Hellesøy

For the last session, I saw Alsak Hellesøy’s most excellent talk on Cucumber. I was totally drained at this point, and took no notes. He gave an example of how to get started with Cucumber, working through a sample scenario, doing outside-in development. So far, I haven’t used RSpec / BDD in anything serious, but I’m looking forward to trying this out — looks pretty interesting.

Ruby Heros / Evening Keynote

The day ended with the Ruby Hero awards, and then the DHH / Tim Ferriss (4 hour workweek) interview/keynote. I think there was probably good content to be had in the keynote about refactoring your lifestyle. Unfortunately, the sound was bad — very muffled/echoy from where I was sitting. Also, the pace was so much different from the firehose of information presented by the technical talks that it was like a panic stop in a formula one racer.

Overall, a good but exhausting day here in Vegas. If you’re here, look me up and say hi, or give me a follow (rubysolo on Twitter).

One interesting problem we faced at VideoPros was file upload. In our application domain, the files tend to be orders of magnitude larger than, say, an avatar attachment for your blog post. HTTP multipart upload works for these large files, but users need an indication that the upload is progressing so they do not refresh the browser or navigate away from the page, killing the upload in mid-stream. To help out with this first facet of the problem, we implemented an apache handler that tracks file upload progress in front of the rails application. You can check it out here.

UPDATE: Here’s a good screencast that goes over getting this set up on OSX — you’ll need some extra arguments to correctly compile and install the apache module.

This worked well, but we still got requests for an alternative: the users wanted to use FTP to upload their files. The workflow would be: users would connect and upload a batch of video files into their FTP “inbox”. After the upload(s) were complete, the users would create video objects in the database and attach the uploaded files. As an interesting twist, the users requested the ability to re-use their existing application usernames and passwords to access FTP. Our goal was to meet this requirement while providing secure, low-maintenance FTP service.

A bit of research turned up PureFTPd, an open-source implementation touted as secure, with an interesting-sounding feature called “virtual users”. My first hope was that we would be able to use the MySQL Auth feature, which reads usernames and passwords from a MySQL table for authentication. If your app is like ours, you have a users table with this info, so it doesn’t seem to be a stretch that we could get this to work. However, further investigation uncovered a problem — Pure’s MySQL Auth only supports plaintext, MD5, MySQL password, and UNIX crypt as formats for the password column. We’re using ReSTful Auth, which means that our password are SHA-1’ed. This, combined with wanting to define users’ home folders programatically rather than storing them in the database, eliminated MySQL auth as a possibility for our implementation.

However, all was not lost. According to the documentation, another option for authentication is Ext Auth, which allows you to run an arbitrary process to perform the authentication. The way this works is pure-ftpd communicates over a socket to another daemon, pure-authd, which launches your authentication process, then sends the plaintext result back to pure-ftpd, which accepts or rejects the login. Sounds like a good candidate for a Ruby script!

To get started, first install Pure-FTPd. In my environment, this was as simple as:

build-pure-ftpd
./configure --with-extauth
make
sudo make install

Next, we need to build an authentication script. The script receives the username and password (along with source IP / port …) as environment variables. It would be trivial to load up our Rails environment and use ActiveRecord to find_by_username, but it would also add several seconds of overhead to load the framework. I opted to go with the less-DRY but faster method of using the mysql gem directly. The first order of business is to load the user record that matches the specified username.

select-user
begin
  dbh = Mysql.real_connect(config['host'], config['username'], config['password'], config['database'])
  res = dbh.query(%Q{
        SELECT *
          FROM users
         WHERE state = 'active'
           AND login = '#{ENV['AUTHD_ACCOUNT'].gsub(/'/, "''")}'
         LIMIT 1
        })
  user = res.fetch_hash
  res.free
rescue Mysql::Error => e
ensure
  dbh.close if dbh
end

If we can’t find a user, we can fail fast and return “auth_ok:0”. If we do find one, we need to mimic the password-hashing scheme from our user model so that we can validate that the password we were given is correct. If the password does not match, then we fail, otherwise, we send back “auth_ok:1”, along with effective UNIX user and group ids and home directory path. Whether or not the user is authenticated, we send back “end”

validate-user
if user['crypted_password'].nil?
  # no active user with given login
  puts "auth_ok:0"
else
  # validate given password
  hashed_pw = Digest::SHA1.hexdigest("--#{user['salt']}--#{ENV['AUTHD_PASSWORD']}--")
  if user['crypted_password'] != hashed_pw
    puts "auth_ok:0"
  else
    puts "auth_ok:1"
    puts "uid:#{UID}"
    puts "gid:#{GID}"
    puts "dir:#{RAILS_ROOT}/private/ftp/#{user['login']}"
  end
endif user[‘crypted_password’].nil?

  # no active user with given login

  puts "auth_ok:0"

else

  # validate given password

  hashed_pw = Digest::SHA1.hexdigest("—#{user[‘salt’]}—#{ENV[‘AUTHD_PASSWORD’]}—")

  if user[‘crypted_password’] != hashed_pw

    puts "auth_ok:0"

  else

    puts "auth_ok:1"

    puts "uid:#{UID}"

    puts "gid:#{GID}"

    puts "dir:#{RAILS_ROOT}/private/ftp/#{user[‘login’]}"

  end

end
puts "end"

Here’s the final product: ftpd-auth-handler

This turned out to be a pretty straightforward implementation, but the functionality is quite useful for us. Shout out to the VideoPros crew for agreeing to share this — hopefully this will help someone who needs to integrate FTP with their Rails app!

Ever forget what all the Rails defined Date/DateTime/Time#strftime formats are? Or forget what ones you added to the project yourself?

Ala rake routes comes rake date_formats:

Sample output from a Rails 2.1 app:

Date

====

            db:‘%Y-%m-%d’   2008-08-20

  long_ordinal:‘&proc’      August 20th, 2008

          long:‘%B %e, %Y’  August 20, 2008

        rfc822:‘%e %b %Y’   20 Aug 2008

        number:‘%Y%m%d’     20080820

         short:‘%e %b’      20 Aug

DateTime

====

            db:‘%Y-%m-%d’   2008-08-20 16:56:21

  long_ordinal:‘&proc’      August 20th, 2008 16:56

          long:‘%B %e, %Y’  August 20, 2008 16:56

        rfc822:‘%e %b %Y’   Wed, 20 Aug 2008 16:56:21 -0600

        number:‘%Y%m%d’     20080820165621

         short:‘%e %b’      20 Aug 16:56
Time

====

            db:‘%Y-%m-%d %H:%M:%S’         2008-08-20 16:56:21

  long_ordinal:‘&proc’                     August 20th, 2008 16:56

          long:‘%B %d, %Y %H:%M’           August 20, 2008 16:56

        rfc822:‘%a, %d %b %Y %H:%M:%S %z’  Wed, 20 Aug 2008 16:56:21 -0600

         short:‘%d %b %H:%M’               20 Aug 16:56

        number:‘%Y%m%d%H%M%S’              20080820165621

          time:‘%H:%M’                     16:56

I’ve recently upgraded to OS X 10.5 (Leopard), and all-in-all, I’m pleased with the experience. My biggest issue has been the default stacks behavior—the icon changes to the last thing added to the stack, making visual identification unnecessarily cumbersome. I worked around this annoyance (as outlined here) by changing the sort to name rather than date added, and adding a dummy folder named “_1” that will sort to the top. For extra bonus points, I customized the icon of the dummy folder. For some yet unknown reason, the most recently downloaded item still peeks through from time to time, but it’s much better than before.

Maybe it’s my Windows history showing through, but I went with the “clean-sweep” erase and install method. For a non-developer, I’d probably recommend the upgrade (and in fact I used that method for my Father-in-law’s MacBook), but I had lots of custom bits scattered about my machine, and didn’t want to be chasing any incompatibility gremlins.

So now, to get my development environment set up on the new machine… Leopard includes a fairly complete Rails stack out of the box, with a non-broken Ruby, readline support, and most of the commonly used gems. Read more here
.

MySQL was not included, but the latest installer (mysql-5.0.45-osx10.4-i686.dmg) for 10.4 from dev.mysql.com downloads worked (mostly) fine. The Server and the StartupItem install and operate correctly. The PrefPane installs, but does not appear to actually do … anything. I’ll have to work on that, but I can live without it for now. After a bit of manual hacking on my database dump file from Tiger (where I was running a 5.1.x beta of MySQL), all my databases are back in place.

One last piece that I needed for my Rails apps—RMagick. I know it’s possible to install RMagick and its dependencies, um, “autoRMagickally” via a package management system like MacPorts or Fink, but I prefer not to. For some background on why not, you can read this article at hivelogic. The last time I was rebuilding my laptop and desktop near the same time, I put together a shell script to automate the process of installing RMagick. I got it back out and dusted off the cobwebs, and voila! RMagick on Leopard. (note: replace wget with “curl -O”, if you don’t have wget installed on your machine) Here’s the code:

install_rmagick.sh
#!/bin/sh

wget http://download.savannah.gnu.org/releases/freetype/freetype-2.3.5.tar.gz

tar xzvf freetype-2.3.5.tar.gz

cd freetype-2.3.5

./configure —prefix=/usr/local

make

sudo make install

cd ..
wget http://superb-west.dl.sourceforge.net/sourceforge/libpng/libpng-1.2.22.tar.bz2

tar jxvf libpng-1.2.22.tar.bz2

cd libpng-1.2.22

./configure —prefix=/usr/local

make

sudo make install

cd ..
wget ftp://ftp.uu.net/graphics/jpeg/jpegsrc.v6b.tar.gz

tar xzvf jpegsrc.v6b.tar.gz

cd jpeg-6b

ln -s `which glibtool` ./libtool

export MACOSX_DEPLOYMENT_TARGET=10.5

./configure —enable-shared —prefix=/usr/local

make

sudo make install

cd ..
wget ftp://ftp.remotesensing.org/libtiff/tiff-3.8.2.tar.gz

tar xzvf tiff-3.8.2.tar.gz

cd tiff-3.8.2

./configure —prefix=/usr/local

make

sudo make install

cd ..
wget http://jaist.dl.sourceforge.net/sourceforge/wvware/libwmf-0.2.8.4.tar.gz

tar xzvf libwmf-0.2.8.4.tar.gz

cd libwmf-0.2.8.4

make clean

./configure

make

sudo make install

cd ..
wget http://www.littlecms.com/lcms-1.17.tar.gz

tar xzvf lcms-1.17.tar.gz

cd lcms-1.17

make clean

./configure

make

sudo make install

cd ..
wget ftp://mirror.cs.wisc.edu/pub/mirrors/ghost/GPL/gs860/ghostscript-8.60.tar.gz

tar zxvf ghostscript-8.60.tar.gz

cd ghostscript-8.60/

./configure  —prefix=/usr/local

make

sudo make install

cd ..
wget ftp://mirror.cs.wisc.edu/pub/mirrors/ghost/GPL/current/ghostscript-fonts-std-8.11.tar.gz

tar zxvf ghostscript-fonts-std-8.11.tar.gz

sudo mv fonts /usr/local/share/ghostscript
wget http://imagemagick.site2nd.org/imagemagick/ImageMagick-6.3.5-9.tar.gz

tar xzvf ImageMagick-6.3.5-9.tar.gz

cd ImageMagick-6.3.5

export CPPFLAGS=-I/usr/local/include

export LDFLAGS=-L/usr/local/lib

./configure —prefix=/usr/local —disable-static —with-modules —without-perl —without-magick-plus-plus —with-quantum-depth=8 —with-gs-font-dir=/usr/local/share/ghostscript/fonts

make

sudo make install

cd ..
sudo gem install RMagick

Rails makes things easy on developers—maybe too easy. It’s not uncommon to reference an association while iterating over a collection of objects, resulting in a performance-devouring N+1 queries being executed. Of course, this is easily fixed with eager loading of the association, but you get my point. Rails can be a big gun, and it’s easy to blow off your foot.

Once your application is moved into production, it becomes important to keep an eye on performance over time in order to get a feel for trends and plan capacity intelligently. Numerous Rails performance measuring tools exist, but I find that it helps to correlate performance of your application with simultaneous lower-level performance metrics of your system (CPU and Memory usage, Load Average, etc). Besides that, hey! Pretty Graphs! Enter munin, an open-source, extensible monitoring tool with a number of out-of-the-box plugins that are useful to SysAdmin type folks. There’s a pretty decent (though brief) writeup on howtoforge that explains a bit more about how to go about getting your own munin. Go ahead and check it out — I’ll chill here until you’re back.

That wasn’t too bad, huh? Okay, so at this point you should have a working munin instance. It will take a few minutes of data collection before your graphs look like anything, but going forward, you’ll have historical graphs of several key metrics for managing your server(s). It’s outside the scope of this article, but you can also set up munin to monitor multiple servers on your network, and/or alert you when critical threshold levels are passed.

Well, knowing CPU usage is great, but it would also be nice to have some idea what the average user experience is like on your site. Does your Rails app perform differently at different times of day? How long does it typically take for your app to render a page? Is the majority of the time spent in the database, or rendering? Is the response time about the same between production deployment versions? I wanted answers to these questions too, so I came up with a small ruby program that watches your Rails log in realtime, and when munin asks, provides summary information about the performance of your application. Below, I’ll go over the code section by section; for those who want it now, scroll to the bottom of the article to find the download link.

The basis of the solution is a Ruby array: we stuff values into it, then compute the average of all the values and clear the array each time munin pings us. Every time a value is added, we also check it against the maximum already seen, so we can report the maximum response time in addition to the average. We keep three of these objects around, one each for DB, Rendering, and Total response times. There is also a mode that lets you look at the current value without consuming it — useful for peeking inside without affecting the data that munin will ultimately see. Here’s the class that implements this functionality:

accumulator
class Accumulator
  def initialize
    @values = Array.new()
    @max = 0
  endclass Accumulator

  def initialize

    values</span> = <span class="co">Array</span>.new()
    <span class="iv">max = 0

  end
def add(value)
@values << value
max</span> = value <span class="r">if</span> value &gt; <span class="iv">max
end
def average(read_only=false)
return_value = if @values.length == 0
nil
else
values</span>.inject(<span class="i">0</span>) {|sum,value| sum + value } / <span class="iv">values.length
end
@values = Array.new() unless read_only

return_value
end

def max(read_only=false)
return_value = @max
@max = 0 unless read_only
return_value
end
end

In the next section of the code, we build our accumulators, and begin tailing the logfile to extract performance numbers. This requires the file-tail gem, available from rubyforge. Note that in my setup, this file resides in a subdirectory under lib in RAILS_ROOT. If you choose to place this file elsewhere, you’ll have to adjust the path to the logfile accordingly. Another thing to note: in our environment, the load balancer continually pings a “heartbeat” action on each node to make sure it is still responsive. As we will be hitting this action repeatedly, it is engineered to be as lightweight as possible. Therefore, any numbers from it are pretty meaningless to our overall statistics, so we don’t want to include them. To keep these numbers from skewing our results, we define an IGNORE_PATTERNS regexp (earlier in the code). If the request matches a pattern we want to ignore, its statistics are not collected.

tail
LOGFILE = File.join(File.dirname(__FILE__), '..', '..', 'log', "#{RAILS_ENV}.log")
$response_data = { :total     => Accumulator.new(),
                   :rendering => Accumulator.new(),
                   :db        => AccumulatorLOGFILE = File.join(File.dirname(FILE), ‘..’, ‘..’, ‘log’, "#{RAILS_ENV}.log")

$response_data = { :total     => Accumulator.new(),

                   :rendering => Accumulator.new(),

                   :db        => Accumulator.new() }
Thread.abort_on_exception = true

logtail = Thread.new do

  File::Tail::Logfile.tail(LOGFILE) do |line|

    if line =~ /^Completed in /

      parts = line.split(/\s\|\s/)

      resp = parts.pop

      requested_url = resp[/http:\/\/[^\]]*/]

      next if requested_url =~ IGNORE_PATTERNS
parts.each do |part|
part.gsub!(/Completed in/, "total")
type, time, pct = part.split(/\s+/)
type = type.gsub(/:/,‘’).downcase.to_sym

$response_data[type].add(time.to_f)
end
end
end
end

So now we have a thread busy gathering our data—how can we expose the data to a munin plugin? There are multiple ways to do this, but I chose to use a small HTTP server listening to requests from the local machine only. We could build such a thing in Rails, but we really don’t need about 90% of the features Rails has to offer. Since we’re running Rails as a Mongrel cluster, we already have a perfect tool at our disposal for writing small HTTP request handlers in Ruby: Mongrel. Here are a couple of pages about how to get started writing Mongrel handlers —it’s pretty straightforward. Here’s our handler:

mongrel handler
class ResponseTimeHandler < Mongrel::HttpHandler
  def initialize(method)
    @method = method
  endclass ResponseTimeHandler < Mongrel::HttpHandler

  def initialize(method)

    @method = method

  end
def process(request, response)
response.start(200) do |head,out|
debug = Mongrel::HttpRequest.query_parse(request.params["QUERY_STRING"]).has_key? "debug"
head["Content-Type"] = "text/plain"
output = $response_data.map do |k,v|
value = v.send(@method, debug)
formatted = value.nil? ? ‘U’ : sprintf(‘%.5f’, value)

"#{k}.value #{formatted}"
end.join("\n")
output << "\n"
out.write output
end
end
end
h = Mongrel::HttpServer.new("127.0.0.1", PORT)

h.register("/avg_response_time", ResponseTimeHandler.new(:average))

h.register("/max_response_time", ResponseTimeHandler.new(:max))

h.run.join

The handler is generic so that it can call an arbitrary method on our collection of data arrays, so we can set up one URI for the average, and one for the maximum. With this in place, we can write a simple munin plugin script that uses Net::HTTP to query our Mongrel to get at the performance data.

Basically, a munin plugin has two usage scenarios. When called with the single argument “config”, it should output information about itself in a format that Munin understands. This includes how to label the chart, scaling information, how many series will be included on the chart, etc. When called with no arguments, the plugin should output the current values of each series. For more information on writing your own munin plugin, start with the HowToWritePlugins munin wiki page. And now, our plugin script:

rails_response_time
#!/usr/bin/env ruby#!/usr/bin/env ruby
# munin plugin to render rails response time graphs

# link to /etc/munin/plugins/avg_response_time and /etc/munin/plugins/max_response_time
require ‘open-uri’

PORT = ENV[‘PORT’] || "8888"
def config

  title = File.basename($0).split(‘_’).map{|s| s.capitalize }.join(‘ ’)

  config=<<END_CONFIG

graph_title #{title}

graph_vlabel response time

graph_category rails

total.label total

rendering.label rendering

db.label db

END_CONFIG

  puts config

end
def get_data(read_only=false)

  qs = read_only ? ‘?debug’ : ‘’

  puts open("http://127.0.0.1:#{PORT}/#{File.basename($0)}#{qs}").read

end
case ARGV.first

when ‘config’

  config

when ‘debug’

  get_data(true)

else

  get_data

end

The script will examine the name with which you linked it in to the munin plugins directory to determine which URI to query. I have also added a debug mode that will show you the current values, so you’re not consuming any data that munin needs to see for an accurate graph. The final piece is a small Daemons wrapper script to control the main log-tailing process, and you should be set. Make sure to restart munin-node so it will notice the new plugins, and after a while, you’ll see something like this:

It is worth noting that the numbers from the Rails log might not be 100% accurate, and this won’t replace the results that you can get from seriously profiling your application. Also, the information you are getting is a bit generic — all actions are lumped together, so there is not a lot of information about the cause of the performance problem. But, for insight into your production application performance, this setup should at least give you some indications about how well your baby is playing in the interwebs.

Download rails_log_monitor.rb

Download rails_response_time

I’ve updated my downloader from earlier to include all sorts of fancy options. It no longer requires wget, it just uses open-uri. It can give the files a fancy name. It can be told where to download the files to. It will skip files that won’t download for some reason. It will even butter your toast if you can find the correct command line switch.

It’s about 3 times bigger than the previous one. But maybe you can learn a little more about optparse, hpricot, file handling, and error handling along the way.

Here it is:

#!/usr/bin/env ruby

require 'optparse'

OPTIONS = { :Verbose => false,
            :Force => false,
            :DownloadDir => '.',
            :DescriptiveFilenames => true
          }
OptionParser.new do |opts|
  opts.banner = "Usage: #{$0} [options]"

  opts.on("-v", "--[no-]verbose", "Run verbosely, default #{OPTIONS[:Verbose]}") do |verbose|
    OPTIONS[:Verbose] = verbose
  end
  opts.on("-f", "--[no-]force", "Force downloads, default #{OPTIONS[:Force]}") do |force|
    OPTIONS[:Force] = force
  end
  opts.on("-d", "--[no-]descriptive", "Use long descriptive filenames, default #{OPTIONS[:DescriptiveFilenames]}") do |long|
    OPTIONS[:DescriptiveFilenames] = long
  end
  opts.on("-p", "--path PATH", "Path to download to, default #{OPTIONS[:DownloadDir]}") do |path|
    OPTIONS[:DownloadDir] = path
  end
  opts.on_tail("-h", "--help", "Print help message") do |help|
    puts opts
    exit
  end
end.parse!

require 'rubygems'
require 'hpricot'
require 'open-uri'
require 'fileutils'

BASE_URL = 'http://www.web2expo.com'

def log(str)
  puts str if OPTIONS[:Verbose]
end

def download(href, filename)
  url = "#{BASE_URL}#{URI.escape(href)}"
  download_file = File.join(OPTIONS[:DownloadDir], filename)
  if OPTIONS[:Force] || !File.exists?(download_file)
    log "downloading #{File.basename(href)}..."
    begin
      File.open(download_file, 'w') { |f| f.write(open(url).read)}
      log "\tsaved as #{download_file}"
    rescue Object => e
      FileUtils.rm(download_file)
      $stderr.puts "ERROR downloading #{url}: #{e.message}"
    end
  else
    log "skipping #{File.basename(href)}... already downloaded as #{download_file}"
  end
end

FileUtils.mkdir_p(OPTIONS[:DownloadDir])
h = Hpricot(open("#{BASE_URL}/pub/w/51/presentations.html"))
h.search('div.presentation').each do |presentation_node|
  href = presentation_node.at('a[@href^="/presentations/rails2007/"]')[:href]
  if OPTIONS[:DescriptiveFilenames]
    name = presentation_node.at('b a').inner_text.strip
    text = presentation_node.inner_text
    speaker = text[/Speaker\(s\):\s+(.*)\s*$/, 1]
    date = Date.parse(text[/Presentation Date:\s+(.*)\s*$/, 1])
    filename = [speaker, date, name, File.basename(href)].compact.map { |s| s.to_s.strip.gsub(/[^\w\.]/, '_').squeeze('_') }.join('-')
  else
    File.basename(href)
  end
  download(href, filename)
end

Updated: Now more bloated!

#!/usr/bin/env ruby

require 'rubygems'
require 'hpricot'
require 'open-uri'

base = 'http://www.web2expo.com'
h = Hpricot(open("#{base}/pub/w/51/presentations.html"))

h.search('div .presentation > a[@href^="/presentations/rails2007/"]').each do |a|
  url = "#{base}#{a[:href]}"
  if File.exists?(File.basename(url))
    puts "skipping #{url}... already downloaded"
  else
    puts "downloading #{url}..."
    `wget --quiet #{url}`
  end
end