Friday, April 20, 2012

Deconstructing an OO Blog Designs in Ruby 1.9

I've become interested in Avdi Grimm's new book Object on Rails, however I found the code to be terse. Avdi is an expert Rubyist and he makes extensive use of Ruby 1.9 with minimal explanation. In all fairness, he lobbies you to buy Peter Cooper's Ruby 1.9 Walkthrough. Instead of purchasing the videos, I wanted to try and deconstruct them myself.

In his first chapter featuring code, Mr. Grimm creates a Blog and Post class. For those of you who remember the original Rails blog demo, the two couldn't look more different.

Blog#post_source

In an effort to encourage Rails developers to think about relationships between classes beyond ActiveRecord::Relation, he creates his own interface for defining how a Blog should interact with a "post source".

1
2
3
4
5
6
7
8
9
10
class Blog
  # ...
  attr_writer :post_source
   
  private
  def post_source
    @post_source ||= Post.public_method(:new)
  end
end

The code above defines the Blog class and makes available post_source= via the attr_writer method. Additionally, it defines the attribute reader as a private method. The idea being that a private method can be changed without breaking the class's API. If we decide we want a new default Post source, we can do it safely.

The magic of this code is in defining the post source as a class's method, in this case, Post.public_method(:new). The #public_method method is defined by Ruby's Object class and is similar to #method method. In short, it gives us a way of not directly calling Post.new, but instead, referring to the method that's responsible for creating new posts. This is logical if you remember that the name of this method is #post_source.

Now let's look how he puts post_source into action.

1
2
3
4
5
6
7
8
9
class Blog
  # ...
  def new_post
    post_source.call.tap do |p|
      p.blog = self
    end
  end
  # ...
end

During my first reading, it wasn't clear at all what was going on here, but if we remember that post_source is responsible for returning the method need to "call", we know that post_source.call is equivalent to Post.new. For the sake of clarity-while-learning for those not familiar with post_source.call, let's substitute it with something more readable so we can understand how tap is being employed.

1
2
3
4
5
6
7
8
class Blog
  # ...
  def new_post
    Post.new.tap do |p|
      p.blog = self
    end
  end
end

The tap method is available to all Ruby objects and serves as a way to have a block "act on" the method's caller and return the object called. Per the docs, "the primary purpose of this method is to 'tap into' a method chain, in order to perform operations on intermediate results within the chain". For some examples on using tap see MenTaLguY's post on Eavesdropping on Expressions. As he says in his post, "you can insert your [code] just about anywhere without disturbing the flow of data". Neat.

In this case, it's being used to tap into the process of creating a new blog post and define the blog to which that post belongs. Because tap returns the object it modifies, #new_post returns the post now assigned to the blog.

Brining it All Together

Avdi's approach may seem cumbersome at first, and it is compared to "the Rails way." But in general, that's the whole point of Object on Rails; to challenge you to see beyond a generic solution to a problem (in this case defining relationships between classes) so you can build more flexible solutions. Let's see some interesting things we might be able to do with this more flexible Blog class. We can imagine this same Blog class being able to handle posts from all sorts of different sources. Let's see if we can get creative.

1
2
3
4
5
6
7
8
class EmailPost < ActionMailer::Base
  def receive(message)
    @blog = Blog.find_by_owner_email(message.from)
    @blog.post_source = EmailPost.public_method(:new)
    @email_post = @blog.new_post(params[:email_post])
    @email_post.publish
  end
end

With this little snippet, we're able to use the Blog class to process a different sort of post. We simply let the blog know the method to call when we want a new post and pass along the arguments we'd expect. Let's see if we can think of something else that's creative.

1
2
3
4
5
6
7
8
9
10
11
12
13
require 'feedzirra'
# execute regularly with cronjob call like curl -d "blog_id=1&url=http://somefeed.com" http://myblog.com/feed_poster"
 
class FeedPostersController
  def create
    @feed = Feedzirra::Feed.fetch_and_parse(params[:url])
    @blog = Blog.find(params[:blog_id])
    @post.post_source = FeedPost.public_method(:new)
    @feed.entries.each do |entry|
      @blog.new_post(entry)
    end
  end
end

We could imagine the FeedPost.new method being the equivalent of a retweet for your blog using an RSS feed! Try having the blog class doing this with an ActiveRecord association! Seems to me the Blog class might need to get a bit more complex to support all these Post sources which makes post_source.call.tap look pretty good!

Tuesday, April 17, 2012

Monitoring cronjob exit codes with Nagios

If you're like me, you've got cronjobs that make email noise if there is an error. While email based alerts are better than nothing, it'd be best to integrate this kind of monitoring into Nagios. This article will break down how to monitor the exit codes from cronjobs with Nagios.

Tweaking our cronjob

The monitoring plugin depends on being able to read some sort of log output file which includes an exit code. The plugin also assumes that the log will be truncated with every run. Here's an example of a cronjob entry which meets those requirements:

1
rsync source dest 2>&1 > /var/log/important_rsync_job.log; echo "Exit code: $?" >> /var/log/important_rsync_job.log

So let's break down a couple of the more interesting points in this command:

  • 2>&1 sends the stderr output to stdout so it can be captured in our log file
  • Notice the single > which will truncate the log every time it is run
  • $? returns the exit code of the last run command
  • Notice the double >> which will append to the log file our exit code

Setting up the Nagios plugin

The check_exit_code plugin is available on GitHub, and couldn't be easier to setup. Simply specify the log file to monitor and the frequency with which it should be updated. Here's the usage statement:

1
2
3
4
5
6
7
8
9
Check log output has been modified with t minutes and contains "Exit code: 0".
        If "Exit code" is not found, assume command is running.
        Check assumes the log is truncated after each command is run.
         
        --help      shows this message
        --version   shows version information
 
        -f          path to log file
        -t          Time in minutes which a log file can be unmodified before raising CRITICAL alert

The check makes sure the log file has been updated within t minutes because we want to check that our cronjob is not only running successfully, but running regularly. Perhaps this should be an optional parameter, or this should be called check_cronjob_exit_code, but for right now, it's getting the job done and cutting back on email noise.

Sunday, March 18, 2012

Check JSON responses with Nagios

As the developer's love affair with JSON continues to grow, the need to monitor successful JSON output does as well. I wanted a Nagios plugin which would do a few things:

  • Confirm the content-type of the response header was "application/json"
  • Decode the response to verify it is parsable JSON
  • Optionally, verify the JSON response against a data file

Verify content of JSON response

For the most part, Perl's LWP::UserAgent class makes short work of the first requirement. Using $response->header("content-type") the plugin is able to check the content-type easily. Next up, we use the JSON module's decode function to see if we can successfully decode $response->content.

Optionally, we can give the plugin an absolute path to a file which contains a Perl hash which can be iterated through in attempt to find corresponding key/value pairs in the decoded JSON response. For each key/value in the hash it doesn't find in the JSON response, it will append the expected and actual results to the output string, exiting with a critical status. Currently there's no way to check a key/value does not appear in the response, but feel free to make a pull request on check_json on my GitHub page.

Thursday, March 15, 2012

Check HTTP redirects with Nagios

Often times there are critical page redirects on a site that may want to be monitored. Often times, it can be as simple as making sure your checkout page is redirecting from HTTP to HTTPS. Or perhaps you have valuable old URLs which Google has been indexing and you want to make sure these redirects remain in place for your PageRank. Whatever your reason for checking HTTP redirects with Nagios, you'll find there are a few scripts available, but none (that I found) which are able to follow more than one redirect. For example, let's suppose we have a redirect chain that looks like this:

1
http://myshop.com/cart >> http://www.myshop.com/cart >> https://www.mycart.com/cart

Following multiple redirects

In my travels, I found check_http_redirect on Nagios Exchange. It was a well designed plugin, written by Eugene Kovalenja in 2009 and licensed under GPLv2. After experimenting with the plugin, I found it was unable to traverse multiple redirects. Fortunately, Perl's LWP::UserAgent class provides a nifty little option called max_redirect. By revising Eugene's work, I've exposed additional command arguments that help control how many redirects to follow. Here's a summary of usage:

1
2
3
4
5
6
-U          URL to retrieve (http or https)
        -R          URL that must be equal to Header Location Redirect URL
        -t          Timeout in seconds to wait for the URL to load. If the page fails to load,
                    check_http_redirect will exit with UNKNOWN state (default 60)
        -c          Depth of redirects to follow (default 10)
        -v          Print redirect chain

If check_http_redirect is unable to find any redirects to follow or any of the redirects results in a 4xx or 5xx status code returned, the plugin will report a critical state code and the nature of the problem. Additionally, if the number of redirects exceeds the depth of redirects to follow as specified in the command arguments, it will notify you of this and exit with an unknown state code. An OK status will be returned only if the redirects result in a successful response to a URL which is a regex match against the options specified in the R argument.

The updated check_http_redirect plugin is available on my GitHub page along with several other Nagios plugins I'll write about in the coming weeks. Pull requests welcome, and thank you to Eugene for his original work on this plugin.

Thursday, March 1, 2012

IPv6 Tunnels with Debian/Ubuntu behind NAT

As part of End Point's preparation for World IPv6 Launch Day, I was asked to get my IPv6 certification from Hurricane Electric. It's a fun little game-based learning program which had me setup a IPv6 tunnel. IPv6 tunnels are used to provide IPv6 for those whose folks whose ISP or hosting provider don't currently support IPv6, by "tunneling" it over IPv4. The process for creating a tunnel is straight forward enough, but there were a few configuration steps I felt could be better explained.

After creating a tunnel, Hurricane Electric kindly provides a summary of your configuration and offers example configurations for several different operating systems and routers. Below is my configuration summary and the example generated by Hurricane Electric.

However, entering these commands change won't survive a restart. For Debian/Ubuntu users an update in /etc/network/interfaces does the trick.

1
2
3
4
5
6
7
8
9
#/etc/network/interfaces
auto he-ipv6
iface he-ipv6 inet6 v4tunnel
  address 2001:470:4:9ae::2
  netmask 64
  endpoint 209.51.161.58
  local 204.8.67.188
  ttl 225
  gateway 2001:470:4:9ae::1

Firewall Configuration

If you're running UFW the updates to /etc/default/ufw are very straightforward. Simply change the IPV6 directive to yes. Restart the firewall and your network interfaces and you should be able to ping6 ipv6.google.com. I also recommend hitting http://test-ipv6.com/ for a detailed configuration test.

Behind NAT

If you're behind a NAT, the configuration needs to be tweaked a bit. First, you'll want to setup a static IP address behind your router. If you're router supports configuration of forwarding more than just TCP/UDP, you'll want to forward protocol 41 (aka IPv6) (NOT PORT 41), which is responsible for IPv6 tunneling over IPv4, to your static address. If you've got a consumer grade router that doesn't support this, you'll just have to put your machine in the DMZ, thus putting your computer "in front" of your router's firewall. Please make sure you are running a local software firewall if you chose this option.

After handling the routing of protocol 41, there is one small configuration change to /etc/network/interfaces. You must change your tunnel's local address from your public IP address, to your private NATed address. Here is an example configuration including both the static IP configuration and the updated tunnel configuration.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#/etc/network/interfaces
auto eth0
iface eth0 inet static
  address 192.168.0.50
  netmask 255.255.255.0
  gateway 192.168.0.1
 
auto he-ipv6
iface he-ipv6 inet6 v4tunnel
  address 2001:470:4:9ae::2
  netmask 64
  endpoint 209.51.161.58
  <b>local 192.168.0.50</b>
  ttl 225
  gateway 2001:470:4:9ae::1

Don't forget to restart your networking interfaces after these changes. I found a good ol' restart was helpful as well, but of course, we don't have this luxury in production, so be careful!

Checking IPv6

If you're reading this article, you're probably responsible for several hosts. For a gentle reminder which of your sites you've not yet setup IPv6, I recommend checking out IPvFoo for Chrome or 4or6 for Firefox. These tools make it easy for you to see which of your sites are ready for World IPv6 Launch Day!

Getting Help

Hurricane Electric provides really great support for their IPv6 tunnel services (which is completely free). Simply email ipv6@he.net and provide them with some useful information such as:

1
2
3
4
cat /etc/network/interfaces
cat netstat -nrA inet6  (these are your IPv6 routing tables)
cat /etc/default/ufw
relevant router configurations
I was very impressed to get a response from a competent person in 15 minutes! Sadly, there is one downside to using this tunnel; IRC is not an allowed.
Due to an increase in IRC abuse, new non-BGP tunnels now have IRC blocked by default. If you are a Sage, you can re-enable IRC by visiting the tunnel details page for that specific tunnel and selecting the 'Unblock IRC' option. Existing tunnels have not been filtered.
I guess ya gotta earn it to use IRC over your tunnel. Good luck!

Saturday, January 21, 2012

MySQL replication monitoring on Ubuntu 10.04 with Nagios and NRPE

If you're using MySQL replication, then you're probably counting on it for some fairly important need. Monitoring via something like Nagios is generally considered a best practice. This article assumes you've already got your Nagios server setup and your intention is to add a Ubuntu 10.04 NRPE client. This article also assumes the Ubuntu 10.04 NRPE client is your MySQL replication master, not the slave. The OS of the slave does not matter.

Getting the Nagios NRPE client setup on Ubuntu 10.04

At first it wasn't clear what packages would be appropriate packages to install. I was initially misled by the naming of the nrpe package, but I found the correct packages to be:

1
sudo apt-get install nagios-nrpe-server nagios-plugins

The NRPE configuration is stored in /etc/nagios/nrpe.cfg, while the plugins are installed in /usr/lib/nagios/plugins/ (or lib64). The installation of this package will also create a user nagios which does not have login permissions. After the packages are installed the first step is to make sure that /etc/nagios/nrpe.cfg has some basic configuration.

Make sure you note the server port (defaults to 5666) and open it on any firewalls you have running. (I got hung up because I forgot I have both a software and hardware firewall running!) Also make sure the server_address directive is commented out; you wouldn't want to only listen locally in this situation. I recommend limiting incoming hosts by using your firewall of choice.

Choosing what NRPE commands you want to support

Further down in the configuration, you'll see lines like command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10. These are the commands you plan to offer the Nagios server to monitor. Review the contents of /usr/lib/nagios/plugins/ to see what's available and feel free to add what you feel is appropriate. Well designed plugins should give you a usage if you execute them from the command line. Otherwise, you may need to open your favorite editor and dig in!

After verifying you've got your NRPE configuration completed and made sure to open the appropriate ports on your firewall(s), let's restart the NRPE service:

1
service nagios-nrpe-server restart

This would also be an appropriate time to confirm that the nagios-nrpe-server service is configured to start on boot. I prefer the chkconfig package to help with this task, so if you don't already have it installed:

1
2
3
4
5
6
7
8
sudo apt-get install chkconfig
chkconfig | grep nrpe
 
# You should see...
nagios-nrpe-server     on
 
# If you don't...
chkconfig nagios-nrpe-server on

Pre flight check - running check_nrpe

Before going any further, log into your Nagios server and run check_nrpe and make sure you can execute at least one of the commands you chose to support in nrpe.cfg. This way, if there are any issues, it is obvious now, while we've not started modifying your Nagios server configuration. The location of your check_nrpe binary may vary, but the syntax is the same:

1
check_nrpe -H host_of_new_nrpe_client -c command_name

If your command output something useful and expected, your on the right track. A common error you might see: Connection refused by host. Here's a quick checklist:

  • Did you start the nagios-nrpe-server service?
  • Run netstat -lunt on the NRPE client to make sure the service is listening on the right address and ports.
  • Did you open the appropriate ports on all your firewall(s)?
  • Is there NAT translation which needs configuration?

Adding the check_mysql_replication plugin

There is a lot of noise out there on Google for Nagios plugins which offer MySQL replication monitoring. I wrote the following one using ideas pulled from several existing plugins. It is designed to run on the MySQL master server, check the master's log position and then compare it to the slave's log position. If there is a difference in position, the alert is considered Critical. Additionally, it checks the slave's reported status, and if it is not "Waiting for master to send event", the alert is also considered critical. You can find the source for the plugin at my Github account under the project check_mysql_replication. Pull that source down into your plugins directory (/usr/lib/nagios/plugins/ (or lib64)) and make sure the permissions match the other plugins.

With the plugin now in place, add a command to your nrpe.cfg.

1
command[check_mysql_replication]=sudo /usr/lib/nagios/plugins/check_mysql_replication.sh -H <slave_host_address></slave_host_address>

At this point you may be saying, WAIT! How will the user running this command (nagios) have login credentials to the MySQL server? Thankfully we can create a home directory for that nagios user, and add a .my.cnf configuration with the appropriate credentials.

1
2
3
4
5
6
7
8
9
10
11
12
usermod -d /home/nagios nagios #set home directory
mkdir /home/nagios
chmod 755 /home/nagios
chown nagios:nagios /home/nagios
 
# create /home/nagios/.my.cnf with your preferred editor with the following:
[client]
user=example_replication_username
password=replication_password
 
chmod 600 /home/nagios/.my.cnf
chown nagios:nagios /home/nagios/.my.cnf

This would again be an appropriate place to run a pre flight check and run the check_nrpe from your Nagios server to make sure this configuration works as expected. But first we need to add this command to the sudoer's file.

1
nagios ALL= NOPASSWD: /usr/lib/nagios/plugins/check_mysql_replication.sh

Wrapping Up

At this point, you should run another check_nrpe command from your server and see the replication monitoring report. If not, go back and check these steps carefully. There are lots of gotchas and permissions and file ownership are easily overlooked. With this in place, just add the NRPE client using the existing templates you have for your Nagios servers and make sure the monitoring is reporting as expected.

Saturday, January 14, 2012

Using Disqus and Ruby on Rails

Recently, I posted about how to import comments from a Ruby on Rails app to Disqus. This is a follow up to that post where I outline the implementation of Disqus in a Ruby on Rails site. Disqus provides what it calls Universal Code which can be added to any site. This universal code is just JavaScript, which asynchronously loads the Disqus thread based on one of two unique identifiers Disqus uses.

Disqus in a development environment

Before we get started, I'd recommend that you have two Disqus "sites"; one for development and one for production. This will allow you to see real content and experiment with how things will really behave once you're in production. Ideally, your development server would be publicly accessible to allow you to fully use the Disqus moderation interface, but it isn't required. Simply register another Disqus site, and make sure that you have your shortname configured by environment. Feel free to use whatever method you prefer for defining these kinds of application preferences. If you're looking for an easy way, considering checking out my article on Working with Constants in Ruby. It might look something like this:

1
2
3
# app/models/article.rb
 
DISQUS_SHORTNAME = Rails.env == "development" ? "dev_shortname".freeze : "production_shortname".freeze

Disqus Identifiers

Each time you load the universal code, you need to specify a few configuration variables so that the correct thread is loaded:

  • disqus_shortname: tells Disqus which website account (called a forum on Disqus) this system belongs to.
  • disqus_identifier: tells Disqus how to uniquely identify the current page.
  • disqus_url: tells Disqus the location of the page for permalinking purposes.
Let's create a Rails partial to set up these variables for us, so we can easily call up the appropriate comment thread.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# app/views/disqus/_thread.html.erb
# assumes you've passed in the local variable 'article' into this partial
 
<div id="disqus_thread"></div>
<script type="text/javascript">
 
    var disqus_shortname = '<%= Article::DISQUS_SHORTNAME %>';
    var disqus_identifier = '<%= article.id %>';
    var disqus_url = '<%= url_for(article, :only_path => false) %>';
 
    /* * * DON'T EDIT BELOW THIS LINE * * */
    (function() {
        var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
        dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
        (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
    })();
</script>

The above code will populate the div#disqus_thread with the correct content based on your disqus_identifier. By setting up a single partial that will always render your threads, it becomes very easy to adjust this code if needed.

Disqus Identifier Gotcha

We found during our testing a surprising and unexpected behavior in how Disqus associates a thread to a URL. In our application, the landing page was designed to show the newest article as well as the Disqus comments thread. We found that once a new article was posted, the comments from the previous article were still shown! It seems Disqus ignored the unique disqus_identifier we had specified and instead associated the thread with the landing page URL. In our case, a simple routing change allowed us to forward the user to the unique URL for that content and thread. In your case, there may not be such an easy work around, so be certain you include both the disqus_identifier and disqus_url JavaScript configuration variables above to minimize the assumptions Disqus will make. When at all possible, always use unique URLs for displaying Disqus comments.

Comment Counters

Often an index page will want to display a count of how many comments are in a particular thread. Disqus uses the same asynchronous approach to loading comment counts. Comment counts are shown by adding code such as the following where you want to display your count:

1
2
3
4
5
6
7
8
9
10
# HTML
<a href="http://example.com/article1.html#disqus_thread"
   data-disqus-identifier="<%=@article.id%>">
This will be replaced by the comment count
</a>
 
# Rails helper
<%= link_to "This will be replaced by the comment count",
    article_path(@article, :anchor => "disqus_thread"),
    :"data-disqus-identifer" => @article.id %>

At first this seemed strange, but it is the exact same pattern used to display the thread. It would likely be best to remove the link text so nothing is shown until the comment count is loaded, but I felt for my example, having some meaning to the test would help understanding. Additionally, you'll need to add the following JavaScript to your page.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# app/view/disqus/_comment_count_javascript.html.erb
# add once per page, just above </body>
 
<script type="text/javascript">
    
    var disqus_shortname = '<%= Article::DISQUS_SHORTNAME %>';
 
    /* * * DON'T EDIT BELOW THIS LINE * * */
    (function () {
        var s = document.createElement('script'); s.async = true;
        s.type = 'text/javascript';
        s.src = 'http://' + disqus_shortname + '.disqus.com/count.js';
        (document.getElementsByTagName('HEAD')[0] || document.getElementsByTagName('BODY')[0]).appendChild(s);
    }());
</script>

Disqus recommends adding it just before the closing </body> tag. You only need to add this code ONCE per page, even if you're planning on showing multiple comment counts on a page. You will need this code on any page with a comment count, so I do recommend putting it in a partial. If you wanted, you could even include it in a layout.

Styling Comment Counts

Disqus provides extensive CSS documentation for its threads, but NONE for its comment counters. In our application, we had some very particular style requirements for these comment counts. I found that in Settings > Appearance, I could add HTML tags around the output of the comments.

This allowed me to style my comments as needed, although these fields are pretty small, so make sure to compress your HTML as much as possible.