Jack Christensen

Surus - PostgreSQL Acceleration for ActiveRecord

2012-02-16T00:00:00-06:00

Surus accelerates ActiveRecord with PostgreSQL specific types and functionality. It enables indexed searching of serialized arrays and hashes. It also can control PostgreSQL synchronous commit behavior. By relaxing PostgreSQL’s durability guarantee, transaction commit rate can be increased by 50% or more.

Installation

gem install surus

Or add to your Gemfile.

gem 'surus'

Hstore

Hashes can be serialized to an hstore column. hstore is a PostgreSQL key/value type that can be indexed for fast searching.

class User < ActiveRecord::Base
  serialize :properties, Surus::Hstore::Serializer.new
end

User.create :properties => { :favorite_color => "green", :results_per_page => 20 }
User.create :properties => { :favorite_colors => ["green", "blue", "red"] }

Even though the underlying hstore can only use strings for keys and values (and NULL for values) Surus can successfully maintain type for integers, floats, bigdecimals, dates, and any value that YAML can serialize. It does this by storing an extra key value pair (or two) to maintain type information.

Because it falls back to YAML serialization for complex types, this means that nested data structures can be serialized to an hstore. In other words, any hash that can be serialized with the normal Rails YAML serialization can be serialized with Surus. But you can get the benefits of PostgreSQL indexing on the top level keys and values for free.

Hstores can be searched with helper scopes.

User.hstore_has_pairs(:properties, "favorite_color" => "green")
User.hstore_has_key(:properties, "favorite_color")
User.hstore_has_all_keys(:properties, "favorite_color", "gender")
User.hstore_has_any_keys(:properties, "favorite_color", "favorite_artist")

Read more in the PostgreSQL hstore documentation.

Array

Ruby arrays can be serialized to PostgreSQL arrays. Surus includes support for text, integer, float, and decimal arrays.

class User < ActiveRecord::Base
  serialize :permissions, Surus::Array::TextSerializer.new
  serialize :favorite_integers, Surus::Array::IntegerSerializer.new
  serialize :favorite_floats, Surus::Array::FloatSerializer.new
  serialize :favorite_decimals, Surus::Array::DecimalSerializer.new
end

User.create :permissions => %w{ read_notes write_notes, manage_topics },
  :favorite_integers => [1, 2, 3],
  :favorite_floats => [1.3, 2.2, 3.1],
  :favorite_decimals => [BigDecimal("3.14"), BigDecimal("4.23"]

Arrays can be searched with helper scopes.

User.array_has(:permissions, "admin")
User.array_has(:permissions, "manage_accounts", "manage_users")
User.array_has_any(:favorite_integers, 7, 11, 42)

Synchronous Commit

PostgreSQL can trade durability for speed. By disabling synchronous commit, transactions will return before the data is actually stored on the disk. This can be substantially faster, but it entails a short window where a crash could cause data loss (but not data corruption). This can be enabled for an entire session or per transaction.

User.synchronous_commit # -> true

User.transaction do
  User.synchronous_commit false
  @user.save
end # This transaction can return before the data is written to the drive

# synchronous_commit returns to its former value outside of the transaction
User.synchronous_commit # -> true

# synchronous_commit can be turned off permanently
User.synchronous_commit false

Head over to Github for more details.

tern - The SQL Fan's Migrator

2011-02-07T00:00:00-06:00

I just released tern – a new migration system.

Tern is designed to simplify migrating database schemas with views, functions, triggers, constraints and other objects where altering one may require dropping and recreating others. For example, if view A selects from view B which selects from table C, then altering C could require five steps: drop A, drop B, alter C, recreate B, and recreate A. Tern can be told to alter C and it will automatically perform the other four steps.

Head over to Github for more details.

tod - Time of Day for Ruby

2010-11-29T00:00:00-06:00

I just released tod – a new ruby gem for time of day handling. It handles parsing, strftime, comparison, and arithmetic.

gem install tod

require 'tod'

TimeOfDay.parse "8"                            # => 08:00:00
TimeOfDay.parse "8pm"                          # => 20:00:00
TimeOfDay.parse "1230"                         # => 12:30:00
TimeOfDay.parse "3:25:58"                      # => 03:25:58
TimeOfDay.parse "515p"                         # => 17:15:00

TimeOfDay.new(8) + 3600                        # => 09:00:00
TimeOfDay.new(8) - 3600                        # => 07:00:00
TimeOfDay.new(23,59,45) + 30                   # => 00:00:15

TimeOfDay.new(8) < TimeOfDay.new(9)            # => true

TimeOfDay.new(8,30).strftime("%H:%M")          # => "08:30"
TimeOfDay.new(17,15).strftime("%I:%M %p")      # => "05:15 PM"

Head over to Github for more details.

assert_db_rejects - Test database constraints in Ruby on Rails

2009-11-14T00:00:00-06:00

If you believe in using database constraints in addition to ActiveRecord validations, you may find assert_db_rejects useful to help you test them.

require 'test_helper'

class PersonTest < ActiveSupport::TestCase
  
  test "database blocks duplicate names" do
    Person.create!( :name => "Test" )
    
    # Assume the people table has a not null constraint on name. The assertion
    # will pass if a ActiveRecord::StatementInvalid is raised when saving p.
    # Validations are bypassed by assert_db_rejects as the purpose is to test
    # the database and not the model
    assert_db_rejects( Person.new( :name => "Test" ) )
  end
end

Solving a Math Puzzle with Ruby

2009-08-08T00:00:00-05:00

On page 212 of the excellent book The Art of Game Design by Jesse Schell, the author presents the following puzzle as a negative example of the puzzle principle of “make it easy to get started”. The problem is to find the value of all the letters.

   CEI
 ×  DA
   GCH
+ DFB 
  ADFH

Next the author has to say that “most players are at a complete loss as to how to begin solving a puzzle like this”. After reading this I had to prove that I could beat it. I started on what turned into a quest of seven hours spread over two days.

Some initial examination of the equation yields a few clues. To start with, a zero would not typically be written on the leftmost digit of a multiplier. So C and D cannot be zero. Furthermore, D×C=D. This means that either C or D must be 1. Since D×I does not equal D we can prove that C is 1. I continued on the path of slowly narrowing done the possible values of each letter until, in theory, I should have arrived at the solution. It did not work out this way in practice. Three times I reached a point where I had a logical contradiction such as having both proved that D was 6 and that D must be less than 5.

After this, I really started to wonder if the problem had a solution. After all, the author had previously discussed player frustration and unfair challenges, and the puzzle was a negative example. Maybe there was no solution. On the other hand, my attack on the problem had been based on a great number of small assertions that built on all previous assertions. One inaccurate assertion would break the whole chain. At this point, I had put four hours into this puzzle and I was not about to stop until I had solved it, or I had proved there is no solution.

It was time to bring the big guns into play and just brute-force the problem. It should be a simple matter to write a script in Ruby that tests every combination of values for a solution. Either I would have the answer, or I would prove there was no answer. As it turned out it, took about three hours to get the script right. Some of that was due to the fact that I decided to try some new features of Ruby 1.9 such as chaining iterators together, but most of it was due to difficulties perfecting a function that would yield every combination of values. But finally I built a solver for this type of puzzle.

def chars_to_number( digits, char_values )
  digits.reverse.chars.each_with_index.inject(0) do |sum, char_and_place|
    char, place = char_and_place 
    sum + char_values[char] * (10**place)
  end
end

def valid_multiplication?( char_mult1, char_mult2, char_product, char_values )
  mult1 = chars_to_number( char_mult1, char_values )
  mult2 = chars_to_number( char_mult2, char_values )
  product = chars_to_number( char_product, char_values )
  mult1 * mult2 == product
end

def valid_addition?( char_add1, char_add2, char_sum, char_values )
  add1 = chars_to_number( char_add1, char_values )
  add2 = chars_to_number( char_add2, char_values )
  sum = chars_to_number( char_sum, char_values )
  add1 + add2 == sum
end

def each_combination( char_set, value_set, char_values={}, &block )
  char_set = char_set.clone
  value_set = value_set.clone
  char_values = char_values.clone
  
  char = char_set.pop
  if char_set.size > 0
    value_set.each do |value|
      char_values[char] = value
      value_set_without_current_value = value_set.clone
      value_set_without_current_value.delete( value )
      each_combination( char_set, value_set_without_current_value, char_values, &block )
    end
  else
    value_set.each do |value|
      char_values[char] = value
      block.call( char_values.clone )
    end
  end
end

mult1 =   "cei"
mult2 =    "da"
add1 =    "gch"
add2 =   "dfb "
answer = "adfh"

char_set = (mult1 + mult2 + add1 + add2 + answer).chars.to_a.uniq.sort
char_set.delete(" ") # space is hard-coded to 0
value_set = (0..9).to_a

each_combination char_set, value_set, " " => 0 do |char_values|
 if valid_multiplication?( mult1, mult2, answer, char_values ) && valid_addition?( add1, add2, answer, char_values ) 
    puts char_values
  end
end

You can download the final code for this solver. The library code is split from the runner for this specific problem and it also includes a test suite.

As it turns out the puzzle did have a solution. Select the space below to reveal the solution.

A = 4
B = 7
C = 1
D = 3
E = 2
F = 8
G = 5
H = 6
I = 9

I would be interested in hearing about any other approaches to this type of puzzle.

Linksys WRT54G2 Connection Problems Solved by DD-WRT

2009-07-11T00:00:00-05:00

I was having issues with random loss of internet connection with my WRT54G2. Searching the internet found other people with similar problems but no definate solutions. I was somewhat resigned to buying a new router, but I decided as a last ditch effort to install DD-WRT. The instuctions on their wiki worked fine. The only hitch was turning off the Windows firewall was necessary for the tftp client to work. Once DD-WRT was installed my problems vanished, and I have some cool extra features.

Note: According to the instructions I followed this process is irreversible and can brick your router. So do this at your own risk.

Selenium Broken by Firefox Upgrade on Ubuntu

2009-06-22T00:00:00-05:00

I use Webrat and Selenium to automate browser testing. There can be an problem with Firefox 3 on Ubuntu that requires configuring Selenium to point the Firefox binary directly.

 
Webrat.configure do |config|
 config.mode = :selenium
 config.selenium_browser_key = '*firefox3 /usr/lib/firefox-3.0.10/firefox'
end

The problem is when Firefox is upgraded the path to the binary will change. That causes an error at the beginning of the test run.

16:10:07.104 WARN - POST /selenium-server/driver/ HTTP/1.1
java.lang.RuntimeException: java.lang.NullPointerException
   at org.openqa.selenium.server.browserlaunchers.BrowserLauncherFactory.createBrowserLauncher(BrowserLauncherFactory.java:145)
   at org.openqa.selenium.server.browserlaunchers.BrowserLauncherFactory.getBrowserLauncher(BrowserLauncherFactory.java:85)
   at org.openqa.selenium.server.BrowserSessionFactory.createNewRemoteSession(BrowserSessionFactory.java:318)
   at org.openqa.selenium.server.BrowserSessionFactory.getNewBrowserSession(BrowserSessionFactory.java:118)
   at org.openqa.selenium.server.BrowserSessionFactory.getNewBrowserSession(BrowserSessionFactory.java:81)
   at org.openqa.selenium.server.SeleniumDriverResourceHandler.getNewBrowserSession(SeleniumDriverResourceHandler.java:658)
   at org.openqa.selenium.server.SeleniumDriverResourceHandler.doCommand(SeleniumDriverResourceHandler.java:392)
   at org.openqa.selenium.server.SeleniumDriverResourceHandler.handleCommandRequest(SeleniumDriverResourceHandler.java:368)
   at org.openqa.selenium.server.SeleniumDriverResourceHandler.handle(SeleniumDriverResourceHandler.java:129)
   at org.mortbay.http.HttpContext.handle(HttpContext.java:1530)
   at org.mortbay.http.HttpContext.handle(HttpContext.java:1482)
   at org.mortbay.http.HttpServer.service(HttpServer.java:909)
   at org.mortbay.http.HttpConnection.service(HttpConnection.java:816)
   at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:982)
   at org.mortbay.http.HttpConnection.handle(HttpConnection.java:833)
   at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
   at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
   at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
Caused by: java.lang.NullPointerException
   at org.openqa.selenium.server.browserlaunchers.FirefoxChromeLauncher.<init>(FirefoxChromeLauncher.java:62)
   at org.openqa.selenium.server.browserlaunchers.Firefox3Launcher.<init>(Firefox3Launcher.java:30)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at org.openqa.selenium.server.browserlaunchers.BrowserLauncherFactory.createBrowserLauncher(BrowserLauncherFactory.java:137)
   ... 17 more

Then each test will fail with a Selenium::CommandError: Selenium::CommandError. The fix is simple, just update the path to the Firefox binary. Hopefully this will save someone some time.

Ruby 1.9 Development with Windows, VirtualBox, and Ubuntu

2009-06-20T00:00:00-05:00

Ruby is more at home on *nix than on Windows. This tutorial covers setting up a Windows machine as a VM host, installing an Ubuntu VM, building Ruby 1.9, and setting up file sharing between Windows and the VM.

First download and install VirtualBox. You should create a VM and mount the Ubuntu 9.04 server ISO in the CD drive. I used the 64-bit version. I experienced numerous random problems with the desktop edition, but the server edition worked well.

Before booting the VM for the first time we should set up the network the VM will use. VirtualBox has several different network options depending on what level of access is needed to and from the VM. In this case, I want complete inbound and outbound access to the VM. The network setup we will use is very simple, but it may not be secure outside of a trusted LAN. In the settings for the VM set the network adapter to bridged.

Boot the VM and install the OS. The network configuration will probably grab a DHCP address automatically, but we really want to assign a static IP so we can easily connect via SSH. Either cancel the network auto-detection or if you are not quick enough choose to go back when it asks you for a host name. Manually configure the network to use a static IP address (you might also want to add an entry to your hosts file on the Windows machine so you don’t have to remember the IP address). When it offers a list of servers to install choose SSH.

After the install completes we should be able to use SSH into our VM.

Before we do anything else we should setup version control for our VM’s /etc directory. This simple step can save much frustration down the road if we ever mess this VM up.

sudo aptitude install etckeeper
sudo etckeeper init
sudo etckeeper commit "initial import"

We could just install Ruby with aptitude and be done, but historically there have been issues especially with RubyGems so I prefer to compile from source. So we need to install the build tools and libraries Ruby will need.

sudo aptitude install build-essential libssl-dev libreadline5-dev zlib1g-dev

Next download, compile, and install the latest version of Ruby 1.9 (1.91.-p129 at the time this was written).

wget ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.1-p129.tar.gz
tar xzf ruby-1.9.1-p129.tar.gz
cd ruby-1.9.1-p129/
./configure
make
make test
sudo make install

At this point we should have Ruby operational. However, we really want to be able to access our files transparently from Windows. So let’s setup SAMBA.

sudo aptitude install samba

Now we need to configure our share. Edit /etc/samba/smb.conf.

Around the middle of the file we need to change the security option. Uncomment it and change it to share.

security = share

Then at the bottom add your share. force user and force group are needed to set the proper owner for any files created from Windows. Since I am using static IP assignments it is easy to only allow the host access to the file shares with hosts allow.

[jack]
   path = /home/jack
   public = yes
   writable = yes
   browsable = yes
   read only = no
   force user = jack
   force group = jack
   hosts allow = 127.0.0.1 192.168.2.5

At this point we have a working Ubuntu Ruby 1.9 development environment that we can access from Windows.