A Sheep Apart

Introducting Diesel - PHP Dependency Injection

2012-02-09T11:51:00.000-08:00

PHPUnit made unit testing PHP an actual pleasant experience, but there's still something missing when it comes to generically injecting stubs and mock behavior into your classes when your classes extend beyond simple relationships.

Either you end up with constructors that take endless lists of parameters or with shifty setter methods that can leave your classes-under-test in undesirable states of non-initialization. I.e $class = new Klass(); $class->setSomeReference($mockedObject);

Diesel was born out of the need to avoid both of these situations in a reusable and easily understandable manner. Diesel is pure PHP and does not rely on attributes or XML files. It's not quite a cake pattern -- PHP does not provide mix-in capabilities -- but does provide a similar concept of defining a default implementation for your production environment use cases and respectively (optionally) granularly stubbing out all of your test use cases.

The Diesel system itself is one small PHP class. It works by relying on some static cooperation from each dependent class to implement a method which will register all of its production dependencies. For non-production use cases (i.e. Testing), it relies on each test to configure a non-static Diesel instance with each dependency for the given class under test -- most commonly each dependency will be stubbed using PHPUnit::getMock().

Dependencies for a class may be registered statically or locally, where a local registration is local to the instance upon which it was registered (i.e. no other Diesels will be affected by it). Both registration methods have the same signature, register($owner, $class, $instantiate). Consumers of Diesel can produce their dependent objects by using its non-static factory method which roughly resembles, create($owner, $class)-- further specified later.

Ok, so what does all of this mean? Let's move into a real world example. One of the utilities built into our build and release tools (Bart) project is a "stop the line" git pre-receive hook. The hook simply queries our development Jenkins server for the status of the latest build. If that build passed, then the commit is permitted, otherwise only commits whose message contains "{buildfix}" may proceed. This is explained in more detail on the project's github home page. The core class behind this feature relies on two other classes: a Jenkins class and a Git class.

In order to test our Stop-the-Line class, we need to stub out method calls to the Jenkins and Git classes. This is a perfect situation for Diesel to inject stub classes. So let's see how it works below. For the eager, you may find the entire test class at Stop The Line Test.php.

First, we must configure Stop_The_Line to work with Diesel. That means defining the registration method and accepting a Diesel instance to its constructor. Wait! Didn't I say earlier that taking injection classes as constructor parameters was bad? Well, I've concluded that it's only bad to the extent that they produce unmanageable lists of params. In Diesel's case, all of your injection is controlled by only one parameter. Not a bad compromise. So, Ok, back to the code.

class Stop_The_Line {
  // The constructor param at the end
  public function __construct($git_dir, $conf, Diesel $di = null) {
    // Use the default static dependencies?
    $this->di = $di ?: new Diesel();

    // Use Diesel to produce an instance of a Jenkins Job
    // Notice how Jenkins params are passed in optional 3rd param here
    $this->job = $di->create($this, 'Jenkins_Job', array(
      'host' => $conf['host'],
      'job_name' => $conf['job_name'],
      'w' => $w,
    ));
  }

  // This method will be automatically called by Diesel IF and ONLY IF
  // ...there is no local or static registration for Stop_The_Line
  public static function dieselify($me)
  {
    Diesel::register_global($me, 'Git', function($params) {
      return new Git($params['git_dir']);
    });

    Diesel::register_global($me, 'Jenkins_Job', function($params) {
      return new Jenkins_Job($params['host'], $params['job_name'], $params['w']);
    });
  }
}

Now, our production code can use Stop_The_Line with its default dependencies simply by omitting the last parameter to the constructor. Test code can inject instances of Jenkins_Job and Git by passing in a so contrived Diesel instance as the last parameter to the constructor.


class Stop_The_Line_Test extends Bart_Base_Test_Case {
  public function testStopTheLine() {
    $job_name = 'the build';
    $conf = array('host' => '...', 'job_name' => $job_name);

    // This is the Diesel ONLY for this test i.e. NO other tests
    // ...will be affected by the dependencies it defines
    $di = new Diesel();

    $this->configureJenkinsJob($di, $job_name, $conf);
    $this->configureGit($di);

    // Our contrived Diesel will be used when STL produces the Jenkins Job
    // ...and it's Git instance
    $stl = new Stop_The_Line('.git', $conf, $di);

    // We expect the line to stop because BOTH checks fail:
    // 1. Jenkins build failed,
    // 2. Commit message did not contain {buildfix}
    $this->assert_throws('Exception', 'Jenkins not healthy', function() use($stl) {
      $stl->verify('hash');
    });

  }  

  private function configureJenkinsJob(Diesel $di, $job_name, $conf) {
    // Use PHPUnit to create a stub job
    $mock_job = $this->getMock('Jenkins_Job', array(), array(), '', false);
  
    // And set it up to say the last build failed
    $mock_job->expects($this->once())
      ->method('is_healthy')
      ->will($this->returnValue(false));

    // Now register this stub job for ONLY this Diesel instance
    $phpu = $this;
    $di->register_local('Git_Hook_Stop_The_Line', 'Jenkins_Job',
      function($params) use($phpu, $conf, $job_name, $mock_job) {
        $phpu->assertEquals($job_name, $params['job_name'],
            'Jenkins job name did not match');

        $phpu->assertEquals($conf['host'], $params['host'],
            'Expected host to match conf');

        return $mock_job;
    });
  }

  private function configureGit(Diesel $di) {
    $mock_git = $this->getMock('Git', array(), array(), '', false);

    // Stop the Line checks if commit message contains {buildfix}
    // Let's see what happens when it doesn't
    $mock_git->expects($this->once())
      ->method('get_commit_msg')
      ->with($this->equalTo('hash'))
      ->will($this->returnValue('The commit message'));

     $di->register_local('Git_Hook_Stop_The_Line', 'Git',
      function($params) use($mock_git) {
       return $mock_git;
     });
  }
}

So as you can see, Diesel lets you granularly control the injection of multiple dependencies per system under test per test. Moreover, it does this in a pure PHP fashion giving programmatic power that just isn't offered (easily or transparently) by XML. In contrast to a annotation based system that injects dependencies via object reflection, Diesel allows you naturally call object constructors and allows those constructors to completely configure themselves in a straightforward manner. A reflection based system requires you to have empty constructors and hides actual implementations from the developer, which can lead to misunderstandings and bugs.

More details can actually be found at the link to the test I provided above. My code above was extracted thence and then tweaked to make it a little simpler (i.e. maybe I made some mistakes). Also, you can see examples how Diesel can be used within a chain of inheritance and for multiple injected classes. In fact, I encourage you to do so as that is where the full utility of Diesel really shines.

Encode AVI for iPad (for Free)

2012-02-02T09:56:00.000-08:00

I wanted to convert an old movie of mine to m4v so that I could watch it on my iPad. I was unable to import it into my iTunes since the format didn't match, so I had to convert it to a format that makes iTunes happy -- namely mpeg4.

Since I didn't want to pay for Handbrake (and they don't have a default iPad format), I decided to just do it myself using ffmpeg.

I am on a mac, so I needed to install ffmpeg. The easiest way to do that was to use homebrew. My first attempt to install failed because home brew had some issues. As it ended up, I had to uninstall home brew and uninstall mac ports (don't forget sudo rm /usr/local/bin/{brew,port})! I went through some drama with my Ruby installation (1.8.7), since home brew wasn't loading, which involved using rvm to install 1.9.3, then 1.9.2, and then back to my system install, which suddenly started working again with the brew install script.

Next: brew install ffmpeg. This got me *all* the required dependencies for ffmpeg in one go! Awesome.

Next, the encoding line:

ffmpeg -i myMovieFile.avi -acodec libfaac -ac 2 -ab 160k -s 1024x768 -vcodec libx264 -vpre iPod640 -b 1200k -f mp4 -threads 0 myMovieFile.ipad.aac

which I adapted from, https://develop.participatoryculture.org/index.php/ConversionMatrix

Testing Scalatra with Immutable Specs2

2012-02-01T18:09:00.000-08:00

import org.specs2.Specification
import org.specs2.mock.Mockito
import org.scalatra.test.ScalatraTests
import org.eclipse.jetty.testing.ServletTester
import org.specs2.specification.After

class MySpec extends Specification { def is = 
  "My spec must" ^
  "verify that scalatra can be tested" ! Specs().assert()

  /**
   * Create a new jetty context each time
   * This lets us mock expectations *per* specification
   */
  case class Specs() extends After with ScalatraTests with Mockito {
    // The servlet tester gives an http context to your tests
    lazy val tester = new ServletTester
    tester.start()

    val myMock = mock[MyObject]

    // Register your servlet with the context and inject the mock
    addServlet(new MyServlet(myMock), "/*")

    // don't leave the jetty context hanging around
    def after { tester.stop() }

    // Pay special attention to the "this" keyword, which will provide the
    // ...method in a scope such that "after" may be called for teardown
    def assert() = this {
      // Just some expectation on the mock object
      myMock.get(0) returns true

      get("/") {
        there was one(myMock).get(0)
      }
    }
  }
}

Puppet: Multiple Classes & Inheritance

2012-01-29T20:00:00.000-08:00

I learned about a very cool thing yesterday: Defined Types in Puppet. Previously, I was trying to use parameterized classes for code reusability, but the problem there is that you can only declare one class resource per node. So puppet would yell if you tried to use the same parameterized class more than once on the same node. That's where defined types come in, they let you essentially declare a constructor like you would for a class, but as a method signature. Then, you can declare as many as you want and give them names.

The tie in to how to use this with classes and inheritance is to run the Defined Type declaration within each of your individual classes.

Defined types are not auto-loaded in the same fashion as the rest of the modules, so it's necessary to define them in a file named "init.pp" Otherwise they won't load and you'll see lots of errors like, "err: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource my_defined_resource at .../modules/module_name/manifests/my_manifest.pp:8 on node node.f.q.d.n"

Specs2 Mockito - Invoke Injected Callback

2012-01-13T15:57:00.000-08:00

Mockito and Specs2 work great together and make testing my code a lot of fun. I ran into a problem yesterday in which I needed to get access to a callback function that was passed into an object that I was mocking. For some concreteness, I was mocking a zookeeper client and needed to invoke the callback function I passed to "zookeeper.watchNode(path, callbackFunction)".

Up until this point, I've only needed to verify arguments from my test. I haven't needed to actually do something with the argument itself. My original approach was to use a custom matcher to capture the parameter using function matchers. But that involved a var and was rather messy.

It turns out there is a MUCH better way to do this. Called "argument capture." In my case, my code looked like this:


      // The signature of the zookeeper watch function
      val listener = capture[(Option[Array[Byte]]) => Unit]

      // create system under test, which will call watchNode internally
      new SUT(zkMock)

      // Use the arg capturers to capture the params SUT passed in
      there was one(zkMock).watchNode(path, listener)

      // Now, call the callback method
      listener.value(Some("some updated value from zookeeper"))

      SUT.someValue must_== "some updated value"

Specs2: IndexOutOfBoundsException: 30

2012-01-12T12:28:00.001-08:00

I was consistently seeing a java.lang.IndexOutOfBoundsException: 30 exception today while working on some code. I had written a spec with a single use case, which was marked as pending because I couldn't test my trait until I flushed out another piece of code upon which it depended.

I'd seen the error in the past, but couldn't remember how (of it) it went away. Since I was working on a pretty much skeleton spec, I figured now would be a good time to get to the bottom of it.

Well, it was pretty simple! In connecting my use case text description to the actual test, I was using a "^" instead of an "!", which also explains the other oddity I was seeing (that I had thought related to pending tests) "No source file found at src/test/scala/...", since the "^" was expecting a String as the result of my test method being run, not a Specs result. Go DSLs!


def is =
"My trait should" ^
"do what I want" ^ SpecsStub().assert() // WRONG, ^ should be !

case class SpecsStub() {
  def assert() = {
    pending
  }
}

Does File Exist in PHP Include Path?

2011-12-02T15:29:00.001-08:00

We built a wrapper script around PHPUnit that loads and configures our application for testing so that test authors need not pepper their test files with redundant lines like "require_once dirname(...) . '/setup/file/included/in/all/tests.php'". This in itself is actually pretty cool, but that's fodder for another post =) The wrapper is a big time saver, but it comes at the price of a little consumption of internal PHPUnit implementation, most importantly how it starts itself up.

The start up invocation changed between 3.5 and 3.6 and one of those changes involved how PHPUnit autoloads classes. This introduced a problem because we needed to be able to support both versions of PHPUnit while the upgrade was underway. Fortunately it turned out that a determining factor between 3.5 and 3.6 was the presence of the "Autoload.php" file in the root of the PHPUnit extension module. Unfortunately, it turned out to be nontrivial to check for the existence of the file using the standard file_exists() method. The problem is that file_exists() doesn't check PHP's include path for relative file paths (unlike the logic of require_once()).

After a little research, we were able to solve the issue by a little bit of fun with file_get_contents, which takes an additional parameter to let it know that you'd like to check the include path for the presence of the file. Additionally, in the interest of performance, it lets you load only the first byte of the file, rather the entire thing.

// Support for newer version of PHPUnit during rollout to all VMs                                                                                         
// This will load the first byte of the file from the include path                                                                                         
// ...if it exists                                                                                                                                        
if (@file_get_contents('PHPUnit/Autoload.php', true, null, 0, 1))                                                                                          
{                                                                                                                                                          
    // File exists, we're on 3.6...
    require 'PHPUnit/Autoload.php';                                                                                                                        
    // ... other stuff
} 
else
{
    // Do stuff for 3.5
}

Check out the docs on file_get_contents for a description of the params.

Notes From Implementing Lean Software Development - From Concept to Cash

2011-11-17T10:53:00.001-08:00

I read the Poppendieck's book on implementing lean software development almost a year ago and I continue to go back to it for advice and insight. I finally brought myself to put together a brief summary on the first half of the book and decided to share it with my blog readers. One of the great things about the book is the set of industry examples they provide to exemplify to the points that they are making. This is especially useful to me, since I often have no debating ammunition aside from my personal beliefs. To make it easy to look up those examples, I'm including the page numbers along with the ideas below.

The book can be purchased from Amazon.

pg 24 Early specification does NOT reduce waste, it encourages it
pg 28 "Do It Right the First Time"

This means TEST first to keep bugs OUT. It does NOT mean think of all possible future needs of the feature.

pg 32 Forecast predictions, not fact. Avoid "analysis paralysis."

Build processes that allow quick feedback and responses, rather than building for an uncertain feature

pg 33 "Plans are useless, but planning is indispensable" - Eisenhower
pg 34 Achieving high value, low stress feature delivery is impossible WITHOUT superb quality (in the form of tests)
pg 38 Sub-optimizing is BAD

E.g. Optimizing the writing of ONLY the feature code, but NOT the test code
--result--> More complex code, higher potential for introducing new bugs, longer to write new code
E.g. Optimizing ONLY your part of the process. The overall process still drags along and you get frustrated.
--solution--> Team ownership!

pg 74 7 Wastes

Relearning by failing to engage current knowledge
--solution?--> Information needs to be accessible. People should not be siloed and should talk often.

pg 101 Increase estimation reliability by decreasing variability (i.e. estimate and commit to smaller projects)
pg 105 FASTER delivery --> Reduce the number of things in WIP (queue theory)
pg 124 Exec thinking is so ingrained that Lean concepts are invisible
pg 126 Group is NOT a team until everyone is COMMITTED

E.g. Sports - track versus rowing

pg 150 Story by Rally does a great job highlighting the DRAWBACKS of Technical DEBT incurred by NOT slowing down to address untested code
pg 151 You must *commit* to action items coming out of Retrospectives

Nothing is accomplished if you only discuss problems.

pg 153 More important than processes is LEARNING (understanding), SHARING, and SOLVING PROBLEMS

Experience over documentation.
Refined documentation far outweighs garrulous documentation

HTML5 Drag and Drop - Chrome Not Working?

2011-11-13T10:59:00.001-08:00

The W3C standard defines seven event types for drag and drop, http://dev.w3.org/html5/spec/dnd.html#dndevents. Pretty cool to have so many options, but the nieve reader may be surprised to learn how they work together. Specifically, for people binding to the "drop" event.

The default behavior of the dragover event is described as such, "Reset the current drag operation to 'none'". So what does this mean for developers binding to the drop event? It means, that unless you prevent the default behavior of dragover, your drop event will not fire. This is not a fun one to learn on your own.

Could not evaluate: Could not retrieve information from source(s)

2011-11-11T12:39:00.001-08:00

Oops! I was evaluating a template, NOT a file.

file {
"/destination/file/path":
source => template("path/to/template");
}

The key is that "source" should be "content"!

file {
"/destination/file/path":
content => template("path/to/template");
}

So, what was happening behind the scenes is that puppet was trying to source the literal value of the evaluated template file. Could be useful?

Reading /etc/rc.status

2011-11-09T21:09:00.000-08:00

What Cool Bash Stuff Did I Learn?

What does RC Stand For? ==> Resource Control
How do I tell vim my file type is Bash?

I can set the following anywhere in the file,

# vim: set filetype=sh
# vim: syntax=sh

How can I make my if()s look prettier? ==> Use {}

[ -e $pathToFile ] || {
   echo >&2 "File $file does not exit"
   exit 1
}

How do I prompt a command to ask me for input?

==> cat file-to-prepend - file-to-append > output-file

Hello Specs2 JUnit, Farewell Henkelmann TestListener

2011-10-31T15:39:00.000-07:00

Out of the box, SBT 10.1 doesn't give you test results in JUnit output. This is a problem for anyone using Jenkins to process test results. Fortunately, last November, Christoph Henkelmann released a plugin to SBT that plugs in to the test execution life cycle of SBT, http://henkelmann.eu/junit_xml_listener. This was great, but had a few bugs with inconsistent results and JUnit data ending up in the wrong file or XML being badly formatted. These bugs occurred inconsistently and infrequently enough to be only a minor annoyance.

Over this past summer, specs2 added a JUnit output option for its test results. At the time, the documentation didn't fully explain the necessary integration points. So I'll explain what I did to get it working. Most of this information is now available at, http://etorreborre.github.com/specs2/guide/org.specs2.guide.Runners.html if you search for "junit".

// Put this in your SBT build file to output to both console
// ...and junit files console is helpful so that you can see the
// ...output in jenkins build history
testOptions in Test += Tests.Argument("junitxml", "console")

If you're running on the command line, you can provide SBT system arguments straight up, like

bash $> sbt junitxml

Finally, on the SBT command line, it won't work with the SBT "test" command, but it will work with the SBT "test-only" command.

sbt-command-line>test-only com.company.class.specs -- junitxml

The other cool learning that I experienced at this time, was with using SBT collections. We have several scala projects and we need to share common SBT properties between them. To accomplish this, we made our own SBT plugin and import all of its settings at the top of each project. One of the settings is the testListener SBT setting, which attaches the junitXmlListener to all our unit test output. Since, I wanted to test out the junitxml flag in only one project before fully committing to it, I couldn't outright remove the test listener for all the projects. So in order to remove only in one project, I used the SBT filter syntax: ~=

testListeners ~= { (listeners: Seq[TestReportListener]) =>
listeners filterNot ( _.isInstanceOf[JUnitXmlTestsListener] )
}

Gerrit Code Review - Unpack error Missing unknown

2011-10-26T19:19:00.002-07:00

We use the Gerrit code review tool at my company; currently version 2.1.8. Gerrit gives us a lot of tools and is an extremely useful tool for code quality and knowledge sharing (i.e. code reviewing). However, Gerrit can often require some TLC. Today was one of those days.

While I was doing some maintenance on our git host today, I noticed that git suggested I prune one of our repos. So, I went ahead and pruned. About half an hour later, one of our developers reported that he couldn't create a new patch set in Gerrit -- neither update or create were working. Pretty soon, I was getting several reports of this from the team. They were all getting the same error:

error: unpack failed: error Missing unknown 3061766be9c324fa47fb4832399b34db5a186276
fatal: Unpack error, check server log
To ssh://git.dev:29418/webapp
! [remote rejected] HEAD -> refs/for/master/master (n/a (unpacker error))

The key part of this message is the git object that's missing, 3061766. The git prune must have identified that the object was dangling, no longer necessary, and removed it. However, for some reason, JGit inside of Gerrit still felt the object served some purpose important enough to throw an exception. I tried to find the object in the remote repo and locally, but to no avail, it was history.

After trying pretty much everything I could find on the internet (http://code.google.com/p/gerrit/issues/detail?id=585, http://groups.google.com/group/repo-discuss/browse_thread/thread/cf7095d3dc364c7e/f2c11756a5a0396f?) and replacing the jgit jar in my tmp .gerritcodereview directory, I finally found a solution.

The solution was to restore a backup of our git repository to a temp directory and use git cat-file to first verify the object 3061766 was valid. Then, I used git verify-pack to find the pack file that contained the commit. Then I copied the pack and its index file (http://progit.org/book/ch9-4.html) into our real repo. Shazzam! Gerrit started accepting change sets again!

In more detail:

tar -xf backup.tar ./path-to-git-dir
cd path-to-git-dir
git cat-file -t 3061766be9c324fa47fb4832399b34db5a186276
cd objects/pack
ls | xargs git verify-pack -v | grep 3061766be9c324fa47fb4832399b34db5a186276
# back track until you find the pack that contains the commit
cd /git/path-to-git-dir/objects/pack
cp /untarred-repo/objects/pack/...{idx,pack} .

Vim Registers Current File Name

2011-10-26T14:14:00.000-07:00

So you guys may remember how I was all obsessed with vim registers a few months ago. Well, today I was typing away at some things and while in insert mode, I wanted to splat the name of the file I was currently editing. So I took a risk:

Ctrl+R % # i.e. Access register %, which holds the name of the current file.

result: the name of the file pasted in!

Pretty cool.

Bash Function Return Values and Exit

2011-09-27T23:33:00.006-07:00

For those lucky souls out there programming in bash, you may be making use of the bash subshell trick that allows you to "return" a value from a function.

function echoTwice() {
  echo "$1"
  echo "$1"
}

declare twice=$(echoTwice "woot")
echo "Twice is: $twice"

Will produce the output,

Twice is: woot woot

The tricky gotcha in this situation involves exiting on error conditions. For example, consider the following enhancement to echoTwice,

function echoTwice() {
  if [ -z "$1" ]; then
    echo "Must provide an argument" >&2
    exit 1
  fi

  echo "$1"
  echo "$1"
}

declare twice=$(echoTwice)
echo "Twice is: $twice"

You would expect to see the error message and nothing else. However, instead you see

Must provide an argument
Twice is:

Why does this happen? It happens because the assignment to $twice happens via a subshell. A subshell is spawned to evaluate the result of echoTwice. When that subshell exits early, bash treats it no differently than had the whole function proceded. Subsequently, rather than quitting your code, processing continues when the subshell completes. So, how do you accomodate this in your code without resorting to ugly globals?

The solution that I have found is to test if $twice was assigned.

if [ -z "$twice" ]; then
  # Error message emitted by echoTwice prints to STD_ERR and
  # ...will not be consumed by the sub-shell
  exit 1
fi

The reason I can't just check the exit status is because declare twice= is an actual execution and will typically have a valid exit status.

EDIT (Nov 2nd, 11): It turns out that you can check the return status! The issue is accurately described that declare twice= has its own exit status, but the exit status arises from the use of the bash built-in "declare" and NOT from the assignment to the variable (See the "declare" section of http://www.gnu.org/software/bash/manual/bashref.html#Bash-Builtins). So, the solution becomes:

declare twice=
twice=$(echoTwice)
if [ $? -ne 0 ]; then
  # Error message emitted by echoTwice prints to STD_ERR and
  # ...will not be consumed by the sub-shell
  exit 1
fi
echo "Twice is: $twice"

Don't Lose Your Phone

2011-09-16T09:53:00.001-07:00

I get lots of credit card offers in the mail and in my email. In my life, I'm probably indirectly responsible for the life of an entire tree just to print all the offers I've received (not to account for the energy expended delivering them, ink for printing, etc.). Today I finally resolved to bring the offers to an end. There is a phone number listed at the bottom of most offers, 1-888-5OPTOUT, which you can call to remove yourself from the list of people agencies will contact with the great deals. According to the FTC, this number is provided by the National Credit Bureau (http://www.ftc.gov/privacy/protect.shtm).

I called the number and was very shocked at the information it exposed about me via the simple fact of the phone number from which I was calling. Essentially, by identifying my phone number the automated service provided me, with no security challenge at all, my home address and full name. Transcript below:

NCB: Are you calling from your home phone?
Me: Yes
NCB: Please verify your address. Is it 123 Street Name, Town, State?
Me: Yes
NCM: Is your last name "Plumber"?
Me: Yes
NCM: Is your first name "Joe"?
Me: Yes
NCM: Please enter your SSN

After this point, I actually had to provide the information to verify it was me.

So, I suppose that knowing someone's phone number may be enough to find their information online already anyway (white pages?), but I was utterly shocked that a service sponsored by the federal government would so easily give your private information away. My conclusion, be extra careful not to lose your phone and don't give your phone number out too willingly.

Git Unpack Error Over HTTP Fetch

2011-06-29T09:49:00.003-07:00

Had a problem today with a server fetching the latest changes from our upstream repository over HTTP.


git fetch origin
error: packfile .git/objects/pack/pack-385ce85680e3c3ff129907559101b9a4544a9da0.pack does not match index
error: packfile .git/objects/pack/pack-385ce85680e3c3ff129907559101b9a4544a9da0.pack cannot be accessed

I didn't have much luck finding any advice on google, so I thought I'd post my own solution, which was extremely simple!


rm  .git/objects/pack/pack-385ce85680e3c3ff129907559101b9a4544a9da0.pack
git gc
git fetch origin

Getting pack 385ce85680e3c3ff129907559101b9a4544a9da0
which contains dcbe1aa3d3e564aea30acc55a7df105bfdc586a2
# success

Essentially, in the version of git we run (1.6.3.1), git fetch over HTTP will blindly pull down every pack regardless of whether or not it is applicable to the downstream checkout. What was happening was the pack somehow got corrupted and an unfetched pack was dependent on that corrupted pack, so git couldn't resolve the mismatch. Removing the corrupted pack let git re-fetch it and the dependent pack that followed.

Don't Miss the CalTrain

2011-05-22T22:46:00.006-07:00

I have a bad track record of getting to the CalTrain stop on time. It's a very depressed experience to see the train receding into the distance as I race around the curve to the train stop on my bike. It's even more depressing to instead see the train loading, sprint down the stairs, under the track, and back up the other side (all the while carrying my bike rather than riding down the ramp), and get close enough to touch the train as the conductor closes the doors before I can get in and then watch the train recede into the distance as I stand there panting.

It's clear that the CalTrain expects us to always be on time and shows no quarter. Why? Well because the CalTrain is always on time. Isn't it? Actually, no, it's not. For one reason or another, the CalTrain can be as much as an hour late, full, or even canceled.

As passengers, we need to be prepared and in the know. So far, the best resources I've found for being on time and keeping tabs on CalTrain delays are a clock and twitter.com.

The programmer in me hates manual labor, so I decided to automate a little system to help me out. It comes in two parts,

A script to (a) check twitter for tweets @caltrain and (b) make my computer speak and pop up a dialog box with the tweets reminding me to pack up and go catch the train
A schedule to run the script at the same time every day, just in time to catch the train

The script itself comes in two parts, AppleScript and Ruby. The AppleScript to hook into my Finder app for the dialog box. The Ruby to parse Twitter's API so that I don't have to do it in AppleScript (shudder).

I've posted the scripts up on github.

The following pages helped me out:

Ruby JSON - http://flori.github.com/json/

Twitter Search - http://search.twitter.com/api/

AppleScript - http://www.tee-boy.com/forums/viewtopic.php?f=8&t=76

MySQL.rc Configuring Your MySQL Client

2011-04-06T10:19:00.004-07:00

I recently wanted to customize the SQL command line prompt so that I'd know the database server to which I was connected. Previously, I'd been using the --prompt option when connecting (e.g. alias dev-db='mysql -h dev-db --prompt "dev-db-mysql>"';), but that gets maintenance heavy as I add more and more databases to my shortcuts.

I resolved to find a more generic way. I tried all sorts of google searches, all of which came up unhelpful. With no choice left, I headed to the MySQL docs. After some clicking around, I came upon this page, http://dev.mysql.com/doc/refman/5.0/en/mysql-commands.html.

This is great news! Apparently, I have several options.

/etc/my.cnf - Globally change the mysql client for all users of a given machine
~/.my.cnf - Change for just me on a given machine
export $MYSQL_PS1 - Change just my prompt without requiring any new files

Since all of the machines I use are already controlled with puppet, it was a simple change for me to add a definition for $MYSQL_PS1 in my profile.

Element in Array with Bash?

2011-03-25T21:44:00.004-07:00

I recently found myself editing a rather long conditional evaluation in Bash that was essentially comparing a variable, $color, against several valid matches. If $color didn't match any of them, the condition evaluated to true. I was a little disappointed with the prospect of maintaining it and the stretch of ORs and ==s across my screen. In a "real" scripting language, I would just use an array and check if $color was an element. Then I realized that Bash lets me use an array, just a little differently. With a combination of echo and grep, I got the job done.


declare -a valid_colors=( 'green' 'red' 'blue' )

echo "${valid_colors[@]}" | grep -qv "$color"
if [ $? -eq 0 ]; then
  # Do your stuff
fi

Oh Shit! Git Amend Ate My Changes!

2011-03-08T18:19:00.005-08:00

Today a developer did a git commit --amend and accidentally overwrote his entire HEAD. (I'm not sure how, but it did happen.) I knew that since git saves (what seems like) everything, if we could find the commit hash for his previous HEAD that we'd be able to cherry-pick it. We scrolled up through his terminal but couldn't find any reference to it =/ So that meant we needed to try finding it the hard way: .git/

So then I started looking through the git-dir and after a little digging, we found therein a log file for each branch (.git/logs/refs/heads/*), the contents of which lists the last several commit hashes made to the branch respectively. Alright! Each file contains a somewhat chronological list of the hashes generated by each change to the tree (rebase, pull, commit, cherry-pick, etc.). We were able to use git show on the hashes near the end of our branch's log file and recover the change!

Conclusion, steps to recover:

git branch -- # e.g. my-feature
tail .git/logs/refs/heads/my-feature
git show $hash # using the hashes from the log file until you find the one you want

UPDATE:

So it turns out the easiest way to do this is to use git reflog, which will list all the changes to your HEAD for the past X changes. Way easier!

Bash PS1 Colors and More Space

2011-03-07T11:48:00.009-08:00

function prompt_color() {
local gr='\033[0;32m' # light green
local red='\033[0;31m' # light red
local bold_red='\033[1;31m';

if [ $1 -ne 0 ]; then
  echo -e "${bold_red}**${1}**${red}"
  return
fi

echo -e "$gr"
}

function prompt() {
local nc='\033[0m' # no color

# User@{2 part hostname} date time directory\nCommand Number $ - me, # - root
# __git_ps1 - show current git branch; http://blog.jasonmeridth.com/2010/05/22/git-ps1.html
local $ps1='[\u@$SHORTNAME \D{%Y-%m-%d} \t \W$(__git_ps1 " (%s)" 2>/dev/null)]'

export PS1=$(echo '$(prompt_color $?)' "'$ps1'" "$nc" ' \n[\!]\$ ');
}

I also employ `echo` above in order to concatenate the color variables with the PS1 string, which should be evaluated ONLY per PS1, not at the time of definition. I originally was using double quotes to surround the entire thing, but then realized that `git_ps1` was getting executed ONLY when PS1 was being defined.

Add Color to PHP Echo in CLI

2011-02-04T23:46:00.006-08:00

I adapted this PHP script from If Not True Then False for CentOS escape codes, which use a semi-colon and not a comma between bold and color codes for foreground color escaping.


/**
 * Color escapes for bash output
 */
class Escape_Colors
{
 private static $foreground = array(
  'black' => '0;30',
  'dark_gray' => '1;30',
  'red' => '0;31',
  'bold_red' => '1;31',
  'green' => '0;32',
  'bold_green' => '1;32',
  'brown' => '0;33',
  'yellow' => '1;33',
  'blue' => '0;34',
  'bold_blue' => '1;34',
  'purple' => '0;35',
  'bold_purple' => '1;35',
  'cyan' => '0;36',
  'bold_cyan' => '1;36',
  'white' => '1;37',
  'bold_gray' => '0;37',
 );

 private static $background = array(
  'black' => '40',
  'red' => '41',
  'magenta' => '45',
  'yellow' => '43',
  'green' => '42',
  'blue' => '44',
  'cyan' => '46',
  'light_gray' => '47',
 );

 /**
  * Make string appear in color
  */
 public static function fg_color($color, $string)
 {
  if (!isset(self::$foreground[$color]))
  {
   throw new Exception('Foreground color is not defined');
  }

  return "\033[" . self::$foreground[$color] . "m" . $string . "\033[0m";
 }

 /**
  * Make string appear with background color
  */
 public static function bg_color($color, $string)
 {
  if (!isset(self::$background[$color]))
  {
   throw new Exception('Background color is not defined');
  }

  return "\033[" . self::$background[$color] . 'm' . $string . "\033[0m";
 }

 /**
  * See what they all look like
  */
 public static function all_fg()
 {
  foreach (self::$foreground as $color => $code)
  {
   echo "$color - " . self::fg_color($color, 'Hello, world!') . PHP_EOL;
  }
 }

 /**
  * See what they all look like
  */
 public static function all_bg()
 {
  foreach (self::$background as $color => $code)
  {
   echo "$color - " . self::bg_color($color, 'Hello, world!') . PHP_EOL;
  }
 }
}

Passenger Root Directive Not Found

2011-01-31T17:24:00.002-08:00

Error:

nginx: [emerg]: unknown directive "passenger_root" in /box/etc/nginx/nginx.conf:18

Solution:

http://markmail.org/message/ektzyf3hm3oyz5un#query:related%3Aektzyf3hm3oyz5un+page:1+mid:i5hukr2cemguxvtj+state:results

Two versions of nginx were installed on my openSUSE box. I believe this was because I had already installed it before running the passenger installation script. I may try running it again and telling it to install the new version into the proper path.

VIM Command Line History Window

2010-12-15T21:43:00.004-08:00

Ever been recording a macro when all of a sudden an unexpected window pops up out of nowhere in the bottom of your window, the contents of which appear to resemble the last few commands you've typed?

Well, it's definitely happened to me. This isn't exactly the sort of thing for which you can search online -- what sort of query can succinctly describe that? Well, today as part of another quest, I finally discovered what's going on!

VIM supports command line history (as can be most easily observed by pressing up or down in command line mode, or just typing ":history"). What I didn't know was that when in Command Mode, typing "q:" will open your command history in a sub-window. You can then navigate this window, make changes (in Normal mode), and re-execute the original commands or (and this is the cool part) modified commands simply by pressing enter! (from Command Mode).

To exit out of the sub-window, just CTRL+C (twice) as you would to cancel out of Command Line mode.

Ignore Missing Files with tar

2010-11-15T12:38:00.013-08:00

I've been experiencing an annoying problem with tar in which it is not creating archives if at least one file in the tar list does not exist. This behavior is totally acceptable and appreciated when I am tarring on demand. However, in this case, I am running a command across several machines each of which will have a different sample of a set of files and I need this done as a cron. And to make matters more complicated the file set involves specific files in a set of directories.

# The novice command
bash$ tar cfz yesterday-logs.tgz \
log-type1/log-type1-2011-11-15.log \
log-type2/log-type2-2011-11-15.log \
log-type3/log-type3-2011-11-15.log \
log-type4/log-type4-2011-11-15.log

This command will work fine if all the directories exist. However, if at least one of the directories or files does not exist, I will experience the following error and an empty tar archive.

tar: log-type3/log-type3-2011-11-15.log: Cannot stat: No such file or directory

tar: Error exit delayed from previous errors.

I solved this issue by using the wonders bash expansion, sub-shell, and std error redirection

bash$ tar cfz yesterday-logs.tgz $( ls log-type{1,2,3,4}/*-2011-11-15.log} 2>/dev/null )

Fun. I suppose there is most likely a flag to tar that I couldn't find in my scan of the manual entry that completely obviate this complexity, but in the meantime...

Update Ruby Gems - CentOS

2010-10-17T22:52:00.004-07:00

Circular dependencies suck. It's a really great thing that the entire package management system for Ruby is written with Ruby, until you need to update Ruby using said system when said system itself needs to be updated.

I'm currently trying to update Ruby and RubyGems on CentOS via yum and life has not been fun. Step 1, rubygems wasn't present, so I installed with yum. That went fine, except it installed version 0.9.2 of gems. Hmm, Ok, no big deal, I can run gem update --system. Nope, fail:


me@host$ sudo gem update ruby
Updating installed gems...
ERROR:  While executing gem ... (Gem::RemoteSourceException)
    HTTP Response 302

Great, rubygems changed their URL since 0.9.2 and apparently whatever http tool gems uses can't follow redirects. Well, after some debugging and total failure to find any help online, I discovered that gems maintains a sources file /usr/lib/ruby/gems/1.8/gems/sources-0.0.1/lib/sources.rb that lists the domain(s) of its repositories. Well, let me just change that to the new URL


module Gem
  @sources = ["http://rubygems.org"]
  def self.sources
    @sources
  end 
end

I ran the command again. STILL getting the 302. What the heck??? Alright, after some more debugging, I discover it's trying to download a yaml description file located at http://rubygems.org/yaml. This loads fine in my browser, but a little telnet shows that I'm being redirected (obvious hint from the 302 earlier).


[me@host]$ telnet rubygems.org 80
Trying 72.4.120.124...
Connected to rubygems.org (72.4.120.124).
Escape character is '^]'.
GET /yaml HTTP/1.1
host: rubygems.org

HTTP/1.1 302 Found
Date: Mon, 18 Oct 2010 05:46:40 GMT
Server: Apache/2.2.3 (Red Hat) mod_ssl/2.2.3 OpenSSL/0.9.8e-fips-rhel5 Phusion_Passenger/2.2.15
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.15
X-UA-Compatible: IE=Edge,chrome=1
X-Runtime: 0.000895
Location: http://production.s3.rubygems.org/yaml
Content-Length: 0
Status: 302
Vary: Accept-Encoding
Content-Type: text/html

Hey, look! There's the redirect URL! A little update and I'm good to go.


module Gem
  @sources = ["http://production.s3.rubygems.org"]
  def self.sources
    @sources
  end
end

Then run,


[me@host]$sudo gem update --system
Updating RubyGems...
Attempting remote update of rubygems-update
Successfully installed rubygems-update-1.3.6
Installing ri documentation for rubygems-update-1.3.6...
Installing RDoc documentation for rubygems-update-1.3.6...
Could not find main page README
Could not find main page README
Could not find main page README
Could not find main page README
Updating version of RubyGems to 1.3.6
Installing RubyGems 1.3.6
ERROR:  Expected Ruby version >= 1.8.6, is 1.8.5
RubyGems system software updated

### Damn
### Maybe update ruby with gem? 

[me@host]$ sudo gem update ruby
Updating installed gems...
Attempting remote update of ruby
ERROR:  While executing gem ... (Gem::GemNotFoundException)
    Could not find ruby (> 0) in any repository

Nope, no go. So it looks like I am manually installing Ruby from the source since the yum repository only has Ruby as modern as version 1.8.5 dated 2006-08-25!! Isn't Ruby and Rails supposed to make life easier??

HTML5 Drag and Drop Upload Testing

2010-09-24T15:32:00.007-07:00

We're building out a testing automation framework at work and one of the most important parts of it involves tests around file uploads. Historically this has been impossible to automate due to security settings in the browser that disallow setting the file path in a file type input element(e.g. to prevent secretly uploading confidential data from client machines). However, some browsers now support configurations to allow disabling this restriction for certain websites. Test automation driver, Selenium, has taken advantage of this and allows a user to set the value of a file type input element. This works great for the simple case of an explicit form and file type input element.

Enter HTML5. Browsers can now respond to the "drop" event and furthermore the XmlHttpRequest specification (http://dev.w3.org/2006/webapi/XMLHttpRequest-2/) has been enhanced to make it possible to programmatically upload a file with javascript (i.e. xhr.send(fileBlob)). Put those two together and an ambitious web developer can now add automatic drag and drop file uploading to their website.

This is great for the look and feel of a website, but presents a challenge for the tester: How do I simulate a file drag and drop event? We were able to solve this issue by skipping the "drag" part and jumping straight to "drop." What does this mean? It means that our test will automatically call the "drop" event handler and provide a mock javascript event to the handler. The handler takes care of creating the xhr and sending it.

There is still one hurdle to overcome. The xhr requires an actual File object (http://dev.w3.org/2006/webapi/FileAPI/). Since it's not possible to create one via standalone javascript, we got around this by creating a new file type input on the page and using Selenium to set its value (a path to a valid file on disk) and then referencing that.

The following Scala snippet uses jQuery for the dirty work:


val s = selenium.getEval("fileUpload1 = window.$j('<input/>').attr({type:'file'}).appendTo('body');")

selenium.text_field("//input[@type='file']").set("path/to/file")


// Trigger the upload event
val mockEventJs = "{originalEvent : {dataTransfer : { files : fileUpload1.get(0).files } } }"

val ss = selenium.getEval("window.uploader.handle_drop(" + mockEventJs + ")")

Arel and Rails 3 - The Rage and Grace

2010-09-18T16:27:00.003-07:00

I've been spending my time recently rebuilding the front end of a logging system we have to use Rails 3. This has been a lot of fun and I've come up with some fun code (pagination and naturaldate). The new ActiveRecord::Relation class has really made my data interactions streamlined and readable -- including (and especially) performant.

Unfortunately, I've found that Arel and Rails 3's journey into SQL's GROUP BY clause was a little brief. While I appreciate ActiveSupport::OrderedHash as a combination of array and hash to effectively express the results of a group by -- and effectively any other such ordered hash data set -- I was frustrated and disappointed by the way it handles multiple properties in the GROUP BY or SELECT clause. Additionally, I was thrown off for some time before I realized the fact that :count() returns an OrderedHash and not a number when dealing with a :group type active relation. Perhaps it's my own inability to figure out how to use Rails' or Arel's (poorly (if at all) documented) methods, but I've found that if I want any information outside of a basic

SELECT COUNT(*), property_name FROM tbl GROUP BY property_name

that I am forced to do more than one query. Hmm, I thought ORM was supposed to make my life easier or at least not make my website slower?

In my specific case, I need to group by several properties and paginate over the results. On top of that, I also needed an aggregate result in my select list. Not to mention the pagination parts.

SELECT prop1, prop2, COUNT(prop2) FROM tbl GROUP BY prop1, prop2

This is how I solved it:

counts = Model.

select("prop1, prop2, COUNT(prop2) AS 'prop3').

group("prop1, prop2")

num_pages = counts.all.size

page = counts.

limit(page_size).

offset(page_num * page_size)

There are two important things to notice above as hacks/sadness

counts.all.size -- This will query the full set from the database as opposed to COUNT(*)
COUNT(prop2) AS 'prop3' -- Hmm, what? Since rails will ignore anything in the select list that isn't a property on the referenced model's table, I have to trick into loading my aggregation result into prop3. Ugly.

Part 2:

Now that all of that is behind me, I'm faced with a new problem! Rails 3 doesn't support :include with :group! Oh pain, why can't life be simple? How did I solve this? Well, more hackery.

include() works by creating an instance of the included class and then uses its :find method to load all the referenced records after the initial model's data has been loaded in what might be called lazy loading, but what seems more like 1 + 1 versus N + 1 loads. Rails is able to programmatically do this in an elegant fashion by loading a reflections property on every ActiveRecord that defines a set of all foreign key relations to the singleton of that foreign key's ActiveRecord (Model) class.

Using this knowledge, I can imitate that same behavior by collecting all the referenced foreign keys in my select class (into :ids) and do the following,

fk1_records = Model1.new.reflections[fk1_class_name.to_sym].klass.find(ids)

I can then iterate over the result set and define the relationships accordingly.

mapper.rb:143:in `default_controller_and_action': missing :action (ArgumentError)

2010-07-27T11:15:00.002-07:00

I'm trying out Rails 3 for a small web project we need done at work. After spending more time than I wanted yesterday getting everything configured correctly on my Mac (namely mysql 64 bit), I was finally ready to start generating some MVC action. I followed the steps on the http://guides.rails.info/getting_started.html Rails 3 intro page to generate a controller for my home index. This added a route to my routes.rb,

get "home/index"

I thought I'd get fancy also add a root controller and view for my website, so I coped that and slightly modified it to

root :to => "home/index"

Cool. Next, I went ahead and tried to generate a new scaffold for my first model, but uh oh, there was a problem!

/Library/Ruby/Gems/1.8/gems/actionpack-3.0.0.beta4/lib/action_dispatch/routing/mapper.rb:143:in `default_controller_and_action': missing :action (ArgumentError)

...

from /Users/.../Code/.../config/routes.rb:2

...

from /Library/Ruby/Gems/1.8/gems/railties-3.0.0.beta4/lib/rails/commands.rb:16

from script/rails:6:in `require'

from script/rails:6

My first reaction was, "!@# yet another problem with Rails 3 and my system," but after some searching for the error turned up nothing, I decided that I had probably screwed something up. Looking a little more closely at the stack trace of the error, I identified that the scaffold generation was in part based on code from my own app and not just the rails gems. Hmm, probabilities rising that I'm the culprit.

The error says that the default controller is missing an action. I checked out the gem, mapper.rb, and saw that it's parsing out some variable named "to" for a controller and action. I then opened up my routes.rb and saw that line of copied code, root :to => "home/index". I then thought back to the new rails syntax that I'd learned in the video tutorials and immediately realized my folly. One quick change,

root :to => "home#index"

and the generator is back to working. Although it can definitely be a pain dealing with open source software, it certainly is nice to be able to see the source of your problems.

Migrating from svn to git

2010-07-07T18:01:00.007-07:00

Several tutorials have already documented the necessary steps for the conversion from svn to git (http://www.gitready.com/beginner/2009/02/04/converting-from-svn.html), so I won't reiterate any of that. What made our migration tricky were two things:

SVN commits without authors
No tags, branches, or trunk paths

The second was easy to fix actually. I didn't specify any flags for standard layout or tags/branches in the git svn clone command. The second took a little bit of guess work. Before I begin, I'll point out that I used the script (after cleaning up the leading and trailing white space)

as conveniently provided in the comments to one of the tutorials. Just to make life easy, I used the following VI macro to fill out the definitions in the file,

yeA= ^[pa <^[pa@domain.com>^[j0

I noticed a blank line in the file, which was generated because of commits to the repo with no user name -- literally just the empty string. Big deal, I just removed it. When I tried to run the "git svn clone" though, it puked when it got to those revisions:

$>git svn clone svn://svn.domain/project -A authors.txt

Author: (no author) not defined in authors.txt file

I googled and found a support article, http://support.github.com/discussions/repos/2690-authors-not-defined-on-svn-import, which didn't actually solve the problem, just avoided it. It turned out the fix was actually easier than I thought. I added the following line to the authors file:

(no author) = no_author

Great List of VIM Tricks

2010-06-14T11:19:00.001-07:00

http://www.slideshare.net/ZendCon/vim-for-php-programmers-presentation

IdeaVIM in IntelliJ Community Edition

2010-03-16T15:08:00.005-07:00

So, I started using IntelliJ last month and I've really been enjoying it. As a previous heavy user of ReSharper with Visual Studio, I've grown accustomed to the convenience and efficiency of an IDE with refactoring and code inspection built in. However, I immediately began to suffer from the lack of VI keybindings.

One of the great things about IntelliJ is the huge repository of plugins. To top that off, the plugins are integrated right into the preferences manager so you can search, download, and enable them all in one easy place. My search in the plugins manager came up empty, so I continued the search online where I did find a plugin listed right on the IntelliJ website! IdeaVIM: http://plugins.intellij.net/plugin/?id=164.

Why wasn't it showing up in my plugins manager? Well, it turns out that the IdeaVIM plugin is only offered to users of the IntelliJ Ultimate edition. Great... I'm not about to pay $250 to get key bindings for one of the most prevalent text editors in the world in my IntelliJ, on top of the fact that the money doesn't go directly to the plug-in developer (Note, I was happy to pay $60 for my copy of ViEmu).

So two things happened next that got me what I needed,

I saw a "download" link on the plug-ins page
I learned that my coworker had successfully installed the plugin on his computer before they restricted it to the Ultimate edition

With that knowledge in hand, I was ready to try a new tactic:

I downloaded and unzipped the plugin and placed it in my ~/Library/Application Support/IntelliJIdea90CE/ directory.
Next, I copy-and-pasted the availables.xml configuration section for the IdeaVIM plugin off my coworker's computer into my own.
Then I restarted IntelliJ and ta-da, I now have VIM!

Where Does Outlook Keep My Email?

2010-01-20T15:45:00.004-08:00

My web searches on this topic were helpful, but incomplete. Our company uses an exchange server to house all of our corporate emails, so technically all of my email is on the exchange server. For performance, however, my email is also stored locally on my machine in my user's local settings: C:\Documents and Settings\%user%\Local Settings\Application Data\Microsoft\Outlook. (Don't forget the local settings part.) This much info I found online.

The complicated part is that my company just switched to exchange -- I'm not sure what we were using prior -- so I have more than one PST (personal storage) in there. On a hunch, I checked out my control panel's "Mail" interface.

Open Control Panel -> Mail
Click E-mail Accounts
Switch to tab "Data Files"
The path to the PST file for my current profile is right there!

That was easy.

Bit (not Byte) Manipulation in Ruby

2009-12-29T12:50:00.005-08:00

I was recently tasked with creating a rough version of the Lempel-Ziv 77 encoder/decoder engine for use in most operating systems (i.e. Windows, Linux, Mac). The application would need to read a binary file and compress it or decompress it to another binary file. The compression algorithm involved a format bit specifying compressed or literal bytes to follow and then distance and length bits of instructions for compressed data. Such an application would clearly involve a good deal of bit manipulation and consequently require a solid bit manipulation library.

The logical language of choice to me was C++ because of its proximity to the memory, inherent ease of bit manipulation, and presence on every computer since I was born. Unfortunately, I can probably barely compile a "hello, world!" application in C++ =( Next I considered Java since it's open source and present on most people's computers. However, my Java skillz have sadly dwindled since college to the point that I frustratingly discarded that project about an hour after I started. Finally, I decided upon Ruby as my language of choice -- mainly because I like coding in Ruby.

My project got off to a good start until I realized that the original research I'd done on manipulating bits in Ruby had been incomplete. Ruby inherently manages characters and bytes synonymously, but bits are another story. Based on the loose typing model of Ruby, any use of bits throughout my code was being converted to their numeric string representation behind the scenes. For example, 0xff was ending up as the string "255" when I was writing it to a file.

Finally, after much worrying, reading of documentation, online research, and irb investigation, I had an answer.

Bytes can be specified in Ruby per bit as such, 255 = 0b1111_1111 (each four bits are separated by an underscore). This was important for me since I was doing a lot of shifting and didn't want to worry about the actual numerical values in my unit testing.
Bytes can be written explicitly to files in Ruby using the << operator along with Array.pack.

File.open("foo.txt", "wb+") { |f| f << [0xff].pack("c") }

Bytes can be easily read using File.each_byte
The byte code for a given character can be accessed using: "a"[0]
Binary file manipulation involving windows must be done using the "b" flag when opening the file. Otherwise, the windows file system will treat certain bytes as termination characters and ignore the remainder of the file. I learned this the hard way because each_byte would just inexplicably stop reading in bytes from my file before the file was finished.

After I had all of this figured out, Ruby proved to a very nice environment for writing the app.

New Class Templates in C#

2009-12-29T12:16:00.002-08:00

My team finally upgraded to .NET 3.5 along with Visual Studio 2008 recently, which has been a huge source of happiness for me. However, since the upgrade, I've been getting annoyed with VS's insistence that I always include the Linq library in each of my new classes. I typically remove all of the default using directives anyway, but since I don't automatically reference the System assembly that contains the Linq definition, I was getting a pre-compile error from R# every time I added a new item to a project. In addition, I've been growing tired of having to type "public" each time I add a new class (since the default is nothing). So, I decided to take action.

A little bit of web searching lead me to the Visual Studio Template Reference. Each time a new item is created in Visual Studio, VS finds the template definition that matches the item and generates the code to match. Templates support logical control flow and variable replacement. By default the C# class templates are stored in "\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\ItemTemplates\CSharp\Code\1033", with similar templates stored nearby. The instructions are geared the reader toward creating their own templates for new types of items, which is a very useful tool (but I can do that with resharper), but I really just wanted to change the default. So I went in and did it: no more "using" and always public.

Before my changes would take effect, I first had to close visual studio and then run the following command,

devenv /installvstemplates

which will rebuild the VS templates run-time cache folder.

Now I can focus on all the fun features of 3.5!

VI - Macros

2009-12-09T08:49:00.005-08:00

Abstracting repetitive steps of work in an on-the-fly macro is a task I find myself doing almost daily during development or database scripting. After hammering a pretty good one today, I was wondering to myself how to go about saving it. I did some Google searching ("save vi macro", "copy vi macro"), but came up empty. Then I realized that I already knew the answer. Whenever I record a new vi macro, I record to the q register. So I knew that my macro must be sitting as plain old text in the q register. So, I pasted the q register to my screen and lo and behold, there was my macro! I created a macros text file to track all my commonly used macros with descriptions of what they do. Yay!

Tide?

2009-10-12T19:07:00.000-07:00

http://www.seafriends.org.nz/oceano/tides.htm

VI and Ruby

2009-10-07T20:53:00.003-07:00

I've decided it's way past time to get up to speed with Ruby. After several years of constantly changing my personal idiom for javascript styling due to learning things in pieces, I decided that I would try to learn the Ruby style up front. The two biggies that I've been offending are,

One blank space between comment hash (#) and first character
Spaces (and two of those, btw) not tabs

Since I'm just starting this blog thing, I haven't had the chance yet to mention that I'm a big VI fan. Naturally, I'm doing my Ruby development in VI. So, first I needed to make some use of VI's find and replace expression with backreferencing power for #1,

# Insert a blank space into all comments not beginning with one blank space but be careful to avoid replacing other uses of # (e.g. #{})

:%s/#$\w$/# \1/

Nice! That was easy. Yet another experience of feeling pretty good about VI.

Now on to style tip #2. Other than the simple search/replace to clean up my existing code, I had to change my tabs to spaces. Some simple changes to my .vimrc and I was ready to go!

#tab = two spaces

set tabstop=2

#use spaces and not the tab character

set expandtab

log4net: No layout set for the appender named

2009-08-21T06:14:00.005-07:00

I recently created a new ASP.NET web project and set it up to use log4net with a RollingFileAppender as its logging framework. However, at first my log file wasn't showing up at all and then after I got it showing up, my log statements were not being written. Fixing this involved two steps.

First, the DLL reference generated by Visual Studio did not automatically set the log4net assembly to copy local. Setting this property on the reference solved the issue of my log file not appearing.

Second, now that I had my file, I was not seeing any log statements written to it. Debugging the process, I saw the following error output from log4net:

AppenderSkeleton: No layout set for the appender named

Looking at my configuration file, I saw that I was setting an appender layout. However, I noticed that my appender layout was based on a type defined in a different project. Checking the reference to that project, I again found that Visual Studio was not copying a local version of the assembly, i.e. the Private flag was set to false. Switching this flag to true fixed the problem and now my logs are showing up!

Long Method Names

2009-08-19T21:12:00.002-07:00

As code completion (or auto complete) has become more ubiquitous and second nature to most software developers, so also has the tendency to spend less time thinking critically about the names that we assign our variables and methods. In place of this, I have observed the likelihood to squeeze the entire purpose of the variable or method into the name (e.g. DoThisUnlessThatIsTrueAndBeGentle()). Perhaps we may soon have the entire algorithm of a method squeezed into a name (with the obvious advantage of not needing to read the method at all!). Have we lost (or forgotten) the advantage of abstracting the purpose of a variable or implementation of a method (or class) behind a simple, distinct name?

Disadvantages:
1. It's almost never possible to completely and accurately name a variable or method. Any attempt to do so may lead to more confusion.
2. Side-by-side comparisons of two versions of the same code become more painstaking with horizontal scrolling -- perhaps a bug is even missed because the developer is too lazy/busy to mess with the scrolling.
3. Constrained to multi-line formatting

Advantages:
1. More descriptive names?

I am often reminded of an article by Joel Spolsky (back when I read his stuff): http://www.joelonsoftware.com/articles/Wrong.html (scroll to the "I'm Hungary" section towards the bottom) when I ruminate on this subject.