My team was provisioning a ruby on rails application on a new Operating System with different ruby and client libraries and we came across a strange bug.

Intermittently we were seeing the failures in the application with the following messages in the passenger log:

NoMethodError: undefined method `each' for nil:NilClass

and

(NoMethodError) "undefined method `each' for nil:NilClass"

/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/mysql2-0.2.4/lib/active_record/connection_adapters/mysql2_adapter.rb:628:in `select'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/connection_adapters/abstract/query_cache.rb:54:in `block in select_all'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `cache_sql'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/connection_adapters/abstract/query_cache.rb:54:in `select_all'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/base.rb:467:in `find_by_sql'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/relation.rb:64:in `to_a'
/usr/local/rvm/gems/ruby-1.9.2-p0/ruby/1.9.1/gems/activerecord-3.0.0/lib/active_record/relation/finder_methods.rb:143:in `all'

We found similar people experiencing the same problem here though the issues page on github didnt state a resolution.

We discovered we were running the MySQL 5.5 client libraries - namely the libmysqlclient 18. mysql2.so was built against  libmysqlclient 18:


[root@host]# ldd ./vendor/bundle/ruby/1.8/gems/mysql2-0.2.13/ext/mysql2/mysql2.so
linux-vdso.so.1 =>  (0x00007fff51788000)
libruby.so.1.8 => /usr/lib64/libruby.so.1.8 (0x00007f47f68e4000)
libmysqlclient.so.18 => /usr/lib64/libmysqlclient.so.18 (0x00007f47f6389000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f47f6165000)
libm.so.6 => /lib64/libm.so.6 (0x00007f47f5ee1000)
librt.so.1 => /lib64/librt.so.1 (0x00007f47f5cd9000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f47f5ad4000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f47f589d000)
libc.so.6 => /lib64/libc.so.6 (0x00007f47f551e000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f47f5307000)
/lib64/ld-linux-x86-64.so.2 (0x0000003eccc00000)
libfreebl3.so => /usr/lib64/libfreebl3.so (0x00007f47f50aa000)



We are running Centos6 and use the Percona MySQL client libraries, so we downgraded from Percona-Server-shared-55 to Percona-Server-shared-51 to provide the earlier client library.

We gem bundled against an earlier version of MySQL client libraries (libmysqlclient 16) and we no longer experienced that intermit failure:


[root@host]# ldd vendor/bundle/ruby/1.8/gems/mysql2-0.2.13/lib/mysql2/mysql2.so
linux-vdso.so.1 =>  (0x00007fff9e7d5000)
libruby.so.1.8 => /usr/lib64/libruby.so.1.8 (0x00007ff304784000)
libmysqlclient_r.so.16 => /usr/lib64/mysql/libmysqlclient_r.so.16 (0x00007ff30436e000)
libz.so.1 => /lib64/libz.so.1 (0x00007ff304159000)
librt.so.1 => /lib64/librt.so.1 (0x00007ff303f51000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff303d33000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007ff303afc000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00007ff3038e3000)
libm.so.6 => /lib64/libm.so.6 (0x00007ff30365e000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007ff30345a000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff3030db000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff302ec4000)
/lib64/ld-linux-x86-64.so.2 (0x0000003a52200000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007ff302c67000)



I'm yet to discover the specific issue with what I think is asynchronous caching race conditions and libmysqlclient18 with the mysql2 gem, but wanted to share my experience with the wider community to avoid others from going down the same rabbit hole I went down today.