Heroku & MongoHQ: ActionView::Template::Error (The operation: #<Moped::Protocol::Commands::Authenticate failed with error "auth fails")

Question

I have a Rails 3 app with MongoID 3 deployed successfully to Heroku (using MongoHQ) -- but then something happened (cannot pinpoint what exactly it was) and the build started crashing on Heroku.

So this is the error I get:

TLDR: Moped::Errors::AuthenticationFailure (The operation: #<Moped::Protocol::Commands::Authenticate ... failed with error "auth fails")

What makes this more difficult to debug is that it's working just fine on localhost. Now I'm running out of ideas what this could be.

app[web.1]: >> Thin web server (v1.4.1 codename Chromeo)
app[web.1]: >> Maximum connections set to 1024
app[web.1]: >> Listening on 0.0.0.0:58731, CTRL+C to stop
heroku[web.1]: State changed from starting to up
app[web.1]: 
app[web.1]: 
app[web.1]: Started GET "/" for 212.86.9.90 at 2012-07-31 08:08:07 +0000
heroku[router]: GET cool-name-123.herokuapp.com/ dyno=web.1 queue=0 wait=0ms service=171ms status=200 bytes=1286
heroku[router]: GET cool-name-123.herokuapp.com/assets/application-8e7bfeeffc9291864e5b42d908c2fdda.css dyno=web.1 queue=0 wait=0ms service=11ms status=200 bytes=92524
heroku[router]: GET cool-name-123.herokuapp.com/assets/application-aa557bde70f1236cdf90c913043c4382.js dyno=web.1 queue=0 wait=0ms service=16ms status=200 bytes=122836
heroku[router]: GET cool-name-123.herokuapp.com/favicon.ico dyno=web.1 queue=0 wait=0ms service=3ms status=200 bytes=0
app[web.1]: 
app[web.1]: 
app[web.1]: Started GET "/originals/new" for 212.86.9.90 at 2012-07-31 08:08:11 +0000
heroku[router]: GET cool-name-123.herokuapp.com/originals/new dyno=web.1 queue=0 wait=0ms service=31ms status=200 bytes=1808
app[web.1]: 
app[web.1]: 
app[web.1]: Started POST "/originals" for 212.86.9.90 at 2012-07-31 08:08:34 +0000
app[web.1]: 
app[web.1]: Moped::Errors::AuthenticationFailure (The operation: #<Moped::Protocol::Commands::Authenticate
app[web.1]:   @length=154
app[web.1]:   @request_id=3
app[web.1]:   @response_to=0
app[web.1]:   @op_code=2004
app[web.1]:   @flags=[]
app[web.1]:   @full_collection_name="app123.$cmd"
app[web.1]:   @limit=-1
app[web.1]:   @skip=0
app[web.1]:   @selector={:authenticate=>1, :user=>"heroku", :nonce=>"xyz", :key=>"xyz"}
heroku[router]: POST cool-name-123.herokuapp.com/originals dyno=web.1 queue=0 wait=0ms service=1486ms status=500 bytes=643
app[web.1]:   @fields=nil>
app[web.1]: failed with error "auth fails"):
app[web.1]:   app/controllers/originals_controller.rb:19:in `block in create'
app[web.1]:   app/controllers/originals_controller.rb:18:in `create'
app[web.1]: 
app[web.1]: 
heroku[router]: GET cool-name-123.herokuapp.com/favicon.ico dyno=web.1 queue=0 wait=0ms service=3ms status=304 bytes=0
app[web.1]: 
app[web.1]: 
app[web.1]: Started GET "/" for 212.86.9.90 at 2012-07-31 08:09:40 +0000
heroku[router]: GET cool-name-123.herokuapp.com/ dyno=web.1 queue=0 wait=0ms service=11ms status=304 bytes=0

I've recently struggled with Heroku, and as a result I think I should have the correct settings for the choice of db drivers & service (using the new yml scheme + Ruby 1.9.3 in the Gemfile).

When I run heroku config --app app_name I get

DATABASE_URL        => postgres://rhalppyjrb:[email protected]/rhalppyjrb
GEM_PATH            => vendor/bundle/ruby/1.9.1
LANG                => en_US.UTF-8
MONGOHQ_URL         => mongodb://heroku:[email protected]:27054/app123
PATH                => bin:vendor/bundle/ruby/1.9.1/bin:/usr/local/bin:/usr/bin:/bin
RACK_ENV            => production
RAILS_ENV           => production
SHARED_DATABASE_URL => postgres://rhalppyjrb:[email protected]/rhalppyjrb

My mongoid.yml looks like this:

production:
  sessions:
    default:
          uri: <%= ENV['MONGOHQ_URL'] %>
          options:
            consistency: :strong
            skip_version_check: true
            safe: true

Gemfile is like this:

source 'https://rubygems.org'

gem 'bundler', '1.2.0.rc'

ruby '1.9.3'

gem 'rails', '3.2.6'
gem 'thin'
gem 'rmagick', :require => 'RMagick'
gem "carrierwave-mongoid", :git => "git://github.com/jnicklas/carrierwave-mongoid.git", :branch => "mongoid-3.0", :require => 'carrierwave/mongoid'

group :assets do
  gem 'sass-rails',   '~> 3.2.3'
  gem 'coffee-rails', '~> 3.2.1'
  gem 'uglifier', '>= 1.0.3'
  gem "therubyracer", :platform => :ruby
end

gem 'jquery-rails'
gem "haml", ">= 3.1.6"
gem "mongoid", ">= 3.0.3"
gem "devise", ">= 2.1.2"
gem "devise_invitable", ">= 1.0.2"
gem "cancan", ">= 1.6.8"
gem "rolify", ">= 3.1.0"
gem "bootstrap-sass", ">= 2.0.4.0"
gem "simple_form"

group :development do
  gem "guard", ">= 0.6.2"
  gem 'rb-fsevent'
  gem 'growl'
  gem "haml-rails", ">= 0.3.4"
  gem "guard-bundler", ">= 0.1.3"
  gem "guard-rails", ">= 0.0.3"
  gem "guard-livereload", ">= 0.3.0"
  gem "guard-rspec", ">= 0.4.3"
end

group :test do
  gem "database_cleaner", ">= 0.8.0"
  gem "mongoid-rspec", "1.4.6"
  gem "email_spec", ">= 1.2.1"
end

group :development, :test do
  gem "factory_girl_rails", ">= 3.5.0"
  gem "rspec-rails", ">= 2.11.0"
end

Answer 1

The issue is that your authentication credentials are no longer correct. Specifically, the user you are trying to authenticate as no longer exist.

The reason is that when you do a rake db:reseed all data is removed, including the DB level users.

Example:

  ➜  skadi git:(master) ✗ mongo         
  MongoDB shell version: 2.0.7
  connecting to: test
  > use skadi_development
  switched to db skadi_development
  > db.system.users.find()
  > db.addUser("joe", "passwordForJoe")
  { "n" : 0, "connectionId" : 191, "err" : null, "ok" : 1 }
  {
    "user" : "joe",
    "readOnly" : false,
    "pwd" : "dac588613249fe92703afb262ec53b82",
    "_id" : ObjectId("5030d5181cefbd2b04a99e30")
  }
  > db.system.users.find()
  { "_id" : ObjectId("5030d5181cefbd2b04a99e30"), "user" : "joe", "readOnly" : false, "pwd" : "dac588613249fe92703afb262ec53b82" }
  > exit
  bye

  ➜  skadi git:(master) ✗ rake db:reseed

  ➜  skadi git:(master) ✗ mongo         
  MongoDB shell version: 2.0.7
  connecting to: test
  > use skadi_development
  switched to db skadi_development
  > db.system.users.find()
  => Nothing

So you when you try to connect to the database, your credentials are no longer valid.

You can check that in you MongoHQ admin panel. You can find it by going to Heroku, selecting your app, clicking on resources and clicking MongoHQ. Then click on the Database users tab.

rake db:reseed used to work as you expected, but it changed with the new version of mongoid. In order to remove all collections, except system.*, you may use rake db:purge.

You may need to use the master branch since that rake task is not yet available in the stable version. Or just use Mongoid.purge! in your code, that works in stable.

Details: https://github.com/mongoid/mongoid/issues/2275