CanCanCan with Rails 5 and defining abilities in the database

Question

I'm using CanCanCan in my Rails 5 app (5.1.3) and defining abilities in the database. This is my structure for defining the abilities -

Person --- Role --- Permission --- Actions. 1 person can have many roles, each role maps to many permissions and each permission has many actions. PermissionsRole and ActionsPermission both have a has_and_belongs_to_many relation.

In the actions table, I've defined the fields name, resource_name, and is_object.

this is in my Ability.rb -

person.roles.first.permissions.each do |permission|
    permission.actions.each do |action|
      if action.is_object
        can action.name.to_sym, action.resource_name.constantize
      else
        can action.name.to_sym, action.resource_name.to_sym
      end
    end
  end

This seems to be working fine if is_object is false ie if it's only a controller without a model. I'm trying to use this to show records of an object and I'm getting the accessdenied error even though all the associations have been setup properly.

Does anyone know how to fix this?

EDIT : I had overwritten the default_scope for the model in question which was causing access denied for all the records. When I remove that it works, but then I see all the records, instead of the records which that user can access. Is there anyway to overwrite the default scope and get the record authorizations working?