How to tell SOLR not to match IDs in a field

Question

I have a filter query to return all the records matching an id in a field

$query->createFilterQuery('categoryId')->setQuery(
  sprintf('categoryId:(%s)', implode(' OR ', $categoryId))
);

this works. But how can I match everything that doesn't have that id? I need to replace OR with AND

$query->createFilterQuery('categoryId')->setQuery(
  sprintf('!categoryId:(%s)', implode(' AND ', $categoryId))
);

but I don't know how to tell SOLR to exclude those ids?