Best structure of Multiple indexed map

Question

Problem

Multiple indexed map :

• A value has several key, that can be used to find the value in a short time (may be less than O(n)).
• The key is NON-UNIQUE, which means key1: val1, key1: val2, key1: val3 could be possible.
• When the value is removed, all equal values must be removed from the map.

Example

Structure of Book queue.

• A Book has multiple authors, and the Authours can be searched in a short time.
• Book is concurrently added or removed in that Queue.
• Authors can write multiple Books.

Process

• Author is from input, then find anyone of the books in queue.
• The book is removed from the queue.

One I thought is the map can be simillar to multimap, but removing the key-value pairs from the map while iterating the authors of the book is unsuitable solution. When the author and books total count is about millions, the elapsing time can get greater.

Can anybody help me to decide the structure?

UPDATE:

As @gdomo suggested(book-author, author-book map), I wrote code as follows:

    public class Book {
        ...
        private List<String> authors;
    }

    Set<Book>> pendingBooks = ConcurrentHashMap.newKeySet();

    ConcurrentHashMap<String, Set<Book>> cacheMapBooksByAuthor = new ConcurrentHashMap<>();

    public static void processAuthorCheck(String author) {
        Set<Book> books = cacheMapBooksByAuthor.put(author, ConcurrentHashMap.newKeySet());
        if (books == null)
            return;

        for (Book book : books) {
            pendingBooks.remove(book);          // REMOVE a BOOK from QUE

            this.processBook(book); // TODO

            List<String> authors = book.getAuthors();
            if (authors == null)
                continue;
            for (String author : authors) {
                Set<Book> tmpBooks = cacheMapBooksByAuthor.get(author).remove(book);        // remove THAT BOOK for all other authors
            }
        }
    }

    public static void pushBook(Book book) {
        List<String> authors = book.getAuthors();
        if (authors == null)
            return;

        for (String authors : authors) {
            Set<Book> books = cacheMapBooksByAuthor.get(authors);
            if (books == null) {
                Set<Book> oldBooks = cacheMapBooksByAuthor.putIfAbsent(authors, books = ConcurrentHashMap.newKeySet());
                if (oldBooks != null)
                    books = oldBooks;
            }
            books.add(book);
        }
    }

I used Set instead of Queue to avoid conflicts of book.

One I'm not satisfied with is release of book variable does't affect authors book variables, as i exprienced in C++. (Because of java object reference)

Is there any better solution over this structure?