Postgres trigger that inserts in a secondary table - sometimes seem to not work

Question

I have a main table, let's call it A. And a trigger.

create trigger my_trigger
    after insert or update or delete
    of column1, column2
    on "A"
    for each row
execute procedure trigger_logic();

create function trigger_logic() returns trigger
    language plpgsql
as
$$
DECLARE
    reference_id varchar;
BEGIN
    IF TG_OP = 'DELETE' then
        reference_id = OLD.id;
    ELSE
        reference_id = NEW.id;
    END IF;
    INSERT INTO B (reference)
    SELECT reference_id
        WHERE NOT EXISTS(
        SELECT 1
        FROM B
        WHERE reference = reference_id
    );
    RETURN NEW;
END;
$$;

Overall works well, but then on several times I ended up with one row in table A and no corresponding row in table B, even if I run afterwards the query with "WHERE NOT EXISTS" and it should have inserted it.

I have 1 million correct record pairs and around 10 with no entry in table B.

I don't really understand how this is possible. Isn't the trigger transactional with the operation in table A. Is there any situation in which this could happened. I also checked the postgres logs and didn't see anything out of the ordinary.