mariadb: select table prevents insertions due to locking

Question

using mariadb 10.3.35, linux 7, no galera cluster, no slave: a simple instance. I have 2 innodb big tables (millions of rows) that are identical, except one is partitioned and the other not:

 CREATE TABLE `AuditRecordDataOld` (
  `pk` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `additionalDetails` longtext DEFAULT NULL,
  `authToken` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `customId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
  `eventStatus` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `eventType` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `module` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `nodeId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `rowProtection` longtext DEFAULT NULL,
  `rowVersion` int(11) NOT NULL,
  `searchDetail1` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
  `searchDetail2` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
  `sequenceNumber` bigint(20) NOT NULL,
  `service` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `timeStamp` bigint(20) NOT NULL,
  PRIMARY KEY (`pk`),
  UNIQUE KEY `auditrecorddata_idx2` (`nodeId`,`sequenceNumber`),
  KEY `auditrecorddata_idx3` (`timeStamp`),
  KEY `auditrecorddata_idx4` (`searchDetail2`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=4;

and the other

CREATE TABLE `AuditRecordData` (
  `pk` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `additionalDetails` longtext DEFAULT NULL,
  `authToken` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `customId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
  `eventStatus` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `eventType` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `module` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `nodeId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `rowProtection` longtext DEFAULT NULL,
  `rowVersion` int(11) NOT NULL,
  `searchDetail1` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
  `searchDetail2` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
  `sequenceNumber` bigint(20) NOT NULL,
  `service` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `timeStamp` bigint(20) NOT NULL,
  PRIMARY KEY (`pk`,`timeStamp`),
  UNIQUE KEY `auditrecorddata_idx2` (`nodeId`,`sequenceNumber`,`timeStamp`),
  KEY `auditrecorddata_idx3` (`timeStamp`),
  KEY `auditrecorddata_idx4` (`searchDetail2`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=4
 PARTITION BY RANGE (`timeStamp`)
(PARTITION `p201912` VALUES LESS THAN (1577833200000) ENGINE = InnoDB,
 PARTITION `p202001` VALUES LESS THAN (1580511600000) ENGINE = InnoDB,
 PARTITION `p202002` VALUES LESS THAN (1583017200000) ENGINE = InnoDB,
 PARTITION `p202003` VALUES LESS THAN (1585692000000) ENGINE = InnoDB,
....
PARTITION `pDefault` VALUES LESS THAN MAXVALUE ENGINE = InnoDB)

To be able to check that all records in AuditRecordDataOld are present are also in AuditRecordData, I made

select pk from AuditRecordDataOld x WHERE not exists(select 1 from AuditRecordData where pk=x.pk);

The select has run 5 hours, returning no row without error. But problem: during these 5 hours, all apps inserting in AuditRecordData crashed. (There are only inserts in this table, never update/delete). java error:

2023-09-20 01:03:01,274 ERROR [org.jboss.as.ejb3.invocation] (default task-3710) WFLYEJB0034: EJB Invocation failed on component StatusRepositorySessionBean for method public abstract java.util.Map org.signserver.statusrepo.StatusRepositorySession.getAllEntries(): javax.ejb.ConcurrentAccessTimeoutException: WFLYEJB0241: EJB 3.1 PFD2 4.8.5.5.1 concurrent access timeout on StatusRepositorySessionBean - could not obtain lock within 5000MILLISECONDS

Questions: can a select query block inserts? The primary key is not an autonumber but a varchar(250) (combined with a timestamp because of partitioning in AuditRecordData , but the varchar pk is unique). What can be done to avoid this kind of query to block inserts/lock the whole table (and so the whole system) for several hours? (Even raising timeout to 5 hours is not a valid option). Isolation level is repeatable read.