I have close to 200M records in my tables. My migration script runs very slowly (for hours).
---My OLD Tables---
orders_old table 30M row count
| id | createdDate | status | price |
|---|---|---|---|
| 1000376453 | 2021-10-14 | completed | 286 |
| 1000368038 | 2021-11-24 | cancelled | 334 |
order_status_history_old table 150M row count
| modified_date | order_id | old_status | new_status |
|---|---|---|---|
| 2022-06-22 | 1000376453 | created | accepted |
| 2022-06-23 | 1000376453 | accepted | completed |
| 2022-07-16 | 1000368038 | created | accepted |
| 2022-07-16 | 1000368038 | accepted | completed |
| 2022-07-20 | 1000368038 | completed | cancelled |
---My NEW Tables---
history_info table
| id | date | updated_by |
|---|---|---|
order_history table
| id | rev | rev_type | status |
|---|---|---|---|
---My PlSql Code---
DO $$
DECLARE
ORDER_TABLE_RECORD RECORD;
HISTORY_INFO_ID INTEGER;
ORDER_STATUS_HISTORY_TABLE_RECORD RECORD;
BEGIN
FOR ORDER_TABLE_RECORD IN SELECT * FROM ORDERS_OLD LOOP
RAISE NOTICE 'ORDER STARTED %', ORDER_TABLE_RECORD.ID;
FOR ORDER_STATUS_HISTORY_TABLE_RECORD IN SELECT * FROM ORDER_STATUS_HISTORY_OLD OH WHERE OH.ORDER_ID = ORDER_TABLE_RECORD.ID LOOP
HISTORY_INFO_ID := NEXTVAL('HISTORY_INFO_SEQ');
INSERT INTO HISTORY_INFO(ID, DATE, UPDATED_BY)
VALUES(HISTORY_INFO_ID, ORDER_STATUS_HISTORY_TABLE_RECORD.last_modified_date, ORDER_STATUS_HISTORY_TABLE_RECORD.status_changer);
INSERT INTO ORDER_HISTORY(ID, REV, REVTYPE, STATUS)
VALUES(ORDER_TABLE_RECORD.ID :: BIGINT, HISTORY_INFO_ID, 1, 'NEW');
END LOOP;
END LOOP;
END $$;
Also, I'm trying alternative scripts like below script but I am getting an error.
DO $$
DECLARE
BEGIN
WITH OSHO_DATA AS (
select osho.order_id, osho.new_order_status, OSHO.LAST_MODIFIED_DATE, OSHO.STATUS_CHANGER
from orders_old oo
left join order_status_history_old osho on osho.order_id = oo.id
),
HI_DATA AS (
INSERT INTO HISTORY_INFO(ID, DATE, UPDATED_BY)
SELECT NEXTVAL('HISTORY_INFO_SEQ'), OSHO_DATA.LAST_MODIFIED_DATE, OSHO_DATA.STATUS_CHANGER
FROM OSHO_DATA LIMIT 1
RETURNING *
)
INSERT INTO ORDER_HISTORY(ID, REV, REVTYPE, STATUS)
SELECT OSHO_DATA.order_id :: BIGINT, HI_DATA.ID, 1, GET_STATUS_CODE(OSHO_DATA.new_order_status)
FROM OSHO_DATA, HI_DATA;
END $$;
How do I migrate my old data to my new tables quickly with performance?
NOTE: The reason why I use loop instead of bulk insert is because I use the id after the first insert in the next insert process.