How can I resolve Memory Error in Python?

Question

Below program, which extracts some specific column values from a text file delimited by separator "~" is throwing memory error when the number of records in the text file are around 6,000,000 PLUS. However the same snippet is working for smaller number of records in the text file. I tried using list comprehension but not able to resolve the memory error. Below is the code snippet:

import os
import csv

print("Analyzing the File!")

tnum = list()
nl = 0
total = list()
fdata = list()
amount = list()

with open(r"C:\Users\aojha\Desktop\20191125\ABC.TXT", "r") as fh:
 next(fh)
 for line in fh:
    line   =  line.rstrip()
    nl+=1

print("Number of records in file are:",nl)

with open(r"C:\Users\aojha\Desktop\20191125\ABC.TXT", "r") as fh:
 next(fh)
 fdata = [line.split("~") for line in fh]

##print(fdata)

for data in fdata:
    tnum.append((data[2][:]).split(" "))
    total.append((data[8][:]))
    amount.append((data[13][:]))
    #print(fdata)

print("Required data from file have been extracted!")}

Sample Input Data is in below format:

P~LNL~22248370~50~22248370~20190916~20191112~20190916~002~I~A~N~003~1638~01~001~400023~-1552~20190916~0200058~001~X~~TMID~~~~~000~000~~000~000~000~~000~000~0~~~~N~~~

Error received is as follows:

> Traceback (most recent call last): File "C:\Users\aojha\Desktop\Python\Random\Type_Count.py", line 23, in <module> fdata = [line.split("~") for line in fh] File "C:\Users\aojha\Desktop\Python\Random\Type_Count.py", line 23, in <listcomp> fdata = [line.split("~") for line in fh] MemoryError

Answer 1

Finally, I have solved this problem. Used sqlite for storing the data as the text file in my case was of 5Gb in size and in order to store the csv/text data in sqlite, I used csv reader and read those data using generator function/