python combine two file which data is inconsistency

Question

now I have two files: A and B. Both are csv files.

csv A has some data:

a b c 
1 2 3
4 5 6 
7 8 9
.....

csv B has some data:

b d
7 0
8 3
2 1
.....

now I want to combine the two csv files A and B for a new file C like that:

a b c d
1 2 3 1
7 8 9 3
......

first I have to read the file with csv.reader,the code is like that:

    def open_func(file_name):
    open_dict={}
    key_list=[]
    fd=csv.reader(open(file_name,'r'))
    j=1
    for line in fd:
        data_len=len(line)
        if not j:
            try:
                for i in range(data_len):
                    open_dict[key_list[i]].append(line[i])
            except:
                #print line 
                continue            
        else:  
            for i in range(data_len):
                key=line[i]
                key_list.append(key)
            for i in range(data_len):
                open_dict[key_list[data_len-i-1]]=[]
            j=0
            continue
    return open_dict

I use dict to read them, and I want to key to combine if it is equal

but I don't know how to do then

notice: the data is more than one million rows.

Answer 1

I strongly suggest you to use pandas to do this, it provides high-performance, easy-to-use data structures and data analysis tools for Python. So you can try it to process large data.

e.g.

>>> import pandas as pd
>>> f1 = pd.DataFrame({'a':[1,4,7], 'b':[2,5,8], 'c':[3,6,9]})
>>> f2 = pd.DataFrame({'b':[7,8,2], 'd':[0,3,1]})
>>> f = f1.merge(f2, how='left', on='b')
>>> f
   a  b  c    d
0  1  2  3  1.0
1  4  5  6  NaN
2  7  8  9  3.0

Or read file from csv:

f1 = pd.read_csv('1.csv',sep="\s+")
f2 = pd.read_csv('2.csv',sep="\s+")

and you can write it to file by this:

f.to_csv('out.csv', index=False)

out.csv:

a,b,c,d
1,2,3,1.0
4,5,6,
7,8,9,3.0

Answer 2

pandas.read_csv make this fairly easy:

Code:

import pandas as pd

df1 = pd.read_csv('file1', sep='\s+')
df2 = pd.read_csv('file2', sep='\s+')

df = df1.merge(df2, on=['b'])
print(df)

Results:

   a  b  c  d
0  1  2  3  1
1  7  8  9  3