I am facing problem in removing the duplicate entries. There are some duplicated pwdHistory in each session in a large ldif file. The entries are like:
dn: uid=mdwu701,ou=people,c=CN,dc=com
passwordRetryCount:0
passwordExpirationTime: 19700101000001Z
pwdHistory: 20171012140955412Z{SSHAS12}5PD67B30S1RgjybrVs61k1nErm/10USOnadfadf==
pwdHistory: 20171012140955412Z{SSHAS12}dafdaf32431adfJKHADKFHKAUD230324I-=23READLLJDASFEDFADF=-
pwdHistory: 2017101140654412Z{SSHAS12}5PD67B30S1RgjybrVs61k1nErm/10USOnkladjflajfda77DAFWE3DCsdfafeqwp+fupN
pwdHistory: 201710123064712Z{SSHAS12}37005PD67B30adsfjadfWQERQEWRVCADFADcfdaf98023=asdfad2/dafggalmjcp
pwdHistory: 2017101140654412Z{SSHAS12}5PD67B30S1RgjybrVs61k1nErm/10USOnkladjflajfda77DAFWE3DCsdfafeqwp+fupN
modifyTimestamp: 20171012165711Z
modifiersName: uid=mdwu701,ou=people,c=CN,dc=com
userPassword: {SSHA512}817XX1zaQaTAqV9wYE3Ao8orl7cBdCktr3Kgnuh1RgbC/vjfiaR2T/58KxEanS1Gzht0z1G1JhAigMLd8/fWsVQ32xATr8n
uid: mdwu701
objectClass: top
objectClass: person
objectClass: inetOrgPerson
sn: mdwu701
dn: mdwu701
createTimestamp: 2017010005943Z
dn: uid=mdwu702,ou=people,c=CN,dc=com
passwordRetryCount:0
passwordExpirationTime: 19700101000001Z
pwdHistory: 20171012140955412Z{SSHAS12}dafdaf32431adfJKHADKFHKAUD230324I-=23READLLJDASFEDFADF=-
pwdHistory: 201710123064712Z{SSHAS12}37005PD67B30adsfjadfWQERQEWRVCADFADcfdaf98023=asdfad2/dafggalmjcp
pwdHistory: 2017101140654412Z{SSHAS12}AAAAda322cdfadf=dafagedcdadafaErm/10USOnkladjflajfda77DAFWE3DCsdfafeqwp+adfe=0923=reafaP0+
pwdHistory: 201710123064712Z{SSHAS12}37005PD67B30adsfjadfWQERQEWRVCADFADcfdaf98023=asdfad2/dafggalmjcp
pwdHistory: 2017101140654412Z{SSHAS12}AAAAda322cdfadf=dafagedcdadafaErm/10USOnkladjflajfda77DAFWE3DCsdfafeqwp+adfe=0923=reafaP0+
modifyTimestamp: 20171211165311Z
modifiersName: uid=mdwu702,ou=people,c=CN,dc=com
userPassword: {SSHA512}jkladf23234kdfdafIKLJDF32-=ADSFCV/ADSFADF32DDAkjdafdgoojerqvccxadfad/adsfadsfre3werweqrpYRE6
uid: mdwu702
objectClass: top
objectClass: person
objectClass: inetOrgPerson
sn: mdwu702
dn: mdwu702
createTimestamp: 20171111115943Z
dn: uid=mdwu703,ou=people,c=CN,dc=com
passwordRetryCount:0
passwordExpirationTime: 19700101000001Z
pwdHistory: 201710123064712Z{SSHAS12}37005PD67B30adsfjadfWQERQEWRVCADFADcfdaf98023=asdfad2/dafggalmjcp
pwdHistory: 201710101408342412Z{SSHAS12}jybrVs61k1nErm/10USOnadfadf==5PD67B30S1Rg
pwdHistory: 2017101140654412Z{SSHAS12}adfadfafeweqxczz3134153SDGFADSREWE3DCbadfadfe33JKLL=ADFA32dafaf34dfeasdfalkdf0=
pwdHistory: 201710123064712Z{SSHAS12}37005PD67B30adsfjadfWQERQEWRVCADFADcfdaf98023=asdfad2/adfcvedtpdGFDE
pwdHistory: 2017101140654412Z{SSHAS12}adfadfafeweqxczz389VGHVHdfgdgfdjkkjkhkhkh3JKLL=ADFA32dafaf34dfeasdfadf2POI=
pwdHistory: 2017101140654412Z{SSHAS12}adfadfafeweqxczz3134153SDGFADSREWE3DCbadfadfe33JKLL=ADFA32dafaf34dfeasdfalkdf0=
modifyTimestamp: 20170912165707Z
modifiersName: uid=mdwu703,ou=people,c=CN,dc=com
userPassword: {SSHA512}817XXqerqer==-adsfad1zaQaTAqV9wWsVQ32xATr8nzzYE3Ao8orl7cBdCktr3Kgnuh1RgbCxf87234dfkljcadsfldfg23d/
uid: mdwu703
objectClass: top
objectClass: person
objectClass: inetOrgPerson
sn: mdwu703
dn: mdwu703
createTimestamp: 20171222224532Z
There are some duplicated in pwdHistory which need to be removed. I tried to use
awk '!/pwdHistory/ || !seen[$0]++' test.ldif
to clean up the duplicate entries. However, it will remove all the duplicated pwdHistory. I want to remove the duplicated pwdHistory in each dn.