I used editcap with option "-d" on a large pcapng file to delete duplicate packets (source file 11GB, new file 5 GB). After that, i extracted all contained files from both pcapng-files (with Networkminer free). I assumed there would be no data loss. Instead, as I compared the two extraction folders (AssembledFiles), I found out that about 30 files were missing in one of them. How is this to explain? Why should deleting duplicate packets result in data loss?
I downloaded Wireshark from "https://www.wireshark.org/download.html" File: Windows Installer (64-bit) - Stable Release 2.6.6 - (v2.6.6-0-gdf942cd8)
Then I got editcap (v 2.6.6.0) from the Wireshark installation folder. <<...... c:\Program Files\Wireshark\ ......>>
Batch:
<<...…
@echo off
echo.
echo.
echo 1. print editcap version
echo.
"c:\Program Files\Wireshark\editcap.exe" -V
echo.
echo.
echo 2. removing duplicate packets from File.pcapng
echo.
"c:\Program Files\Wireshark\editcap.exe" -d "File.pcapng" "File_nodup.pcapng"
echo.
echo.
echo 3. converting File.pcapng and File_nodup.pcapng to pcap format
echo (to make it readable for NetworkMiner free, forgot to mention this step...)
echo.
"c:\Program Files\Wireshark\editcap.exe" -F pcap "File.pcapng" "File.pcap"
echo.
"c:\Program Files\Wireshark\editcap.exe" -F pcap "File_nodup.pcapng" "File_nodup.pcap"
echo.
echo done.
pause
......>>
Result:
<<......
Editcap (Wireshark) 2.6.6 (v2.6.6-0-gdf942cd8)
Copyright 1998-2019 Gerald Combs and contributors. License GPLv2+: GNU GPL version 2 or later http://www.gnu.org/licenses/old-licenses/gpl-2.0.html This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Compiled (64-bit) with GLib 2.42.0, with zlib 1.2.11.
Running on 64-bit Windows 10, build 17763, with Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz (with SSE4.2), with 7841 MB of physical memory, with locale C, binary plugins supported (1 loaded).
Built using Microsoft Visual Studio 2017 (VC++ 14.12, build 25835).
- removing duplicate packets from File.pcapng
13625734 packets seen, 6814005 packets skipped with duplicate window of 5 packets.
- converting File.pcapng and File_nodup.pcapng to pcap format
(to make it readable for NetworkMiner free, forgot to mention this step...)
done.
......>>
New Batch:
@echo off
echo.
echo.
echo 1. converting File.pcapng to pcap format
"c:\Program Files\Wireshark\editcap.exe" -F pcap "File.pcapng" "File.pcap"
echo.
echo.
echo 2. removing duplicate packets from File.pcap (parameter "-F pcap" is needed to avoid output to be reconverted to pcapng...)
"c:\Program Files\Wireshark\editcap.exe" -d -F pcap "File.pcap" "File_nodup.pcap"
echo.
echo done.