I am not sure if it is possible to basically gzip the stream of log lines as they arrive at the logback appender, rather than compressing the file when we log-rotate. Is that at all possible, and if so, how to achieve that and is there a lot of benefit of compressing "on the fly" rather than the whole file?
Logback - compressing log lines before writing to file
520 Views Asked by Bober02 At
1
There are 1 best solutions below
Related Questions in JAVA
- Add image to JCheckBoxMenuItem
- How to access invisible Unordered List element with Selenium WebDriver using Java
- Inheritance in Java, apparent type vs actual type
- Java catch the ball Game
- Access objects variable & method by name
- GridBagLayout is displaying JTextField and JTextArea as short, vertical lines
- Perform a task each interval
- Compound classes stored in an array are not accessible in selenium java
- How to avoid concurrent access to a resource?
- Why does processing goes slower on implementing try catch block in java?
- Redirect inside java interceptor
- Push toolbar content below statusbar
- Animation in Java on top of JPanel
- JPA - How to query with a LIKE operator in combination with an AttributeConverter
- Java Assign a Value to an array cell
Related Questions in LOGGING
- Is Log4j2 xml configuration case sensitive?
- Logback stopped logging after splitting shared config file
- logging setup best practices
- C Simple Logging Management
- OpenShift Pyramid logging to file
- Log of dependency does not show
- Node/Express access logger from request object
- How does one locate all git log messages in the git object database?
- Logging error when executing Maven SonarQube plugin
- refactor 'execute and log' pattern
- CMD specifying columns to save?
- How to get information about error from HttpContext in WCF services
- Django not logging all errors
- Empty space at beginning of rsyslog log file
- Avoid log trace of external framework J2EE
Related Questions in COMPRESSION
- How to use deflate/inflate SetDictionary with raw deflate/inflate?
- C# How to get file/ copy file from a bzip2 (.bz2) file without extracting the file
- How can I compress four floats into a string?
- Create ZIP File Then Send to Client
- compress json data from rest node.js use express compression
- Advanced Data Compression
- Tools to minify CDD and JS files
- How to use multiple threads for zlib compression (same input source)
- Data compression in RDBMS like Oracle, MySQL etc
- Haskell - Lempel-Ziv 78 Compression - very slow, why?
- Python: how to create tar file and compress it on the fly with external module, using different compression methods not available in tarfile module?
- Why isn't lossless compression automatic on computers?
- PHP Image Compression Before Upload
- Compression of char size integer by removing leading zeroes
- BMP Image Compression and Decompression in java
Related Questions in GZIP
- Convert JSON.gz to JSON in node js
- How can i read a json file in gzip?
- Does anyone know how to read gzip file(gzip in thr spoolSourceDirectory) in Flume process?
- Grep a gzipped file?
- Compress json array list in java(Spring mvc) and decompress it in javascript(angular js)
- Seeking on a gz connection is unpredictable
- How do I grep GZ files to extract PNG files?
- I want to create a script for unzip (.tar.gz) file via (Python)
- getting CRC error while decompressing a gzip file in python
- Delete specific line(pattern) from .gz file using python for large file size
- How do I link a gzipped version of a stylesheet?
- Glassfish4 gzip encoding issue (corrupted responses)
- Apache settings to send gzipped CSS/JS files to browser
- Open a gz file using Minizip Library
- gzip and pipe to output (performance consideration)
Related Questions in LOGBACK-CLASSIC
- Logback ClassCastException for LoggerFactory
- logback. class not found exception
- Logback level element vs. level attribute
- Logback - compressing log lines before writing to file
- Adding custom header in dropwizard access logs
- ch.qos.logback.contrib json logging alternative
- Logback created files not visible in file manager sometimes. When I restart the device then it is visible
- marker is not an rvalue in logback
- Logback error when migrating a PDF2 plugin from DITA-OT 4.0.1 to 4.1.2
- Logbback xml - Conditionally add MDC Variables if they exist?
- Converting logback.xml to application.properties
- How to print http status in logs with springboot logback
- Logs not printed in Logback- Java Web App
- from Java 11 on windows10, how best to detect if System.out is Ansi enabled
- JSON Layout with Pattern in Logback Java
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Sure. You can simply keep a gzip compression process open and feed it lines as they come in. That would significantly reduce the space required by the log file, and would not take any more CPU resources on average, since you were going to eventually compress it anyways.
The downside is that at any point in time the compressed log file will not yet contain many of the already supplied log lines, since there is a latency and burstiness to the compression process. Many lines will need to be accumulated before a compressed block is emitted. Second, the compressed file will not be a valid gzip file until it is closed. You would still be able to decompress what's there, but it will not have the trailer with a check value. If the process is killed or the machine crashes, you are left with an invalid gzip file that doesn't have the most recent several log lines. Of course, the most recent log lines may be exactly the ones that you're most interested in, to find out what the heck happened.
All of those downsides can be cured with a specialized approach for this application, which is implemented in gzlog.h/gzlog.c. gzlog assures that after each line is written, the gzipped log file is complete and valid, and contains that log line. Furthermore, it can reconstruct the gzip file with the last provided log line even if the gzlog process itself is interrupted in the middle of adding a log line.