Logzip: Extracting Hidden Structures via Iterative Clustering for Log\n Compression

System logs record detailed runtime information of software systems and are\nused as the main data source for many tasks around software engineering. As\nmodern software systems are evolving into large scale and complex structures,\nlogs have become one type of fast-growing big data in industry. In particular,\nsuch logs often need to be stored for a long time in practice (e.g., a year),\nin order to analyze recurrent problems or track security issues. However,\narchiving logs consumes a large amount of storage space and computing\nresources, which in turn incurs high operational cost. Data compression is\nessential to reduce the cost of log storage. Traditional compression tools\n(e.g., gzip) work well for general texts, but are not tailed for system logs.\nIn this paper, we propose a novel and effective log compression method, namely\nlogzip. Logzip is capable of extracting hidden structures from raw logs via\nfast iterative clustering and further generating coherent intermediate\nrepresentations that allow for more effective compression. We evaluate logzip\non five large log datasets of different system types, with a total of 63.6 GB\nin size. The results show that logzip can save about half of the storage space\non average over traditional compression tools. Meanwhile, the design of logzip\nis highly parallel and only incurs negligible overhead. In addition, we share\nour industrial experience of applying logzip to Huawei's real products.\n