[News]: pgBadger 3 released : now with parallel parsing

DALIBO is proud to announce the release of pgBadger v3, the newPostgreSQL log analyzer. pgBadger is built for speed with fully detailed reports from your PostgreSQL log file.
This new release brings significant improvements. All pgBadger usersshould upgrade as soon as possible.
[h=2]pgBadger 3 parallel log parsing[/h]The first versions of pgBadger were bound to only one CPU. ThePostgreSQL log files were scanned sequentially. Analyzing very large logfiles could take several hours.
This limitation is now removed. You can use as many CPU cores as youwant and scan your logs in parallel.
To enable parallel processing, you just have to use the -j N option, Nbeing the number of cores you want to use.
Please note that the parallel mode has a little drawback. With thismethod, some queries may be truncated. If you enable N cores, thenresult may differ in a maximum of N queries per log file.
However, this is a minor issue: parallel mode is interesting if you havemillions of queries to analyze. And if you have millions of queries in alog file, you can afford to loose a few as it’s quite unlikely that thelost queries would have changed the overall results.
However, to avoid this problem, you can use the pgBadger “per-fileparallel mode” to analyze your logs but with lower performance than thestandard parallel mode. To enable this behaviour, you have to use the"-J N" option instead of “-j N”. In per-file mode, the performancesstart being really interesting when there’s hundreds of small log files(e.g. 10MB rotation size limit) and with at least 8 cores.
[h=2]How fast is pgBadger 3?[/h]The goal was to allow pgBadger to use as many cores as specified to haveparallel log parsing. Here are some performance results using pgbadgeron five log files for a total of 9.5 GB:

  • with 1 core => 1h 41m 18s
  • with 2 cores => 50m 25s
  • with 4 cores => 25m 39s
  • with 8 cores => 15m 58s

We feel this performance gain is quite interesting :slight_smile:
[h=2]New binary format[/h]In addition to the classic HTML, TXT and Tsung output formats, pgBadger3 is now able to generate a binary input/output format. This new formatis useful if you only want to store the log statistics and generate theHTML report with graph later.
In a nutshell, the two main activities of pgBadger are parsing andreporting. With this binary format, you can now split those activitiesand run them at different times. For exemple, you can parse your logonce a day, and generate the HTML reports only when needed.
You can also combine several binary files to . For Instance, you maycreate a binary report every week and aggregate the last 4 week reportsto build a monthly report in HTML.
This new binary format is also compatible with other tools such aspgShark https://github.com/dalibo/pgshark/.
[h=2]More stats, more pie charts![/h]This major release also has additional features:

  • New pie graph to show number of autovacuum per table and number oftuples removed by autovacuum per table
  • No more distinction between log_duration, log_statement andlog_min_duration_statement format
  • New report of tuples/pages removed in report of VACUUM by table
  • New VACUUM and ANALYZE hourly reports and graphs

… and many bugfixes .
For the complete list of changes, please checkout the release note onhttps://github.com/dalibo/pgbadger/blob/master/ChangeLog
[h=2]Deprecated options[/h]WARNING : for the sake of simplicity, the’’–enable-log_min_duration’’ and ‘’–enable-log_duration’’ command lineoptions have been removed. pgbadger is now parsing any log_duration,log_statement and log_min_duration_statement lines without distinctionand adapt the reports following those lines.
If you are running pgBadger using cron, please take care: if one oftheses options appears in the command line, pgbadger will refuse to start.
[h=2]Links & Credits[/h]DALIBO would like to thank the developers who submitted patches and theusers who reported bugs and feature requests, especially Matt Romaine,Luke Cyca, Kevin Brannen, Adam Schroder, pilat66, Euler Taveira deOliveira, stuntmunkee, pierrestroh, Vipul, Dirk-Jan Bulsink and VincentLaborie.
pgBadger is an open project. Any contribution to build a better tool iswelcome. You just have to send your ideas, features requests or patchesusing the GitHub tools or directly on our mailing list.
Links :

[HR][/HR]About pgBadger :
pgBagder is a new generation log analyzer for PostgreSQL, created byGilles Darold, also author of ora2pg migration tool. pgBadger is a fastand easy tool to analyze your SQL traffic and create HTML5 reports withdynamics graphs. pgBadger is the perfect tool to understand the behaviorof your PostgreSQL server and identify which SQL queries need to beoptimized.
Docs, Download & Demo at http://dalibo.github.com/pgbadger/
[HR][/HR]About DALIBO :
DALIBO is the leading PostgreSQL company in France, providing support,trainings and consulting to its customers since 2005. The companycontributes to the PostgreSQL community in various ways, including :code, articles, translations, free conferences and workshops
Check out DALIBO’s open source projects at http://dalibo.github.com