Hack MySQL News

mysqlsla v2 is finally “done” and released. About 3 months ago, when v1.8 was released, I said it would be coming “soon,” but time just flew by and here we are. Oh well. In any case, the v1 branch is dead to me and v2 is all the rave (at least for me). If you don’t care about the differences and all you want is your default top 10 report from a slow log, for example, then all you need to know is: mysqlsla -lt slow SLOW_LOG

For those interested in what has changed to warrant a new major version number, here’s the briefing of changes/overhauls:

1. Almost ALL new command line options (–log-type a.k.a. -lt is the most important); see the documentation
2. Customizable reports. In v1 the report was hard-coded. Now with -rf FILE you can format your own report. See the doc on reports.
3. FULL filtering and sorting. Somewhere around v1.7 (if I remember correctly) filtering based on things like the connection ID, etc. was introduced. v2 is more abstracted and allows more complex filtering on any “meta-property” available in the log (and sorting by most of them, too). See the doc on filters.
4. Replays to compact fat text log files into super-compact binary files.
5. Slightly better query abstraction.
6. Full support for microslow patched slow logs.
7. User-defined logs so that, one day, we may begin parsing, filtering, analyzing and sorting custom logs from things like MySQL Proxy.
8. And in general, what I called the “mysqlsla v2 Library: 6 long, detailed pages documenting just about single, tiny aspect of mysqlsla (plus with the “installer” you get a mysqlsla man page, too).

As this is pretty much an entire code overhaul, please please report bugs, problems, suggestions and whatever. Please also send me your log files: slows, generals, binaries. They help me tremendously because I cannot sit around inventing up all the weird kinds of stuff I’ve seen in other people’s logs.

And as always, thank you all for your support, and thanks to those people who provided feedback, patches and such.

mysqlsla v1.8 is available at:
http://hackmysql.com/scripts/mysqlsla-1.8-DEBUG
I am releasing it publicly without updating the mysqlsla web page or documentation because, instead, I am waiting until I finish mysqlsla v2. After working with v1.8 I realized the code needed a major re-think and overhaul. v2 will reflect this and will be a far superior log hacking and analyzing tool, capable of far more than v1.8 is now.

But for now, 1.8 fixes several good (or bad?) bugs:

• –only-hosts did not work for general logs
• Multi-line comments using /* */ caused everything after the first line to be ignored in raw logs
• “Change user” commands were not handled in general logs
• CHANGE, DROP and RESET statements were filtered out
• A certain variant of the Connect command in general logs was not handled
• A few more SQL capitalization/beautifications were added

If you use mysqlsla, you should really get v1.8 due to these bug fixes. And if you have used mysqlsla before and found it didn’t do what you needed: tell me because v2 open to suggestions. My goal for v2 is to make it the MySQL log hacking tool, capable of parsing, ordering, analyzing, sorting, etc. a MySQL log in any way possible and then to serve as a liaison to other scripts, such as Baron’s mk-query-profiler; e.g. have mysqlsla process and filter a log and then feed SQL statements to mk-query-profiler or any other script. Currently, v1.8 is too stuck in its own ways to help other scripts. Such selfishness shall be expunged.

mysqlreport v3.5 has been released which has the following fixes, changes, updates:

• Fixed bug: incorrect checking of MySQL version caused mysqlreport to ignore InnoDB status values in some cases. Simplified version from 3 integers (major, minor, patch) to 1 (50024 = 5.0.24, 60001 = 6.0.1, etc.) for easier, more accurate version checking all around.
• Fixed potential bug: read_relative_infiles might have incorrectly divided infiles with multiple status value sets.
• Fixed bug: if wait_timeout was greater than –relative, mysqlreport would lose its connection while sleeping. Now the connection is closed and reopened for each live relative report collection.
• Infiles can now be given SHOW VARIABLES output (in addition to the old method of inserting values manually like: key_buffer_size=128M)
• Slow query time is beautified for microsecond, millisecond, and second resolution (format_u_time):
  0.000000 - 0.000999 = 0 - 999 µ
  0.001000 - 0.999999 = 1 ms - 999.999 ms
  1.000000 - n.nnnnnn = 1 s - n.nnnnn s
• Added a lot more debugging output.

mysqlreport v3.4a is ready which, like the recently released v3.4, fixes a bug in the relative reports. This time, it was the InnoDB reports which were not correct when made relative. Changes in v3.4a are:

• Fixed bug: InnoDB report values were wrong in relative reports because 13 InnoDB status values needed to be excluded from being made relative. (thanks Debbie)
• Changed InnoDB Buffer Pool Read ratio to Read hit %

The Guide to Understanding mysqlreport has been accordingly updated, too.

mysqlreport v3.4 is ready which primarily fixes a bug in v3.3 concerning infiles for relative reports.

Changes in v3.4 are:

• Fixed bug: –relative infiles wouldn’t work unless the SHOW STATUS values were prefixed with a line matching: /Variable_name[\s|]+Value/. Now mysqlreport looks for Aborted_clients which should always be present. (thanks Debbie)
• Fixed formatting problem: Created Temp Table Size was too small for >99M
• Removed redundant %Total: column label from Slow line
• Added more debug info

mysqlreport v3.3 is ready. Changes:

• –all is now the default option and all the sub-report options like –dms, –sas, etc. have been removed. All reports that can be made are made automatically.
• Slow line now displays long_query_time and log_slow_queries
• Created Temp Table line now displays tmp_table_size
• The mysqlreport tgz and zip now extract into their own directory

The mysqlreport Documentation and The Guide To Understanding mysqlreport have been updated to reflect these changes.

mysqlreport v3.2 has been released. Mark Leith’s post, Aggregating SHOW STATUS Output, reminded me that a user had asked me a few months ago to do this for mysqlreport. I forgot until Mark’s post.

I have finished implementing this feature with mysqlreport. The new version has three new command line options: –relative (-r), –report-count (-c), and –detach. All of these are explained in the doc. This feature required substantial alterations to the script, so please tell me if anything doesn’t work.

Yesterday I released mysqlreport v3.1. No large changes:

• Added support for table_open_cache system var (MySQL 5.1.3+).
• Removed default assignments for $mycnf{’host’}, $mycnf{’port’}, and $mycnf{’socket’}.
• Connecting to MySQL now tries socket if one is given and actually a socket, else TCP/IP host if one is given, else generic “host=localhost” which relies on the MySQL client lib to do something useful.
• Updated help for clarity.
• Fixed bug: spelled “fsync” wrong.

I also updated The Guide To Understanding mysqlreport to include InnoDB reports and the mysqlreport documentation.

It’s strange what articles on one’s site become “popular.” I wrote Non-technical Guide to Isolating Slow MySQL Queries some time ago when I was an internal system admin at a data center. I wasn’t “customer-facing” but I still got requests from customers who wanted me to look at their MySQL server. Since that wasn’t my primary responsibility, I had to enlist their help to save me time. Hence, I wrote this article to help not-technically-inclined customers through the first few steps.

My webstats have shown this page to be hit more often than I would have expected. Consequently, I felt that I should update it, so I did.

mysqlsla (MySQL Statement Log Analyzer) v1.5 is finally done. Some people may recall a semi-release of v1.4 a few months ago, but that version wasn’t documented or otherwise mentioned at hackmysql.com.

Now, the real next release of mysqlsla is finished, and with documentation too! (However, I have removed the guide for now; it still needs to be completely rewritten). From the old v1.3 to the present, a lot has changed due to a lot of feedback, suggestions, problems, and bugs reported from users. Essentially, the script has been completely written, but here is a brief list of the changes:

• COMPLETE redesign from v1.3: more simple to use and more informative
• Log types are now mutually exclusive (but multiple logs of same type can still be combined)
• Captures USE statements for querys when possible (to make EXPLAIN work)
• –time-all-queries only works with general and slow logs
• Nothing is filtered by default
• Auto-filter -*,+SELECT,+USE enabled for –time-all-queries and –time-each-query unless overriden by –filter
• Handles compound statements in slow logs
• Mimicking of mysqldumpslow removed; slow log values now “automatic” when parsing a slow log
• Removed options: –order, –correlate, –hide, –rows-read, –mysqldumpslow, –frequency (now called Count)
• Added options: –debug, –flat, –examples, –sort, –databases, –distribution, –mp, –no-report, –np, –nth-percent
• Improved log parsing routines
• Handles DBI/MySQL errors cleanly instead of dumping them to stderr
• Greatly minimized memory usage (can now parse multi-gig logs without running out of memory)

Thank you to all those who helped to improve this script. Please, keep submitting suggestions, bugs, problems, etc.