Hack MySQL

Archive for the ‘mysqlsla’ Category

Hack MySQL status update

without comments

I haven’t posted any update in 2 months not because I haven’t been doing anything but rather the opposite: I have been exceptionally busy. As Baron has noted, I work with Percona doing the maintenance and development for Maatkit, of which release 2325 was just published. Consequently, I haven’t been working on my own scripts, though hopefully that will change, and in some ways it must. Here’s what I have in mind for my own scripts:

mysqlsla: I’m going to create a hybrid between this and mk-log-parser. Whereas mysqlsla does some things that mk-log-parser doesn’t, mk-log-parser has a better, more maintainable code base. Furthermore, I’m currently working on other log handling code which will affect some future plans I had for mysqlsla such as grouping results and filtering by timestamp.

mysqlreport: Apparently, it’s “bug-free” because no one has reported a single problem with it in so long that I couldn’t even tell you when it was last released. Regardless, I have plans for this, too, such as better interaction with other scripts by decoupling the code, data and presentation further.

mysqlidxchk: This has needed serious work since its release which wasn’t possible for various “technical” reasons. But I can fix it now and make it better, more helpful.

In general, I’m also moving my entire code base to Google Code, too, since I’ll be re-using some Maatkit common modules in my own scripts. Plus, that will make it easier to track issues and changes for me and you.

Written by Daniel Nichter

September 19th, 2008 at 1:03 pm

mysqlsla v2.00 released

without comments

mysqlsla v2 is finally “done” and released. About 3 months ago, when v1.8 was released, I said it would be coming “soon,” but time just flew by and here we are. Oh well. In any case, the v1 branch is dead to me and v2 is all the rave (at least for me). If you don’t care about the differences and all you want is your default top 10 report from a slow log, for example, then all you need to know is: mysqlsla -lt slow SLOW_LOG

For those interested in what has changed to warrant a new major version number, here’s the briefing of changes/overhauls:

  1. Almost ALL new command line options (–log-type a.k.a. -lt is the most important); see the documentation
  2. Customizable reports. In v1 the report was hard-coded. Now with -rf FILE you can format your own report. See the doc on reports.
  3. FULL filtering and sorting. Somewhere around v1.7 (if I remember correctly) filtering based on things like the connection ID, etc. was introduced. v2 is more abstracted and allows more complex filtering on any “meta-property” available in the log (and sorting by most of them, too). See the doc on filters.
  4. Replays to compact fat text log files into super-compact binary files.
  5. Slightly better query abstraction.
  6. Full support for microslow patched slow logs.
  7. User-defined logs so that, one day, we may begin parsing, filtering, analyzing and sorting custom logs from things like MySQL Proxy.
  8. And in general, what I called the “mysqlsla v2 Library: 6 long, detailed pages documenting just about single, tiny aspect of mysqlsla (plus with the “installer” you get a mysqlsla man page, too).

As this is pretty much an entire code overhaul, please please report bugs, problems, suggestions and whatever. Please also send me your log files: slows, generals, binaries. They help me tremendously because I cannot sit around inventing up all the weird kinds of stuff I’ve seen in other people’s logs.

And as always, thank you all for your support, and thanks to those people who provided feedback, patches and such.

Written by Daniel Nichter

July 12th, 2008 at 1:39 pm

mysqlsla: v1.8 released, v2 coming

without comments

mysqlsla v1.8 is available at:

http://hackmysql.com/scripts/mysqlsla-1.8-DEBUG

I am releasing it publicly without updating the mysqlsla web page or documentation because, instead, I am waiting until I finish mysqlsla v2. After working with v1.8 I realized the code needed a major re-think and overhaul. v2 will reflect this and will be a far superior log hacking and analyzing tool, capable of far more than v1.8 is now.

But for now, 1.8 fixes several good (or bad?) bugs:

  • –only-hosts did not work for general logs
  • Multi-line comments using /* */ caused everything after the first line to be ignored in raw logs
  • “Change user” commands were not handled in general logs
  • CHANGE, DROP and RESET statements were filtered out
  • A certain variant of the Connect command in general logs was not handled
  • A few more SQL capitalization/beautifications were added

If you use mysqlsla, you should really get v1.8 due to these bug fixes. And if you have used mysqlsla before and found it didn’t do what you needed: tell me because v2 open to suggestions. My goal for v2 is to make it the MySQL log hacking tool, capable of parsing, ordering, analyzing, sorting, etc. a MySQL log in any way possible and then to serve as a liaison to other scripts, such as Baron’s mk-query-profiler; e.g. have mysqlsla process and filter a log and then feed SQL statements to mk-query-profiler or any other script. Currently, v1.8 is too stuck in its own ways to help other scripts. Such selfishness shall be expunged.

Written by Daniel Nichter

April 19th, 2008 at 10:53 am

Posted in MySQL,mysqlsla

mysqlsla v1.7a released

without comments

mysqlsla v1.7a has been released. It fixes a strange bug for general logs with connection IDs greater than 10 million.

Written by Daniel Nichter

November 9th, 2007 at 8:36 am

Posted in mysqlsla

mysqlsla v1.7 released

without comments

mysqlsla v1.7 is ready because v1.6 (and v1.5) had a bug: numbers greater than 999 were printed wrong; internally, though, they were still used correctly. Therefore: do not use v1.6.

Also, v1.7 has a –milliseconds option to make time values less than 1 second print as millisecond values. This option is necessary for slow logs from servers patched with the slow query log millisecond patch.

Written by Daniel Nichter

October 30th, 2007 at 3:22 pm

Posted in mysqlsla