At base, the four guiding requirements are:

1. Detect and discard invalid time, date, and datetime values
2. Detect zero-equivalent temporal values
3. Do #1 and #2 using only MySQL functions
4. Work in MySQL 4.0 and newer

My tests cases for invalid temporal values are:

• 00:00:60
• 00:60:00
• 999-00-00
• 999-01-01
• 0000-00-00
• 2009-00-00
• 2009-13-00
• 999-00-00 00:00:00
• 999-01-01 00:00:00
• 0000-00-00 00:00:00
• 1000-00-00 00:00:00
• 2009-00-00 00:00:00
• 2009-13-00 00:00:00
• 2009-05-26 00:00:60
• 2009-05-26 00:60:00
• 2009-05-26 24:00:00

And my test cases for first real temporal values are:

• 00:00:00
• 00:00:01
• 1000-01-01
• 2009-01-01
• 1000-01-01 00:00:00
• 2009-01-01 00:00:00

And there is only one real zero-equivalent temporal value: 00:00:00.

So the first requirement is to find a MySQL function that returns NULL for all those invalid values, and that function is TO_DAYS with one exception:

mysql> SELECT TO_DAYS('999-01-01 00:00:00');
+-------------------------------+
| TO_DAYS('999-01-01 00:00:00') |
+-------------------------------+
|                        364878 |
+-------------------------------+

That date is only valid if years before 1000 are handled but the MySQL manual says that,

TO_DAYS() is not intended for use with values that precede the advent of the Gregorian calendar (1582)

so we’re already way past the limit of its intended use and, moreover, the supported lower limit of a date or datetime is 1000-01-01, so says the manual. It’s reasonable to not bother with pre-year 1000 dates so I’ll overlook this.

Excepting pre-year 1000 dates, TO_DAYS() returns NULL for all the invalid values. By contrast, UNIX_TIMESTAMP() returns zero for all the invalid values and TIME_TO_SEC() returns a mix of NULL, zero, and values. So the apparent winner for requirement #1 is TO_DAYS(), but…

Requirement #2 complicates the issue because the time 00:00:00 is valid and zero-equivalent but TO_DAYS() returns NULL for it. We need a hack that handles all the cases, and here it is:

SELECT IF(TIME_FORMAT(?,'%H:%i:%s')=?, TIME_TO_SEC(?), TO_DAYS(?))

That says, basically: if the value is a time then evaluate it with TIME_TO_SEC(), else evaluate it with TO_DAYS(). It works so well in fact that it satisfies all four requirements. 00:00:00 evaluates to zero, all the invalid values evaluate to NULL, and all the valid values evaluate to various non-null values. I have to use TIME_FORMAT() instead of just TIME() because TIME() wasn’t introduced until MySQL v4.1 (fourth requirement).

The hack works because of this (substituting TIME() for TIME_FORMAT()):

mysql> SELECT TIME('00:00:00');
+------------------+
| TIME('00:00:00') |
+------------------+
| 00:00:00         |
+------------------+

mysql> SELECT TIME('00-00-00');
+------------------+
| TIME('00-00-00') |
+------------------+
| 00:00:00         |
+------------------+

mysql> SELECT TIME('2010-05-26');
+--------------------+
| TIME('2010-05-26') |
+--------------------+
| 00:20:10           |
+--------------------+

mysql> SELECT TIME('2010-05-26 10:10:10');
+-----------------------------+
| TIME('2010-05-26 10:10:10') |
+-----------------------------+
| 10:10:10                    |
+-----------------------------+

As you can see, TIME() (or TIME_FORMAT()) returns the exact same value if the given value is a time, otherwise it interprets the value–which is a date or datetime–as a time causing it to return a different value than the given value. Thus we discern time values from date and datetime values and evaluate them separately with TIME_TO_SEC().

I tested on MySQL v4.0, 4.1, 5.0 and 5.1 and all pass. The only difference is 4.0 verses the others for the pre-year 1000 dates, but I’m ignoring these anyway.

Of course all the preceding could have been accomplished in code by looking at the column type and choosing the correct MySQL function to evaluate the value and check if it’s zero-equivalent, but I was curious to see if it could be done using only MySQL since, after all, it is MySQL that permits these silly, invalid temporals values.

If you know a simpler, more elegant solution that meets the four requirements and passes all the tests, please share!

Thus I created a new tools page where I list and briefly profile free, open-source MySQL tools. As the intro paragraph states, MySQL Forge does this, too, but imho the forge is a dense jungle in which it is difficult to discern the useful bits from the less-than-useful bits. My tools page is meant to 1) inform people that the Hack MySQL tools are retired, 2) list replacements for them, and 3) give people new to the MySQL universe a quick, simple list of tools they’ll probably want to become familiar with.

Kind thanks to all who used, wrote about, contributed to and recommend the Hack MySQL tools over the years.

For those interested in what has changed to warrant a new major version number, here’s the briefing of changes/overhauls:

1. Almost ALL new command line options (–log-type a.k.a. -lt is the most important); see the documentation
2. Customizable reports. In v1 the report was hard-coded. Now with -rf FILE you can format your own report. See the doc on reports.
3. FULL filtering and sorting. Somewhere around v1.7 (if I remember correctly) filtering based on things like the connection ID, etc. was introduced. v2 is more abstracted and allows more complex filtering on any “meta-property” available in the log (and sorting by most of them, too). See the doc on filters.
4. Replays to compact fat text log files into super-compact binary files.
5. Slightly better query abstraction.
6. Full support for microslow patched slow logs.
7. User-defined logs so that, one day, we may begin parsing, filtering, analyzing and sorting custom logs from things like MySQL Proxy.
8. And in general, what I called the “mysqlsla v2 Library: 6 long, detailed pages documenting just about single, tiny aspect of mysqlsla (plus with the “installer” you get a mysqlsla man page, too).

As this is pretty much an entire code overhaul, please please report bugs, problems, suggestions and whatever. Please also send me your log files: slows, generals, binaries. They help me tremendously because I cannot sit around inventing up all the weird kinds of stuff I’ve seen in other people’s logs.

And as always, thank you all for your support, and thanks to those people who provided feedback, patches and such.

But for now, 1.8 fixes several good (or bad?) bugs:

• –only-hosts did not work for general logs
• Multi-line comments using /* */ caused everything after the first line to be ignored in raw logs
• “Change user” commands were not handled in general logs
• CHANGE, DROP and RESET statements were filtered out
• A certain variant of the Connect command in general logs was not handled
• A few more SQL capitalization/beautifications were added

If you use mysqlsla, you should really get v1.8 due to these bug fixes. And if you have used mysqlsla before and found it didn’t do what you needed: tell me because v2 open to suggestions. My goal for v2 is to make it the MySQL log hacking tool, capable of parsing, ordering, analyzing, sorting, etc. a MySQL log in any way possible and then to serve as a liaison to other scripts, such as Baron’s mk-query-profiler; e.g. have mysqlsla process and filter a log and then feed SQL statements to mk-query-profiler or any other script. Currently, v1.8 is too stuck in its own ways to help other scripts. Such selfishness shall be expunged.

• Fixed bug: incorrect checking of MySQL version caused mysqlreport to ignore InnoDB status values in some cases. Simplified version from 3 integers (major, minor, patch) to 1 (50024 = 5.0.24, 60001 = 6.0.1, etc.) for easier, more accurate version checking all around.
• Fixed potential bug: read_relative_infiles might have incorrectly divided infiles with multiple status value sets.
• Fixed bug: if wait_timeout was greater than –relative, mysqlreport would lose its connection while sleeping. Now the connection is closed and reopened for each live relative report collection.
• Infiles can now be given SHOW VARIABLES output (in addition to the old method of inserting values manually like: key_buffer_size=128M)
• Slow query time is beautified for microsecond, millisecond, and second resolution (format_u_time):
  0.000000 – 0.000999 = 0 – 999 µ
  0.001000 – 0.999999 = 1 ms – 999.999 ms
  1.000000 – n.nnnnnn = 1 s – n.nnnnn s
• Added a lot more debugging output.

• Fixed bug: InnoDB report values were wrong in relative reports because 13 InnoDB status values needed to be excluded from being made relative. (thanks Debbie)
• Changed InnoDB Buffer Pool Read ratio to Read hit %

The Guide to Understanding mysqlreport has been accordingly updated, too.

Changes in v3.4 are:

• Fixed bug: –relative infiles wouldn’t work unless the SHOW STATUS values were prefixed with a line matching: /Variable_name[\s|]+Value/. Now mysqlreport looks for Aborted_clients which should always be present. (thanks Debbie)
• Fixed formatting problem: Created Temp Table Size was too small for >99M
• Removed redundant %Total: column label from Slow line
• Added more debug info

• –all is now the default option and all the sub-report options like –dms, –sas, etc. have been removed. All reports that can be made are made automatically.
• Slow line now displays long_query_time and log_slow_queries
• Created Temp Table line now displays tmp_table_size
• The mysqlreport tgz and zip now extract into their own directory

The mysqlreport Documentation and The Guide To Understanding mysqlreport have been updated to reflect these changes.

]]> http://hackmysql.com/blog/2008/01/09/mysqlreport-v33-released/feed/ 0 http://hackmysql.com/blog/2007/05/26/mysqlreport-v32-released/ http://hackmysql.com/blog/2007/05/26/mysqlreport-v32-released/#comments Sat, 26 May 2007 21:41:50 +0000 Daniel Nichter http://hackmysql.com/wp/2007/05/26/mysqlreport-v32-released/ mysqlreport v3.2 has been released. Mark Leith’s post, Aggregating SHOW STATUS Output, reminded me that a user had asked me a few months ago to do this for mysqlreport. I forgot until Mark’s post.

I have finished implementing this feature with mysqlreport. The new version has three new command line options: –relative (-r), –report-count (-c), and –detach. All of these are explained in the doc. This feature required substantial alterations to the script, so please tell me if anything doesn’t work.

]]> http://hackmysql.com/blog/2007/05/26/mysqlreport-v32-released/feed/ 0