I have decided to “officially” retire Hack MySQL, which is to say that the web site (pages, tools, etc.) will no longer be maintained. The MySQL universe has changed significantly since 2004 when I began writing tools. Similarly, my professional and personal lives have changed, too. I don’t think Hack MySQL is needed or relevant any longer, and I certainly do not have the time to maintain it.
The web site will remain online for perhaps another year (since Media Temple has always provided free hosting for this site–thanks!), then I will remove it or archive it somewhere else.
For further MySQL resources, see Planet MySQL. That aggregate is, imho, the pulse of the MySQL community.
I’ve been thinking a lot about invalid and zero temporal values and how to detect them with MySQL date and time functions because mk-table-checksum has to handle “everything” correctly and efficiently. The requirements are complex because we have to take into account what MySQL allows to be stored verses what it allows to be used in certain operations and functions, how it sorts a mix of real and invalid temporal values for
MAX(), how to detect a temporal value as equivalent to zero, and how different MySQL versions might affect any of the aforementioned.
At base, the four guiding requirements are:
- Detect and discard invalid time, date, and datetime values
- Detect zero-equivalent temporal values
- Do #1 and #2 using only MySQL functions
- Work in MySQL 4.0 and newer
My tests cases for invalid temporal values are:
And my test cases for first real temporal values are:
And there is only one real zero-equivalent temporal value:
So the first requirement is to find a MySQL function that returns
NULL for all those invalid values, and that function is
TO_DAYS with one exception:
mysql> SELECT TO_DAYS('999-01-01 00:00:00'); +-------------------------------+ | TO_DAYS('999-01-01 00:00:00') | +-------------------------------+ | 364878 | +-------------------------------+
That date is only valid if years before 1000 are handled but the MySQL manual says that,
TO_DAYS() is not intended for use with values that precede the advent of the Gregorian calendar (1582)
so we’re already way past the limit of its intended use and, moreover, the supported lower limit of a date or datetime is
1000-01-01, so says the manual. It’s reasonable to not bother with pre-year 1000 dates so I’ll overlook this.
Excepting pre-year 1000 dates,
NULL for all the invalid values. By contrast,
UNIX_TIMESTAMP() returns zero for all the invalid values and
TIME_TO_SEC() returns a mix of
NULL, zero, and values. So the apparent winner for requirement #1 is
Requirement #2 complicates the issue because the time
00:00:00 is valid and zero-equivalent but
NULL for it. We need a hack that handles all the cases, and here it is:
SELECT IF(TIME_FORMAT(?,'%H:%i:%s')=?, TIME_TO_SEC(?), TO_DAYS(?))
That says, basically: if the value is a time then evaluate it with
TIME_TO_SEC(), else evaluate it with
TO_DAYS(). It works so well in fact that it satisfies all four requirements.
00:00:00 evaluates to zero, all the invalid values evaluate to
NULL, and all the valid values evaluate to various non-null values. I have to use
TIME_FORMAT() instead of just
TIME() wasn’t introduced until MySQL v4.1 (fourth requirement).
The hack works because of this (substituting
mysql> SELECT TIME('00:00:00'); +------------------+ | TIME('00:00:00') | +------------------+ | 00:00:00 | +------------------+ mysql> SELECT TIME('00-00-00'); +------------------+ | TIME('00-00-00') | +------------------+ | 00:00:00 | +------------------+ mysql> SELECT TIME('2010-05-26'); +--------------------+ | TIME('2010-05-26') | +--------------------+ | 00:20:10 | +--------------------+ mysql> SELECT TIME('2010-05-26 10:10:10'); +-----------------------------+ | TIME('2010-05-26 10:10:10') | +-----------------------------+ | 10:10:10 | +-----------------------------+
As you can see,
TIME_FORMAT()) returns the exact same value if the given value is a time, otherwise it interprets the value–which is a date or datetime–as a time causing it to return a different value than the given value. Thus we discern time values from date and datetime values and evaluate them separately with
I tested on MySQL v4.0, 4.1, 5.0 and 5.1 and all pass. The only difference is 4.0 verses the others for the pre-year 1000 dates, but I’m ignoring these anyway.
Of course all the preceding could have been accomplished in code by looking at the column type and choosing the correct MySQL function to evaluate the value and check if it’s zero-equivalent, but I was curious to see if it could be done using only MySQL since, after all, it is MySQL that permits these silly, invalid temporals values.
If you know a simpler, more elegant solution that meets the four requirements and passes all the tests, please share!
Generally I avoid going down rabbit holes but today I decided to see how deep a particular testing rabbit hole went. This post is a third in what seems be a continuing series of programming anecdotes. It’s not particularly MySQL-related so you can stop reading here unless you grok code stuff.
Before beginning work on issue 720 I ran the mk-table-checksum test suite to make sure it was in working order. No sense writing new tests and code when the old tests and code aren’t reliable. I actually made one seemingly innocuous change to the test suite in preparation for the issue: I changed the –replicate checksum table from MyISAM to InnoDB.
Surprisingly, the test suite proved unstable. Random tests would fail at random times. Some instability was due to new tests for other issues that I wrote poorly and hadn’t been run a zillion times yet. But other instability was due to switching the checksum table to InnoDB. I knew this because I could switch it back to InnoDB and the tests were ok. Thus began my descent into the rabbit hole.
In particular the test for issue 982 was not finishing or, if it did, it took forever. I discovered that the problem had to do with –lock even though it shouldn’t since the docu says that –replicate and –lock are useless together because the former eliminates locking concerns. Obviously not. So for this I created issue 1027.
Then I turned my attention to test 207_issue_51.t for issue 51 which was opened 2 years ago and closed 1 year ago as WontFix because the problem could not be reproduced. Some might have thought it forgotten amongst the nearly 1,000 issues since its time, but it was found again at the end of this rabbit hole and revivified.
It turns out that issue 51 is, as the original reporter stated (partially), caused by –wait when (this is the part the original reporter didn’t state) the –replicate checksum table is InnoDB because –wait enables –lock which disables AutoCommit so writes to the checksum table may not be committed.
So the rabbit hole lead from issue 720 to issue 982 to issue 1027 to issue 51. Then end results will be a needed bug fix and faster, more stable tests. Sometimes it pays to follow the rabbit.
I’m surprised, and flattered, to see that people still use, write and recommend mysqlsla, mysqlreport and–most surprisingly–mysqlsniffer. In truth, however, I consider all the original Hack MySQL tools as retired. Maatkit consumes the majority of my development time and provides better replacements for all the Hack MySQL tools. The mk tools are better because–most importantly–they’re tested, their code is more robust, and they benefit from the collected knowledge and experience of the community’s top minds (whereas the Hack MySQL tools are brain-children of only my knowledge and experience circa several years ago).
Thus I created a new tools page where I list and briefly profile free, open-source MySQL tools. As the intro paragraph states, MySQL Forge does this, too, but imho the forge is a dense jungle in which it is difficult to discern the useful bits from the less-than-useful bits. My tools page is meant to 1) inform people that the Hack MySQL tools are retired, 2) list replacements for them, and 3) give people new to the MySQL universe a quick, simple list of tools they’ll probably want to become familiar with.
Kind thanks to all who used, wrote about, contributed to and recommend the Hack MySQL tools over the years.
Optimizing Oracle Performance by Cary Millsap and Jeff Holt uses Oracle to make its points, but these points apply also to MySQL. The primary lesson I took away from this book is: all else aside, optimize/fix the user-action that provides the most economic benefit to the company; do this by profiling just that action and optimizing/fixing the most time-consuming events even if they are “idle” or “wait” events.
The authors call the aforementioned approach to performance optimization “Method R”. It’s meant to be deterministic and teachable unlike “Method C”–the conventional method–whereby one uses their best judgment and experience to find the cause(s) of problems and fix them. I agree, and Method R is fundamentally, imho, just the scientific method in practice. Therefore, I like Method R because it puts “science” back into “computer science.”
The book also discusses queueing theory. It’s a whirlwind tour (~60 pages) but the authors provide everything you need to get started, including helper scripts and Excel worksheets. I’m pretty sure that I’ll be working with this theory more in my job; when I do, I’ll begin with what the authors have given me (“stand on the shoulders of giants“).
One criticism/clarification comes to mind: Method R is reactive. Let’s say your MySQL configuration is terrible so you’re not getting the most from your server as you could. Method R may indirectly expose this only if the configuration is the root cause of a slow user-action. So the configuration is only examined and fixed if the user deems their action unacceptably slow. However, users don’t always complain; sometimes they just “live with it” because they don’t care or they don’t think it can be fixed or they’re afraid to complain or it’s always been that way so they’re not even aware that things could be better. Thus, I think a more holistic view of performance optimization requires both a proactive method and a reactive method. Method R is a great reactive method, but someone should be checking stuff even when there doesn’t seem to be a problem. The authors don’t say “Method R is all you ever need to do”–I’m just making a clarification here.
Oracle extended SQL traces are used throughout the book to investigate performance issues. Does MySQL have anything similar? Nothing as cohesive comes to my mind (correct me if I’m wrong). I think we can achieve the same thing via microslow logs, the community PROFILE feature, session status values, and scripts to glue it all together. That’s a lot a of disparate pieces. I’d rather have MySQL extended SQL traces (in a format more easily parsable than Oracle’s).
In summary, Optimizing Oracle Performance is a must-read for any database professional. I think its emphases on providing the business the most “bang for its buck” and the deterministic nature of Method R are timeless and timely lessons for those of us who earn our livings by engaging in the science of computing.