Hack MySQL Blog

I’m speaking at Percona Live MySQL Conference & Expo 2012. My two talk are: Getting Started with Drizzle 7.1 and Verifying MySQL Replication Safely With pt-table-checksum 2.0. No, there’s no relationship between those topics; they’re just things I know well.

I’ve been stalking Drizzle for many years. When it went GA last year, I began hacking Drizzle, focusing on plugins which give it nearly all its functionality. Recently, I helped overhaul the configuration, administration, and plugin sections of the Drizzle docs. I’m also frequently poking around the plugins’ source code. Consequently, I know a lot about making Drizzle work. My talk with transfer the bulk of the best of that knowledge to you so that you can return to your place of work/hobby/world domination and start using Drizzle 7.1 yourself without having to resolve some of the mysteries I had to resolve by reading the source code.

As for pt-table-checksum 2.0, part of Percona Toolkit, it’s a complete re-write of the venerable pt-table-checksum 1.0 which worked very well for years but required some fine-tuning. Well, in certain cases it didn’t work as well, which required more fine-tuning. So Baron redesigned the entire tool (with help and feedback from a lot of people at Percona), and I programmed it. It works wonderfully, and most of the time you can “just run it” and it will Just Do The Right Thing, but you will nonetheless benefit if you come to this talk and poke and prod its internal with me.

If you’ve never been to this conference before (formerly it was just the “MySQL Conference & Expo”, but it’s always been, afaik, at this time and place), you should really come because it’s very enlightening. It’s the Super Bowl/World Cup/Wimbledon/etc. of MySQL conferences in my humble opinion.

Although I no longer maintain hackmysql.com, I still actively develop MySQL tools. An article I wrote, 10 essential MySQL tools for admin, was published today. I hope no one’s feelings are hurt if their tool isn’t in the list, but it was rather difficult to compile the list given that so many tools are either not actively developed, not tested, or not well documented. Given lag time between writing and publishing, I was not able to write about Yoshinori’s MHA or newer tools. And given length constraints, I was not able to write about more tools. In any case, the world of MySQL tools is alive and well.

Last week I wrote about my experience compiling Drizzle 7 on Mac OS X 10.6. Then David Shrewsbury informed me of his nearly identical blog post: Installing Drizzle from source on OS X. Once Drizzle 7 was running on my box, I immediately looked to see what plugins where available because Drizzle uses a lot of plugins and they are one of its notable differences from MySQL. In my humble opinion, Drizzle’s plugins will primarily influence how database professionals evaluate and decide whether or not to use Drizzle because so many of Drizzle’s features are plugins. Therefore, let’s look briefly at some the plugins included with Drizzle 7.

The plugin directory of the Drizzle 7 tarball lists 79 plugins. Each plugin has a plugin.ini file that tells Drizzle its name (title) and description (among other things):

This list demonstrates to what extent features of Drizzle are outside its kernel and provided instead by plugins. Even SLEEP() is a plugin! By comparison, many of these features are built-in with MySQL, like authentication and replication. It is not my intent in this blog post to discuss the pros and cons of this “(almost) everything is a plugin” design decision (I like it actually), so let’s instead play with a plugin to see how it treats us; let’s enable auth_file because I recently read Eric Day’s blog post Writing Authentication Plugins for Drizzle.

Enabling a Plugin: auth_file

For my first attempt, I simply start drizzled with the option --plugin-add=auth_file. That does not work:

$ cat var/drizzled.err
Could not load auth file: Could not open users file: /opt/drizzle7/etc/drizzle.users

Plugin 'auth_file' init function returned error.

Aborting

That is, at least, a nice error message: I know what failed and why. I could probably just make that file, but since I don’t know what format it expects my inclination is to consult the Drizzle reference manual because I’m habituated to consulting the MySQL reference manual whenever I have questions. Sadly, however, the Drizzle plugin documentation is severely limited and does not list the plugin, nor does the Drizzle authentication documentation page. Eric’s blog post mentioned earlier describes the file format, but I’m going to pretend to be a regular user and consult the regular sources of information. If the database’s docu doesn’t say how to enable/configure the plugin, my next guess is drizzled --help, but that doesn’t list auth_file. Then I try drizzled --plugin-add=auth_file --help and that does list:

Options used by auth_file:
  --auth-file.users arg (=/opt/drizzle7/etc/drizzle.users)
                                        File to load for usernames and
                                        passwords

It’s nice that --help is dynamic, but it still doesn’t tell me what file format auth_file expects. If I could start drizzled with the plugin, then perhaps I could do SELECT * FROM data_dictionary.plugins WHERE plugin_name='auth_file' but I can’t do that either. So as a naïve user, I’m out of luck unless I guess.

For the sake of making this example work, I’ll stop being the naïve user and instead consult the list of plugins above that says “Authenticate against a list of username:password entries in a plain text file.” and Eric’s blog post mentioned earlier which says the same things. So I create /opt/drizzle7/etc/drizzle.users with one line foo:bar and then attempt to start drizzled with the plugin again:

$ drizzle
Welcome to the Drizzle client..  Commands end with ; or \g.
Your Drizzle connection id is 1
Connection protocol: mysql
Server version: 2011.03.13 Source distribution (drizzle)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

drizzle> SELECT * FROM data_dictionary.plugins WHERE plugin_name='auth_file';
+-------------+----------------+-----------+-------------+
| PLUGIN_NAME | PLUGIN_TYPE    | IS_ACTIVE | MODULE_NAME |
+-------------+----------------+-----------+-------------+
| auth_file   | Authentication | YES       | auth_file   |
+-------------+----------------+-----------+-------------+

Success! But wait… I still connected with my account username (daniel) and no password. Why? Because another authentication module is enabled:

drizzle> SELECT * FROM data_dictionary.plugins WHERE plugin_name LIKE 'auth%';
+-------------+----------------+-----------+--------------------------+
| PLUGIN_NAME | PLUGIN_TYPE    | IS_ACTIVE | MODULE_NAME              |
+-------------+----------------+-----------+--------------------------+
| auth_all    | Authentication | YES       | Allow-All-Authentication |
| auth_file   | Authentication | YES       | auth_file                |
+-------------+----------------+-----------+--------------------------+

It seems I need to also start drizzled with --plugin-remove=auth_all. This produces the expected result:

$ drizzle
ERROR 1045 (28000): Access denied for user 'daniel' (using password: NO)

$ drizzle --user foo --password=bar
Welcome to the Drizzle client..  Commands end with ; or \g.
Your Drizzle connection id is 5
Connection protocol: mysql
Server version: 2011.03.13 Source distribution (drizzle)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

drizzle>

Of course not all plugins have to be configured; some are self-evident, like sleep… or are they?

drizzle> SELECT SLEEP(7);
+----------+
| SLEEP(7) |
+----------+
|        0 |
+----------+
1 row in set (0.000624 sec)

Notice the last line: the query did not sleep for 7 seconds; it returned immediately. It seems that the sleep plugin is broken or I’m not using it correctly. SLEEP(1.0) sleeps for 1.0 seconds, but SLEEP(7.0) returns immediately. SLEEP() is probably not a mission-critical feature of the database, but its failure in Drizzle highlights the need for more testing and documentation of plugins.

A final word on this topic before we move on: without documentation I don’t know how to configure real authentication and security. How does one, for example, grants certain users certain access to certain databases or tables? Are there column-level grants? What privileges are needed to enable or disable dynamic features? In brief: Drizzle authentication/security is either under-developped or under-documented. I hope it’s just the latter because anyone can write docs, fewer can write good authentication modules.

Active Plugins and Their Types

Let’s continue to looking at what information about plugins is exposed inside Drizzle by executing SELECT * FROM data_dictionary.plugins ORDER BY plugin_name:

+----------------------+-----------------------+-----------+------------------------------+
| PLUGIN_NAME          | PLUGIN_TYPE           | IS_ACTIVE | MODULE_NAME                  |
+----------------------+-----------------------+-----------+------------------------------+
| abs                  | Function              | YES       | Math Functions               |
| acos                 | Function              | YES       | Math Functions               |
| ascii                | Function              | YES       | ascii                        |
| asin                 | Function              | YES       | Math Functions               |
| auth_file            | Authentication        | YES       | auth_file                    |
| console              | Listen                | YES       | console                      |
| default_replicator   | TransactionReplicator | YES       | default_replicator           |
| Error_message_stderr | ErrorMessage          | YES       | errmsg_stderr                |

That is an abbreviated list; the complete list is 169 plugins of various types. Two questions result from this: why are there more plugins here than listed above, and what are the various plugin types? I don’t know the answer to the first question, but I’m sure a Drizzle developer does? My semi-educated guess is that Drizzle also has modules which provide several functions/plugins. Don’t quote me on that though.

The second question can be answered easily, and it gives us an insight into how plugins are organized and what aspects of Drizzle they implement:

drizzle> SELECT plugin_type, COUNT(plugin_name) AS n_plugins
      -> FROM data_dictionary.plugins
      -> GROUP BY plugin_type ORDER BY n_plugins DESC;
+-----------------------+-----------+
| plugin_type           | n_plugins |
+-----------------------+-----------+
| TableFunction         |        79 |
| Function              |        74 |
| StorageEngine         |         5 |
| Listen                |         4 |
| Authentication        |         1 |
| Catalog               |         1 |
| Daemon                |         1 |
| ErrorMessage          |         1 |
| Logging               |         1 |
| Scheduler             |         1 |
| TransactionReplicator |         1 |
+-----------------------+-----------+

All the TableFunction plugins are related to the DATA_DICTIONARY and the INFORMATION_SCHEMA, so they’re probably something that end users will just expect to work and wouldn’t seek to replace or tweak. The Function plugins cover a range of functions from HEX() and VERSION() to KILL(). These are probably also aspects of the database that users will just expect to work, but I like that we have the ability to plug in a new RAND(), for example. Similarly, plugin types Catalog, Daemon, ErrorMessage and Scheduler are core aspects of the database that few users will want to tweak. On the other hand, the remaining types allow for a lot of innovation and development.

The Listen plugins relate to protocols that Drizzle can speak. My build includes the console “protocol”, MySQL and Drizzle TCP/IP protocols, and the standard Unix socket protocol. In theory, Drizzle could speak directly to other database technologies, like memcached, with a plugin to enable/translate the foreign “language”. This is nice because several database technologies are often used in conjunction.

The Authentication plugins will probably also become very important and popular because authentication/security is a central aspect of a database. Many big corporations and governments have complex security requirements and there a lot of security/authentication methods. From time to time I hear people criticize MySQL’s authentication as too rudimentary. This is no longer a problem in Drizzle.

StorageEngine plugins will no doubt play a central role in Drizzle’s future just as it does in MySQL’s present. I won’t say any more about this because, for one, it’s already well understood (i.e. the benefits of pluggable storage engines), and, two, Drizzle is new and there’s a lot of storage engines so it remains to be seen which storage engines become the most influential (like how MyISAM was influential in MySQL’s early history, then gave way to InnoDB).

Another important, pluggable aspect of Drizzle that is not well represented by this list but hinted to by TransactionReplicator is replication. Unlike MySQL, replication in Drizzle is pluggable. Like storage engines, pluggable replication opens the door new to innovations like RabbitReplication, but also like storage engines, Drizzle is new so it remains to be seen which system of replication will become the most influential. Or, perhaps, no one system will dominate and users will need to choose the best replication plugin for their particular environment.

Lastly, the plugin type that interests me since I’ve written a couple of log parsers and I’m the primary coder for mk-query-digest: Logging. I’ll be the one to say it: original/order versions of the MySQL statement logs (i.e. the general log and the slow log) sucked. The general log is a nightmare to parse and it’s really nothing more than queries and timestamps–no “actionable” information. The slow log is better since it has information like query execution time, etc., but it wasn’t really until Percona extensively extended the slow log (both in configurability and output) that the slow log allowed us to do a lot of valuable analyses of query events inside MySQL. Therefore I’m very happy that logging in Drizzle is pluggable so that logging can be done well from the start. Let’s see what logging awesomeness Drizzle has for us.

Query Logging with logging_query

Sadly the Drizzle plugin documentation doesn’t yet mention any logging plugins. The Drizzle logging documentation lists Unix Syslog and File logging, but at the time of writing, both of those pages are blank. From the list of plugins at the start of this blog post, we see logging_query which, by its description, seems to act like a general log since it logs “all” queries, not just slow queries. It’s our only option so we’ll try to enable and use it.

I first tried to simply enable the plugin by starting drizzled with --plugin-add. The server started but there was no logging to be found anywhere, and the plugin did not even show as enabled in the DATA_DICTIONARY. A little confused, I consulted:

$ drizzled --plugin-add=logging_query --help
...
Options used by logging-query:
  --logging-query.enable                Enable logging to CSV file
  --logging-query.filename arg          File to log to
  --logging-query.pcre arg              PCRE to match the query against
  --logging-query.threshold-slow arg (=0)
                                        Threshold for logging slow queries, in
                                        microseconds
  --logging-query.threshold-big-resultset arg (=0)
                                        Threshold for logging big queries, for
                                        rows returned
  --logging-query.threshold-big-examined arg (=0)
                                        Threshold for logging big queries, for
                                        rows examined

Perhaps I must “doubly” enable the plugin: add it with --plugin-add and turn it on with --logging-query.enable. I try that but still no query log can be found anywhere and Drizzle throws no errors. So, based on reading Ronald’d old blog post about this very topic, Drizzle query logging, I add a third option: --logging-query.filename=$BASEDIR/var/query-log.csv. That makes the plugin come alive: the plugin is listed as enabled in the DATA_DICTIONARY and var/query-log.csv exists:

$ cat var/query-log.csv
1303070901062988,1,1,"","select @@version_comment limit 1","Query",1077,243,243,1,0,0,0,1,"beatrice.local"
1303070902100576,1,2,"","select * from data_dictionary.plugins where plugin_name='logging_query'","Query",1038665,2498,575,2,170,0,0,1,"beatrice.local"

I won’t repeat what Ronald already said in his aforementioned blog post, but he makes the important note: “Unfortunately the log does not yet provide a header.” Yes, that’s very unfortunate because there is no official documentation so the current query log format is essentially useless to anyone but hackers like Ronald and people like me who care enough to search out the answers that I need.

Conclusion

The Drizzle 7 GA tarball comes with roughly 70 usable plugins in its plugin/ directory. More plugins, over 100, are shown in the DATA_DICTIONARY.plugins table. These plugins represent the majority of features provided by Drizzle 7, from query logging, to user authentication, to storage engines, to replication. At present, many of the provided plugins are either bare-bones implementations of features or not sufficiently documented to be usable by a normal user.

It’s easy to criticize, but I make these observations to help bring attention to areas that need work because I want Drizzle to develop and mature quickly. The pluggable storage engine feature of MySQL helped spur the development of storage engines resulting in innovation and competition that still, today, provides users great, free choices. The pluggable design has been taken to the extreme in Drizzle, extending far beyond just storage engines, to nearly everything the database server can do. I hope this spurs similar advancements in features that are lacking in Drizzle, as well as cool new features no one has imagined yet.

From this brief look at the plugins provided with Drizzle 7, I suggest the follow two things to the Drizzle developers:

1. Document everything about Drizzle at docs.drizzle.org. Now that Drizzle has a GA release, I think good, complete documentation is the single most important task that will allow average users to actually use Drizzle, thereby giving Drizzle traction in the market, thereby creating the need for further development based on user feedback and bugs.
2. Develop a standard for how plugins should behave. For example: auth_file has a nice error message, but logging_query does nothing silently until you enable it “just right”.

Thanks to those who did the hard work of making Drizzle pluggable and writing the first generation of plugins.

I’ve been thinking a lot about invalid and zero temporal values and how to detect them with MySQL date and time functions because mk-table-checksum has to handle “everything” correctly and efficiently. The requirements are complex because we have to take into account what MySQL allows to be stored verses what it allows to be used in certain operations and functions, how it sorts a mix of real and invalid temporal values for MIN() and MAX(), how to detect a temporal value as equivalent to zero, and how different MySQL versions might affect any of the aforementioned.

At base, the four guiding requirements are:

1. Detect and discard invalid time, date, and datetime values
2. Detect zero-equivalent temporal values
3. Do #1 and #2 using only MySQL functions
4. Work in MySQL 4.0 and newer

My tests cases for invalid temporal values are:

• 00:00:60
• 00:60:00
• 999-00-00
• 999-01-01
• 0000-00-00
• 2009-00-00
• 2009-13-00
• 999-00-00 00:00:00
• 999-01-01 00:00:00
• 0000-00-00 00:00:00
• 1000-00-00 00:00:00
• 2009-00-00 00:00:00
• 2009-13-00 00:00:00
• 2009-05-26 00:00:60
• 2009-05-26 00:60:00
• 2009-05-26 24:00:00

And my test cases for first real temporal values are:

• 00:00:00
• 00:00:01
• 1000-01-01
• 2009-01-01
• 1000-01-01 00:00:00
• 2009-01-01 00:00:00

And there is only one real zero-equivalent temporal value: 00:00:00.

So the first requirement is to find a MySQL function that returns NULL for all those invalid values, and that function is TO_DAYS with one exception:

mysql> SELECT TO_DAYS('999-01-01 00:00:00');
+-------------------------------+
| TO_DAYS('999-01-01 00:00:00') |
+-------------------------------+
|                        364878 |
+-------------------------------+

That date is only valid if years before 1000 are handled but the MySQL manual says that,

TO_DAYS() is not intended for use with values that precede the advent of the Gregorian calendar (1582)

so we’re already way past the limit of its intended use and, moreover, the supported lower limit of a date or datetime is 1000-01-01, so says the manual. It’s reasonable to not bother with pre-year 1000 dates so I’ll overlook this.

Excepting pre-year 1000 dates, TO_DAYS() returns NULL for all the invalid values. By contrast, UNIX_TIMESTAMP() returns zero for all the invalid values and TIME_TO_SEC() returns a mix of NULL, zero, and values. So the apparent winner for requirement #1 is TO_DAYS(), but…

Requirement #2 complicates the issue because the time 00:00:00 is valid and zero-equivalent but TO_DAYS() returns NULL for it. We need a hack that handles all the cases, and here it is:

SELECT IF(TIME_FORMAT(?,'%H:%i:%s')=?, TIME_TO_SEC(?), TO_DAYS(?))

That says, basically: if the value is a time then evaluate it with TIME_TO_SEC(), else evaluate it with TO_DAYS(). It works so well in fact that it satisfies all four requirements. 00:00:00 evaluates to zero, all the invalid values evaluate to NULL, and all the valid values evaluate to various non-null values. I have to use TIME_FORMAT() instead of just TIME() because TIME() wasn’t introduced until MySQL v4.1 (fourth requirement).

The hack works because of this (substituting TIME() for TIME_FORMAT()):

mysql> SELECT TIME('00:00:00');
+------------------+
| TIME('00:00:00') |
+------------------+
| 00:00:00         |
+------------------+

mysql> SELECT TIME('00-00-00');
+------------------+
| TIME('00-00-00') |
+------------------+
| 00:00:00         |
+------------------+

mysql> SELECT TIME('2010-05-26');
+--------------------+
| TIME('2010-05-26') |
+--------------------+
| 00:20:10           |
+--------------------+

mysql> SELECT TIME('2010-05-26 10:10:10');
+-----------------------------+
| TIME('2010-05-26 10:10:10') |
+-----------------------------+
| 10:10:10                    |
+-----------------------------+

As you can see, TIME() (or TIME_FORMAT()) returns the exact same value if the given value is a time, otherwise it interprets the value–which is a date or datetime–as a time causing it to return a different value than the given value. Thus we discern time values from date and datetime values and evaluate them separately with TIME_TO_SEC().

I tested on MySQL v4.0, 4.1, 5.0 and 5.1 and all pass. The only difference is 4.0 verses the others for the pre-year 1000 dates, but I’m ignoring these anyway.

Of course all the preceding could have been accomplished in code by looking at the column type and choosing the correct MySQL function to evaluate the value and check if it’s zero-equivalent, but I was curious to see if it could be done using only MySQL since, after all, it is MySQL that permits these silly, invalid temporals values.

If you know a simpler, more elegant solution that meets the four requirements and passes all the tests, please share!

I’m surprised, and flattered, to see that people still use, write and recommend mysqlsla, mysqlreport and–most surprisingly–mysqlsniffer. In truth, however, I consider all the original Hack MySQL tools as retired. Maatkit consumes the majority of my development time and provides better replacements for all the Hack MySQL tools. The mk tools are better because–most importantly–they’re tested, their code is more robust, and they benefit from the collected knowledge and experience of the community’s top minds (whereas the Hack MySQL tools are brain-children of only my knowledge and experience circa several years ago).

Thus I created a new tools page where I list and briefly profile free, open-source MySQL tools. As the intro paragraph states, MySQL Forge does this, too, but imho the forge is a dense jungle in which it is difficult to discern the useful bits from the less-than-useful bits. My tools page is meant to 1) inform people that the Hack MySQL tools are retired, 2) list replacements for them, and 3) give people new to the MySQL universe a quick, simple list of tools they’ll probably want to become familiar with.

Kind thanks to all who used, wrote about, contributed to and recommend the Hack MySQL tools over the years.