Hack MySQL

Archive for the ‘MySQL’ Category

Drizzle 7 plugins

with one comment

Last week I wrote about my experience compiling Drizzle 7 on Mac OS X 10.6. Then David Shrewsbury informed me of his nearly identical blog post: Installing Drizzle from source on OS X. Once Drizzle 7 was running on my box, I immediately looked to see what plugins where available because Drizzle uses a lot of plugins and they are one of its notable differences from MySQL. In my humble opinion, Drizzle’s plugins will primarily influence how database professionals evaluate and decide whether or not to use Drizzle because so many of Drizzle’s features are plugins. Therefore, let’s look briefly at some the plugins included with Drizzle 7.

The plugin directory of the Drizzle 7 tarball lists 79 plugins. Each plugin has a plugin.ini file that tells Drizzle its name (title) and description (among other things):

Plugin Title Description
ascii UDF Plugin for ASCII UDF Plugin for returning ASCII value of a character
auth_all Allow All Authentication Default authentication module.
auth_file File-based Authentication Authenticate against a list of username:password entries in a plain text file.
auth_http HTTP Authentication Plugin HTTP based authentications
auth_ldap LDAP Authentication Authenticate against an LDAP server.
auth_pam PAM Authenication Plugin PAM based authenication.
auth_test Authentication Test A hard-coded authentication test plugin
benchmark UDF Plugin for Benchmark UDF Plugin for Benchmark
catalog Catalog related DDL functions. Provides functions for DDL operations on catalogs.
charlength UDF Plugin for CharLength UDF Plugin for returning number of characters in a string
coercibility_function COERCIBILITY() Function Gets coercibility for an expression
collation_dictionary Character And Collation Dictionary Dictionary plugins for character set and collation system.
compression compression UDFs UDF Plugin for compression
connection_id Connection ID UDF UDF Plugin for connection_id function
console Console Client Plugin that provides a command console in the drizzled process.
crc32 crc32 UDF UDF Plugin for crc32
debug Kernel debugger utilities. crash(), assert(), and additional developer style tools.
default_replicator Default Replicator A simple default replicator which replicates all write events to all appliers
drizzle_protocol Drizzle Protocol Module Drizzle protocol module.
errmsg_stderr Error Message Plugin Errmsg Plugin that sends messages to stderr.
error_dictionary Error Code data dictionary tables. List the error codes and messages genearted by Drizzle.
filtered_replicator Filtered Replicator A simple filtered replicator which allows a user to filter out events based on a schema or table name
function_dictionary Function data dictionary tables. List the functions available to SQL developers.
function_engine Data Engine Function Engine used for Table Functions.
gearman_udf Gearman Client UDFs UDF plugin that provides a Gearman client interface
haildb HailDB storage engine Uses the HailDB library to implement a storage engine for Drizzle.
hello_events An example event observer Plugin An example event observer Plugin for Drizzle
hello_world Example Plugin UDF Plugin for Hello World.
hex_functions HEX() and UNHEX() functions converting strings to and from hexadecimal
information_schema_dictionary Information Schema Dictionary Creates dictionary tables for the ANSI information schema tables
innobase InnoDB Storage Engine Transactional Tables using InnoDB
length UDF Plugin for Length UDF Plugin for returning byte length of a string
logging_gearman Gearman Logging Plugin Logging Plugin that logs to Gearman.
logging_query Query Logging Plugin Logging Plugin that logs all queries.
logging_stats Statistics Plugin for Drizzle Statistics Plugin for Drizzle
math_functions Math Functions Math Functions
md5 UDF Plugin for MD5 UDF Plugin for MD5
memcached_functions Memcached Client UDFs UDF plugin for libmemcached
memcached_query_cache Query Cache Query Cache for Memcached
memcached_stats Memcached Stats in DATA_DICTIONARY tables Some DATA_DICTIONARY tables that provide Memcached stats
memory Memory Storage Engine Volatile memory based tables
multi_thread Multi-Thread Scheduler One Thread Per Connection Scheduler
myisam MyISAM Storage Engine Legacy non-transactional MySQL tables
mysql_protocol MySQL Protocol Module MySQL protocol module that is based on the old libmysql code.
mysql_unix_socket_protocol MySQL Unix Socket Protocol Module MySQL Unix Socket Protocol Module
pbms PrimeBase Media Stream Daemon Provides BLOB streaming service for storage engines
performance_dictionary Performance Dictionary Table functions related to performance issues.
protocol_dictionary Protocol Dictionary Provides dictionary tables for protocol counters
rabbitmq RabbitMQ Replicator Replicator plugin that publishes transactions to RabbitMQ
rand_function RAND() function Function for pseudo-random number generation
regex_policy Regex Policy Authorization plugin which implements a regex to object mapping
registry_dictionary Registry Dictionary Provides dictionary tables for plugin system.
replication_dictionary Registry Dictionary Provides dictionary tables for replication system
reverse_function REVERSE function function that reverses a string
rot13 ROT 13 A wonderful ROT 13 implementation
schema_dictionary Schema Dictionary Creates dictionary tables for federated schema data.
schema_engine Schema Engine This implements the default file based Schema engine.
session_dictionary Session Dictionary Dictionary for session information, includes processlist system.
show_dictionary Show Dictionary Creates dictionary tables for SHOW commands
show_schema_proto SHOW_SCHEMA_PROTO(db) Function Shows a text representation of the schema (database) proto
shutdown_function Shutdown function Use this function to shutdown the database.
signal_handler Default Signal Handler Default Signal Handler
simple_user_policy Simple User Policy Authorization plugin which implements a simple policy-based strategy
slave Replication Slave Provides slave replication services
sleep Sleep Function Function Plugin for sleep().
status_dictionary Status Dictionary Dictionary system for status, variables, commands.
storage_engine_api_tester Storage Engine API Tester Test engine that enforces SE API calling conventions very strictly
string_functions String Functions String Functions
substr_functions SUBSTRING() and SUBSTRING_INDEX() functions SUBSTRING and SUBSTRING_INDEX functions
syslog Syslog Syslog interface for query log, error messages, and functions
table_cache_dictionary Table Cached and Table Definition Dictionary A look into the fabulous world of what is in your caches.
tableprototester Table Proto Message Tester Used only to test other bits of the server.
transaction_log Transaction Log Log of Transaction Messages
trigger_dictionary Trigger Dictionary Table for describing what triggers have been loaded.
user_locks User Locks. Provides lock functions to users.
utility_dictionary Utility data dictionary tables. Generate data to be used for tests, etc.
utility_functions Utility Functions Utility Functions
uuid_function UUID() plugin UUID() function as generated by libuuid
version version UDF UDF Plugin for version function

This list demonstrates to what extent features of Drizzle are outside its kernel and provided instead by plugins. Even SLEEP() is a plugin! By comparison, many of these features are built-in with MySQL, like authentication and replication. It is not my intent in this blog post to discuss the pros and cons of this “(almost) everything is a plugin” design decision (I like it actually), so let’s instead play with a plugin to see how it treats us; let’s enable auth_file because I recently read Eric Day’s blog post Writing Authentication Plugins for Drizzle.

Enabling a Plugin: auth_file

For my first attempt, I simply start drizzled with the option --plugin-add=auth_file. That does not work:

$ cat var/drizzled.err 
Could not load auth file: Could not open users file: /opt/drizzle7/etc/drizzle.users

Plugin 'auth_file' init function returned error.

Aborting

That is, at least, a nice error message: I know what failed and why. I could probably just make that file, but since I don’t know what format it expects my inclination is to consult the Drizzle reference manual because I’m habituated to consulting the MySQL reference manual whenever I have questions. Sadly, however, the Drizzle plugin documentation is severely limited and does not list the plugin, nor does the Drizzle authentication documentation page. Eric’s blog post mentioned earlier describes the file format, but I’m going to pretend to be a regular user and consult the regular sources of information. If the database’s docu doesn’t say how to enable/configure the plugin, my next guess is drizzled --help, but that doesn’t list auth_file. Then I try drizzled --plugin-add=auth_file --help and that does list:

Options used by auth_file:
  --auth-file.users arg (=/opt/drizzle7/etc/drizzle.users)
                                        File to load for usernames and 
                                        passwords

It’s nice that --help is dynamic, but it still doesn’t tell me what file format auth_file expects. If I could start drizzled with the plugin, then perhaps I could do SELECT * FROM data_dictionary.plugins WHERE plugin_name='auth_file' but I can’t do that either. So as a naïve user, I’m out of luck unless I guess.

For the sake of making this example work, I’ll stop being the naïve user and instead consult the list of plugins above that says “Authenticate against a list of username:password entries in a plain text file.” and Eric’s blog post mentioned earlier which says the same things. So I create /opt/drizzle7/etc/drizzle.users with one line foo:bar and then attempt to start drizzled with the plugin again:

$ drizzle
Welcome to the Drizzle client..  Commands end with ; or \g.
Your Drizzle connection id is 1
Connection protocol: mysql
Server version: 2011.03.13 Source distribution (drizzle)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

drizzle> SELECT * FROM data_dictionary.plugins WHERE plugin_name='auth_file';
+-------------+----------------+-----------+-------------+
| PLUGIN_NAME | PLUGIN_TYPE    | IS_ACTIVE | MODULE_NAME |
+-------------+----------------+-----------+-------------+
| auth_file   | Authentication | YES       | auth_file   | 
+-------------+----------------+-----------+-------------+

Success! But wait… I still connected with my account username (daniel) and no password. Why? Because another authentication module is enabled:

drizzle> SELECT * FROM data_dictionary.plugins WHERE plugin_name LIKE 'auth%';
+-------------+----------------+-----------+--------------------------+
| PLUGIN_NAME | PLUGIN_TYPE    | IS_ACTIVE | MODULE_NAME              |
+-------------+----------------+-----------+--------------------------+
| auth_all    | Authentication | YES       | Allow-All-Authentication | 
| auth_file   | Authentication | YES       | auth_file                | 
+-------------+----------------+-----------+--------------------------+

It seems I need to also start drizzled with --plugin-remove=auth_all. This produces the expected result:

$ drizzle
ERROR 1045 (28000): Access denied for user 'daniel' (using password: NO)

$ drizzle --user foo --password=bar
Welcome to the Drizzle client..  Commands end with ; or \g.
Your Drizzle connection id is 5
Connection protocol: mysql
Server version: 2011.03.13 Source distribution (drizzle)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

drizzle> 

Of course not all plugins have to be configured; some are self-evident, like sleep… or are they?

drizzle> SELECT SLEEP(7);
+----------+
| SLEEP(7) |
+----------+
|        0 | 
+----------+
1 row in set (0.000624 sec)

Notice the last line: the query did not sleep for 7 seconds; it returned immediately. It seems that the sleep plugin is broken or I’m not using it correctly. SLEEP(1.0) sleeps for 1.0 seconds, but SLEEP(7.0) returns immediately. SLEEP() is probably not a mission-critical feature of the database, but its failure in Drizzle highlights the need for more testing and documentation of plugins.

A final word on this topic before we move on: without documentation I don’t know how to configure real authentication and security. How does one, for example, grants certain users certain access to certain databases or tables? Are there column-level grants? What privileges are needed to enable or disable dynamic features? In brief: Drizzle authentication/security is either under-developped or under-documented. I hope it’s just the latter because anyone can write docs, fewer can write good authentication modules.

Active Plugins and Their Types

Let’s continue to looking at what information about plugins is exposed inside Drizzle by executing SELECT * FROM data_dictionary.plugins ORDER BY plugin_name:

+----------------------+-----------------------+-----------+------------------------------+
| PLUGIN_NAME          | PLUGIN_TYPE           | IS_ACTIVE | MODULE_NAME                  |
+----------------------+-----------------------+-----------+------------------------------+
| abs                  | Function              | YES       | Math Functions               | 
| acos                 | Function              | YES       | Math Functions               | 
| ascii                | Function              | YES       | ascii                        | 
| asin                 | Function              | YES       | Math Functions               | 
| auth_file            | Authentication        | YES       | auth_file                    | 
| console              | Listen                | YES       | console                      | 
| default_replicator   | TransactionReplicator | YES       | default_replicator           | 
| Error_message_stderr | ErrorMessage          | YES       | errmsg_stderr                | 

That is an abbreviated list; the complete list is 169 plugins of various types. Two questions result from this: why are there more plugins here than listed above, and what are the various plugin types? I don’t know the answer to the first question, but I’m sure a Drizzle developer does? My semi-educated guess is that Drizzle also has modules which provide several functions/plugins. Don’t quote me on that though.

The second question can be answered easily, and it gives us an insight into how plugins are organized and what aspects of Drizzle they implement:

drizzle> SELECT plugin_type, COUNT(plugin_name) AS n_plugins
      -> FROM data_dictionary.plugins
      -> GROUP BY plugin_type ORDER BY n_plugins DESC;
+-----------------------+-----------+
| plugin_type           | n_plugins |
+-----------------------+-----------+
| TableFunction         |        79 | 
| Function              |        74 | 
| StorageEngine         |         5 | 
| Listen                |         4 | 
| Authentication        |         1 | 
| Catalog               |         1 | 
| Daemon                |         1 | 
| ErrorMessage          |         1 | 
| Logging               |         1 | 
| Scheduler             |         1 | 
| TransactionReplicator |         1 | 
+-----------------------+-----------+

All the TableFunction plugins are related to the DATA_DICTIONARY and the INFORMATION_SCHEMA, so they’re probably something that end users will just expect to work and wouldn’t seek to replace or tweak. The Function plugins cover a range of functions from HEX() and VERSION() to KILL(). These are probably also aspects of the database that users will just expect to work, but I like that we have the ability to plug in a new RAND(), for example. Similarly, plugin types Catalog, Daemon, ErrorMessage and Scheduler are core aspects of the database that few users will want to tweak. On the other hand, the remaining types allow for a lot of innovation and development.

The Listen plugins relate to protocols that Drizzle can speak. My build includes the console “protocol”, MySQL and Drizzle TCP/IP protocols, and the standard Unix socket protocol. In theory, Drizzle could speak directly to other database technologies, like memcached, with a plugin to enable/translate the foreign “language”. This is nice because several database technologies are often used in conjunction.

The Authentication plugins will probably also become very important and popular because authentication/security is a central aspect of a database. Many big corporations and governments have complex security requirements and there a lot of security/authentication methods. From time to time I hear people criticize MySQL’s authentication as too rudimentary. This is no longer a problem in Drizzle.

StorageEngine plugins will no doubt play a central role in Drizzle’s future just as it does in MySQL’s present. I won’t say any more about this because, for one, it’s already well understood (i.e. the benefits of pluggable storage engines), and, two, Drizzle is new and there’s a lot of storage engines so it remains to be seen which storage engines become the most influential (like how MyISAM was influential in MySQL’s early history, then gave way to InnoDB).

Another important, pluggable aspect of Drizzle that is not well represented by this list but hinted to by TransactionReplicator is replication. Unlike MySQL, replication in Drizzle is pluggable. Like storage engines, pluggable replication opens the door new to innovations like RabbitReplication, but also like storage engines, Drizzle is new so it remains to be seen which system of replication will become the most influential. Or, perhaps, no one system will dominate and users will need to choose the best replication plugin for their particular environment.

Lastly, the plugin type that interests me since I’ve written a couple of log parsers and I’m the primary coder for mk-query-digest: Logging. I’ll be the one to say it: original/order versions of the MySQL statement logs (i.e. the general log and the slow log) sucked. The general log is a nightmare to parse and it’s really nothing more than queries and timestamps–no “actionable” information. The slow log is better since it has information like query execution time, etc., but it wasn’t really until Percona extensively extended the slow log (both in configurability and output) that the slow log allowed us to do a lot of valuable analyses of query events inside MySQL. Therefore I’m very happy that logging in Drizzle is pluggable so that logging can be done well from the start. Let’s see what logging awesomeness Drizzle has for us.

Query Logging with logging_query

Sadly the Drizzle plugin documentation doesn’t yet mention any logging plugins. The Drizzle logging documentation lists Unix Syslog and File logging, but at the time of writing, both of those pages are blank. From the list of plugins at the start of this blog post, we see logging_query which, by its description, seems to act like a general log since it logs “all” queries, not just slow queries. It’s our only option so we’ll try to enable and use it.

I first tried to simply enable the plugin by starting drizzled with --plugin-add. The server started but there was no logging to be found anywhere, and the plugin did not even show as enabled in the DATA_DICTIONARY. A little confused, I consulted:

$ drizzled --plugin-add=logging_query --help
...
Options used by logging-query:
  --logging-query.enable                Enable logging to CSV file
  --logging-query.filename arg          File to log to
  --logging-query.pcre arg              PCRE to match the query against
  --logging-query.threshold-slow arg (=0)
                                        Threshold for logging slow queries, in 
                                        microseconds
  --logging-query.threshold-big-resultset arg (=0)
                                        Threshold for logging big queries, for 
                                        rows returned
  --logging-query.threshold-big-examined arg (=0)
                                        Threshold for logging big queries, for 
                                        rows examined

Perhaps I must “doubly” enable the plugin: add it with --plugin-add and turn it on with --logging-query.enable. I try that but still no query log can be found anywhere and Drizzle throws no errors. So, based on reading Ronald’d old blog post about this very topic, Drizzle query logging, I add a third option: --logging-query.filename=$BASEDIR/var/query-log.csv. That makes the plugin come alive: the plugin is listed as enabled in the DATA_DICTIONARY and var/query-log.csv exists:

$ cat var/query-log.csv 
1303070901062988,1,1,"","select @@version_comment limit 1","Query",1077,243,243,1,0,0,0,1,"beatrice.local"
1303070902100576,1,2,"","select * from data_dictionary.plugins where plugin_name='logging_query'","Query",1038665,2498,575,2,170,0,0,1,"beatrice.local"

I won’t repeat what Ronald already said in his aforementioned blog post, but he makes the important note: “Unfortunately the log does not yet provide a header.” Yes, that’s very unfortunate because there is no official documentation so the current query log format is essentially useless to anyone but hackers like Ronald and people like me who care enough to search out the answers that I need.

Conclusion

The Drizzle 7 GA tarball comes with roughly 70 usable plugins in its plugin/ directory. More plugins, over 100, are shown in the DATA_DICTIONARY.plugins table. These plugins represent the majority of features provided by Drizzle 7, from query logging, to user authentication, to storage engines, to replication. At present, many of the provided plugins are either bare-bones implementations of features or not sufficiently documented to be usable by a normal user.

It’s easy to criticize, but I make these observations to help bring attention to areas that need work because I want Drizzle to develop and mature quickly. The pluggable storage engine feature of MySQL helped spur the development of storage engines resulting in innovation and competition that still, today, provides users great, free choices. The pluggable design has been taken to the extreme in Drizzle, extending far beyond just storage engines, to nearly everything the database server can do. I hope this spurs similar advancements in features that are lacking in Drizzle, as well as cool new features no one has imagined yet.

From this brief look at the plugins provided with Drizzle 7, I suggest the follow two things to the Drizzle developers:

  1. Document everything about Drizzle at docs.drizzle.org. Now that Drizzle has a GA release, I think good, complete documentation is the single most important task that will allow average users to actually use Drizzle, thereby giving Drizzle traction in the market, thereby creating the need for further development based on user feedback and bugs.
  2. Develop a standard for how plugins should behave. For example: auth_file has a nice error message, but logging_query does nothing silently until you enable it “just right”.

Thanks to those who did the hard work of making Drizzle pluggable and writing the first generation of plugins.

Written by Daniel Nichter

April 17th, 2011 at 3:05 pm

Detecting invalid and zero temporal values

without comments

I’ve been thinking a lot about invalid and zero temporal values and how to detect them with MySQL date and time functions because mk-table-checksum has to handle “everything” correctly and efficiently. The requirements are complex because we have to take into account what MySQL allows to be stored verses what it allows to be used in certain operations and functions, how it sorts a mix of real and invalid temporal values for MIN() and MAX(), how to detect a temporal value as equivalent to zero, and how different MySQL versions might affect any of the aforementioned.

At base, the four guiding requirements are:

  1. Detect and discard invalid time, date, and datetime values
  2. Detect zero-equivalent temporal values
  3. Do #1 and #2 using only MySQL functions
  4. Work in MySQL 4.0 and newer

My tests cases for invalid temporal values are:

  • 00:00:60
  • 00:60:00
  • 999-00-00
  • 999-01-01
  • 0000-00-00
  • 2009-00-00
  • 2009-13-00
  • 999-00-00 00:00:00
  • 999-01-01 00:00:00
  • 0000-00-00 00:00:00
  • 1000-00-00 00:00:00
  • 2009-00-00 00:00:00
  • 2009-13-00 00:00:00
  • 2009-05-26 00:00:60
  • 2009-05-26 00:60:00
  • 2009-05-26 24:00:00

And my test cases for first real temporal values are:

  • 00:00:00
  • 00:00:01
  • 1000-01-01
  • 2009-01-01
  • 1000-01-01 00:00:00
  • 2009-01-01 00:00:00

And there is only one real zero-equivalent temporal value: 00:00:00.

So the first requirement is to find a MySQL function that returns NULL for all those invalid values, and that function is TO_DAYS with one exception:

mysql> SELECT TO_DAYS('999-01-01 00:00:00');
+-------------------------------+
| TO_DAYS('999-01-01 00:00:00') |
+-------------------------------+
|                        364878 | 
+-------------------------------+

That date is only valid if years before 1000 are handled but the MySQL manual says that,

TO_DAYS() is not intended for use with values that precede the advent of the Gregorian calendar (1582)

so we’re already way past the limit of its intended use and, moreover, the supported lower limit of a date or datetime is 1000-01-01, so says the manual. It’s reasonable to not bother with pre-year 1000 dates so I’ll overlook this.

Excepting pre-year 1000 dates, TO_DAYS() returns NULL for all the invalid values. By contrast, UNIX_TIMESTAMP() returns zero for all the invalid values and TIME_TO_SEC() returns a mix of NULL, zero, and values. So the apparent winner for requirement #1 is TO_DAYS(), but…

Requirement #2 complicates the issue because the time 00:00:00 is valid and zero-equivalent but TO_DAYS() returns NULL for it. We need a hack that handles all the cases, and here it is:

SELECT IF(TIME_FORMAT(?,'%H:%i:%s')=?, TIME_TO_SEC(?), TO_DAYS(?))

That says, basically: if the value is a time then evaluate it with TIME_TO_SEC(), else evaluate it with TO_DAYS(). It works so well in fact that it satisfies all four requirements. 00:00:00 evaluates to zero, all the invalid values evaluate to NULL, and all the valid values evaluate to various non-null values. I have to use TIME_FORMAT() instead of just TIME() because TIME() wasn’t introduced until MySQL v4.1 (fourth requirement).

The hack works because of this (substituting TIME() for TIME_FORMAT()):

mysql> SELECT TIME('00:00:00');
+------------------+
| TIME('00:00:00') |
+------------------+
| 00:00:00         | 
+------------------+

mysql> SELECT TIME('00-00-00');
+------------------+
| TIME('00-00-00') |
+------------------+
| 00:00:00         | 
+------------------+

mysql> SELECT TIME('2010-05-26');
+--------------------+
| TIME('2010-05-26') |
+--------------------+
| 00:20:10           | 
+--------------------+

mysql> SELECT TIME('2010-05-26 10:10:10');
+-----------------------------+
| TIME('2010-05-26 10:10:10') |
+-----------------------------+
| 10:10:10                    | 
+-----------------------------+

As you can see, TIME() (or TIME_FORMAT()) returns the exact same value if the given value is a time, otherwise it interprets the value–which is a date or datetime–as a time causing it to return a different value than the given value. Thus we discern time values from date and datetime values and evaluate them separately with TIME_TO_SEC().

I tested on MySQL v4.0, 4.1, 5.0 and 5.1 and all pass. The only difference is 4.0 verses the others for the pre-year 1000 dates, but I’m ignoring these anyway.

Of course all the preceding could have been accomplished in code by looking at the column type and choosing the correct MySQL function to evaluate the value and check if it’s zero-equivalent, but I was curious to see if it could be done using only MySQL since, after all, it is MySQL that permits these silly, invalid temporals values.

If you know a simpler, more elegant solution that meets the four requirements and passes all the tests, please share!

Written by Daniel Nichter

May 26th, 2010 at 5:48 pm

Hack MySQL tools retired, succeeded

without comments

I’m surprised, and flattered, to see that people still use, write and recommend mysqlsla, mysqlreport and–most surprisingly–mysqlsniffer. In truth, however, I consider all the original Hack MySQL tools as retired. Maatkit consumes the majority of my development time and provides better replacements for all the Hack MySQL tools. The mk tools are better because–most importantly–they’re tested, their code is more robust, and they benefit from the collected knowledge and experience of the community’s top minds (whereas the Hack MySQL tools are brain-children of only my knowledge and experience circa several years ago).

Thus I created a new tools page where I list and briefly profile free, open-source MySQL tools. As the intro paragraph states, MySQL Forge does this, too, but imho the forge is a dense jungle in which it is difficult to discern the useful bits from the less-than-useful bits. My tools page is meant to 1) inform people that the Hack MySQL tools are retired, 2) list replacements for them, and 3) give people new to the MySQL universe a quick, simple list of tools they’ll probably want to become familiar with.

Kind thanks to all who used, wrote about, contributed to and recommend the Hack MySQL tools over the years.

Written by Daniel Nichter

May 23rd, 2010 at 12:45 pm

Posted in MySQL

Tagged with ,

MySQL features timeline

without comments

I’ve begun a MySQL features timeline which is a quick reference showing as of what version MySQL features were added, changed or removed. The manual tells us this, of course, but I wanted a quicker reference. The list is far from complete as there’s a huge number of features to cover. I’ll continue to improve it and help is appreciated. Send me a quick email saying “feature x added/removed/changed as of version y” and I’ll do the rest. — If someone has already done this, please give me the url so I don’t reinvent the wheel.

Written by Daniel Nichter

October 26th, 2009 at 6:28 pm

Posted in MySQL

Tagged with ,

mysqlsla v2.00 released

without comments

mysqlsla v2 is finally “done” and released. About 3 months ago, when v1.8 was released, I said it would be coming “soon,” but time just flew by and here we are. Oh well. In any case, the v1 branch is dead to me and v2 is all the rave (at least for me). If you don’t care about the differences and all you want is your default top 10 report from a slow log, for example, then all you need to know is: mysqlsla -lt slow SLOW_LOG

For those interested in what has changed to warrant a new major version number, here’s the briefing of changes/overhauls:

  1. Almost ALL new command line options (–log-type a.k.a. -lt is the most important); see the documentation
  2. Customizable reports. In v1 the report was hard-coded. Now with -rf FILE you can format your own report. See the doc on reports.
  3. FULL filtering and sorting. Somewhere around v1.7 (if I remember correctly) filtering based on things like the connection ID, etc. was introduced. v2 is more abstracted and allows more complex filtering on any “meta-property” available in the log (and sorting by most of them, too). See the doc on filters.
  4. Replays to compact fat text log files into super-compact binary files.
  5. Slightly better query abstraction.
  6. Full support for microslow patched slow logs.
  7. User-defined logs so that, one day, we may begin parsing, filtering, analyzing and sorting custom logs from things like MySQL Proxy.
  8. And in general, what I called the “mysqlsla v2 Library: 6 long, detailed pages documenting just about single, tiny aspect of mysqlsla (plus with the “installer” you get a mysqlsla man page, too).

As this is pretty much an entire code overhaul, please please report bugs, problems, suggestions and whatever. Please also send me your log files: slows, generals, binaries. They help me tremendously because I cannot sit around inventing up all the weird kinds of stuff I’ve seen in other people’s logs.

And as always, thank you all for your support, and thanks to those people who provided feedback, patches and such.

Written by Daniel Nichter

July 12th, 2008 at 1:39 pm