Hack MySQL

Archive for the ‘logging_query’ tag

Drizzle 7 plugins

with one comment

Last week I wrote about my experience compiling Drizzle 7 on Mac OS X 10.6. Then David Shrewsbury informed me of his nearly identical blog post: Installing Drizzle from source on OS X. Once Drizzle 7 was running on my box, I immediately looked to see what plugins where available because Drizzle uses a lot of plugins and they are one of its notable differences from MySQL. In my humble opinion, Drizzle’s plugins will primarily influence how database professionals evaluate and decide whether or not to use Drizzle because so many of Drizzle’s features are plugins. Therefore, let’s look briefly at some the plugins included with Drizzle 7.

The plugin directory of the Drizzle 7 tarball lists 79 plugins. Each plugin has a plugin.ini file that tells Drizzle its name (title) and description (among other things):

Plugin Title Description
ascii UDF Plugin for ASCII UDF Plugin for returning ASCII value of a character
auth_all Allow All Authentication Default authentication module.
auth_file File-based Authentication Authenticate against a list of username:password entries in a plain text file.
auth_http HTTP Authentication Plugin HTTP based authentications
auth_ldap LDAP Authentication Authenticate against an LDAP server.
auth_pam PAM Authenication Plugin PAM based authenication.
auth_test Authentication Test A hard-coded authentication test plugin
benchmark UDF Plugin for Benchmark UDF Plugin for Benchmark
catalog Catalog related DDL functions. Provides functions for DDL operations on catalogs.
charlength UDF Plugin for CharLength UDF Plugin for returning number of characters in a string
coercibility_function COERCIBILITY() Function Gets coercibility for an expression
collation_dictionary Character And Collation Dictionary Dictionary plugins for character set and collation system.
compression compression UDFs UDF Plugin for compression
connection_id Connection ID UDF UDF Plugin for connection_id function
console Console Client Plugin that provides a command console in the drizzled process.
crc32 crc32 UDF UDF Plugin for crc32
debug Kernel debugger utilities. crash(), assert(), and additional developer style tools.
default_replicator Default Replicator A simple default replicator which replicates all write events to all appliers
drizzle_protocol Drizzle Protocol Module Drizzle protocol module.
errmsg_stderr Error Message Plugin Errmsg Plugin that sends messages to stderr.
error_dictionary Error Code data dictionary tables. List the error codes and messages genearted by Drizzle.
filtered_replicator Filtered Replicator A simple filtered replicator which allows a user to filter out events based on a schema or table name
function_dictionary Function data dictionary tables. List the functions available to SQL developers.
function_engine Data Engine Function Engine used for Table Functions.
gearman_udf Gearman Client UDFs UDF plugin that provides a Gearman client interface
haildb HailDB storage engine Uses the HailDB library to implement a storage engine for Drizzle.
hello_events An example event observer Plugin An example event observer Plugin for Drizzle
hello_world Example Plugin UDF Plugin for Hello World.
hex_functions HEX() and UNHEX() functions converting strings to and from hexadecimal
information_schema_dictionary Information Schema Dictionary Creates dictionary tables for the ANSI information schema tables
innobase InnoDB Storage Engine Transactional Tables using InnoDB
length UDF Plugin for Length UDF Plugin for returning byte length of a string
logging_gearman Gearman Logging Plugin Logging Plugin that logs to Gearman.
logging_query Query Logging Plugin Logging Plugin that logs all queries.
logging_stats Statistics Plugin for Drizzle Statistics Plugin for Drizzle
math_functions Math Functions Math Functions
md5 UDF Plugin for MD5 UDF Plugin for MD5
memcached_functions Memcached Client UDFs UDF plugin for libmemcached
memcached_query_cache Query Cache Query Cache for Memcached
memcached_stats Memcached Stats in DATA_DICTIONARY tables Some DATA_DICTIONARY tables that provide Memcached stats
memory Memory Storage Engine Volatile memory based tables
multi_thread Multi-Thread Scheduler One Thread Per Connection Scheduler
myisam MyISAM Storage Engine Legacy non-transactional MySQL tables
mysql_protocol MySQL Protocol Module MySQL protocol module that is based on the old libmysql code.
mysql_unix_socket_protocol MySQL Unix Socket Protocol Module MySQL Unix Socket Protocol Module
pbms PrimeBase Media Stream Daemon Provides BLOB streaming service for storage engines
performance_dictionary Performance Dictionary Table functions related to performance issues.
protocol_dictionary Protocol Dictionary Provides dictionary tables for protocol counters
rabbitmq RabbitMQ Replicator Replicator plugin that publishes transactions to RabbitMQ
rand_function RAND() function Function for pseudo-random number generation
regex_policy Regex Policy Authorization plugin which implements a regex to object mapping
registry_dictionary Registry Dictionary Provides dictionary tables for plugin system.
replication_dictionary Registry Dictionary Provides dictionary tables for replication system
reverse_function REVERSE function function that reverses a string
rot13 ROT 13 A wonderful ROT 13 implementation
schema_dictionary Schema Dictionary Creates dictionary tables for federated schema data.
schema_engine Schema Engine This implements the default file based Schema engine.
session_dictionary Session Dictionary Dictionary for session information, includes processlist system.
show_dictionary Show Dictionary Creates dictionary tables for SHOW commands
show_schema_proto SHOW_SCHEMA_PROTO(db) Function Shows a text representation of the schema (database) proto
shutdown_function Shutdown function Use this function to shutdown the database.
signal_handler Default Signal Handler Default Signal Handler
simple_user_policy Simple User Policy Authorization plugin which implements a simple policy-based strategy
slave Replication Slave Provides slave replication services
sleep Sleep Function Function Plugin for sleep().
status_dictionary Status Dictionary Dictionary system for status, variables, commands.
storage_engine_api_tester Storage Engine API Tester Test engine that enforces SE API calling conventions very strictly
string_functions String Functions String Functions
substr_functions SUBSTRING() and SUBSTRING_INDEX() functions SUBSTRING and SUBSTRING_INDEX functions
syslog Syslog Syslog interface for query log, error messages, and functions
table_cache_dictionary Table Cached and Table Definition Dictionary A look into the fabulous world of what is in your caches.
tableprototester Table Proto Message Tester Used only to test other bits of the server.
transaction_log Transaction Log Log of Transaction Messages
trigger_dictionary Trigger Dictionary Table for describing what triggers have been loaded.
user_locks User Locks. Provides lock functions to users.
utility_dictionary Utility data dictionary tables. Generate data to be used for tests, etc.
utility_functions Utility Functions Utility Functions
uuid_function UUID() plugin UUID() function as generated by libuuid
version version UDF UDF Plugin for version function

This list demonstrates to what extent features of Drizzle are outside its kernel and provided instead by plugins. Even SLEEP() is a plugin! By comparison, many of these features are built-in with MySQL, like authentication and replication. It is not my intent in this blog post to discuss the pros and cons of this “(almost) everything is a plugin” design decision (I like it actually), so let’s instead play with a plugin to see how it treats us; let’s enable auth_file because I recently read Eric Day’s blog post Writing Authentication Plugins for Drizzle.

Enabling a Plugin: auth_file

For my first attempt, I simply start drizzled with the option --plugin-add=auth_file. That does not work:

$ cat var/drizzled.err 
Could not load auth file: Could not open users file: /opt/drizzle7/etc/drizzle.users

Plugin 'auth_file' init function returned error.

Aborting

That is, at least, a nice error message: I know what failed and why. I could probably just make that file, but since I don’t know what format it expects my inclination is to consult the Drizzle reference manual because I’m habituated to consulting the MySQL reference manual whenever I have questions. Sadly, however, the Drizzle plugin documentation is severely limited and does not list the plugin, nor does the Drizzle authentication documentation page. Eric’s blog post mentioned earlier describes the file format, but I’m going to pretend to be a regular user and consult the regular sources of information. If the database’s docu doesn’t say how to enable/configure the plugin, my next guess is drizzled --help, but that doesn’t list auth_file. Then I try drizzled --plugin-add=auth_file --help and that does list:

Options used by auth_file:
  --auth-file.users arg (=/opt/drizzle7/etc/drizzle.users)
                                        File to load for usernames and 
                                        passwords

It’s nice that --help is dynamic, but it still doesn’t tell me what file format auth_file expects. If I could start drizzled with the plugin, then perhaps I could do SELECT * FROM data_dictionary.plugins WHERE plugin_name='auth_file' but I can’t do that either. So as a naïve user, I’m out of luck unless I guess.

For the sake of making this example work, I’ll stop being the naïve user and instead consult the list of plugins above that says “Authenticate against a list of username:password entries in a plain text file.” and Eric’s blog post mentioned earlier which says the same things. So I create /opt/drizzle7/etc/drizzle.users with one line foo:bar and then attempt to start drizzled with the plugin again:

$ drizzle
Welcome to the Drizzle client..  Commands end with ; or \g.
Your Drizzle connection id is 1
Connection protocol: mysql
Server version: 2011.03.13 Source distribution (drizzle)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

drizzle> SELECT * FROM data_dictionary.plugins WHERE plugin_name='auth_file';
+-------------+----------------+-----------+-------------+
| PLUGIN_NAME | PLUGIN_TYPE    | IS_ACTIVE | MODULE_NAME |
+-------------+----------------+-----------+-------------+
| auth_file   | Authentication | YES       | auth_file   | 
+-------------+----------------+-----------+-------------+

Success! But wait… I still connected with my account username (daniel) and no password. Why? Because another authentication module is enabled:

drizzle> SELECT * FROM data_dictionary.plugins WHERE plugin_name LIKE 'auth%';
+-------------+----------------+-----------+--------------------------+
| PLUGIN_NAME | PLUGIN_TYPE    | IS_ACTIVE | MODULE_NAME              |
+-------------+----------------+-----------+--------------------------+
| auth_all    | Authentication | YES       | Allow-All-Authentication | 
| auth_file   | Authentication | YES       | auth_file                | 
+-------------+----------------+-----------+--------------------------+

It seems I need to also start drizzled with --plugin-remove=auth_all. This produces the expected result:

$ drizzle
ERROR 1045 (28000): Access denied for user 'daniel' (using password: NO)

$ drizzle --user foo --password=bar
Welcome to the Drizzle client..  Commands end with ; or \g.
Your Drizzle connection id is 5
Connection protocol: mysql
Server version: 2011.03.13 Source distribution (drizzle)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

drizzle> 

Of course not all plugins have to be configured; some are self-evident, like sleep… or are they?

drizzle> SELECT SLEEP(7);
+----------+
| SLEEP(7) |
+----------+
|        0 | 
+----------+
1 row in set (0.000624 sec)

Notice the last line: the query did not sleep for 7 seconds; it returned immediately. It seems that the sleep plugin is broken or I’m not using it correctly. SLEEP(1.0) sleeps for 1.0 seconds, but SLEEP(7.0) returns immediately. SLEEP() is probably not a mission-critical feature of the database, but its failure in Drizzle highlights the need for more testing and documentation of plugins.

A final word on this topic before we move on: without documentation I don’t know how to configure real authentication and security. How does one, for example, grants certain users certain access to certain databases or tables? Are there column-level grants? What privileges are needed to enable or disable dynamic features? In brief: Drizzle authentication/security is either under-developped or under-documented. I hope it’s just the latter because anyone can write docs, fewer can write good authentication modules.

Active Plugins and Their Types

Let’s continue to looking at what information about plugins is exposed inside Drizzle by executing SELECT * FROM data_dictionary.plugins ORDER BY plugin_name:

+----------------------+-----------------------+-----------+------------------------------+
| PLUGIN_NAME          | PLUGIN_TYPE           | IS_ACTIVE | MODULE_NAME                  |
+----------------------+-----------------------+-----------+------------------------------+
| abs                  | Function              | YES       | Math Functions               | 
| acos                 | Function              | YES       | Math Functions               | 
| ascii                | Function              | YES       | ascii                        | 
| asin                 | Function              | YES       | Math Functions               | 
| auth_file            | Authentication        | YES       | auth_file                    | 
| console              | Listen                | YES       | console                      | 
| default_replicator   | TransactionReplicator | YES       | default_replicator           | 
| Error_message_stderr | ErrorMessage          | YES       | errmsg_stderr                | 

That is an abbreviated list; the complete list is 169 plugins of various types. Two questions result from this: why are there more plugins here than listed above, and what are the various plugin types? I don’t know the answer to the first question, but I’m sure a Drizzle developer does? My semi-educated guess is that Drizzle also has modules which provide several functions/plugins. Don’t quote me on that though.

The second question can be answered easily, and it gives us an insight into how plugins are organized and what aspects of Drizzle they implement:

drizzle> SELECT plugin_type, COUNT(plugin_name) AS n_plugins
      -> FROM data_dictionary.plugins
      -> GROUP BY plugin_type ORDER BY n_plugins DESC;
+-----------------------+-----------+
| plugin_type           | n_plugins |
+-----------------------+-----------+
| TableFunction         |        79 | 
| Function              |        74 | 
| StorageEngine         |         5 | 
| Listen                |         4 | 
| Authentication        |         1 | 
| Catalog               |         1 | 
| Daemon                |         1 | 
| ErrorMessage          |         1 | 
| Logging               |         1 | 
| Scheduler             |         1 | 
| TransactionReplicator |         1 | 
+-----------------------+-----------+

All the TableFunction plugins are related to the DATA_DICTIONARY and the INFORMATION_SCHEMA, so they’re probably something that end users will just expect to work and wouldn’t seek to replace or tweak. The Function plugins cover a range of functions from HEX() and VERSION() to KILL(). These are probably also aspects of the database that users will just expect to work, but I like that we have the ability to plug in a new RAND(), for example. Similarly, plugin types Catalog, Daemon, ErrorMessage and Scheduler are core aspects of the database that few users will want to tweak. On the other hand, the remaining types allow for a lot of innovation and development.

The Listen plugins relate to protocols that Drizzle can speak. My build includes the console “protocol”, MySQL and Drizzle TCP/IP protocols, and the standard Unix socket protocol. In theory, Drizzle could speak directly to other database technologies, like memcached, with a plugin to enable/translate the foreign “language”. This is nice because several database technologies are often used in conjunction.

The Authentication plugins will probably also become very important and popular because authentication/security is a central aspect of a database. Many big corporations and governments have complex security requirements and there a lot of security/authentication methods. From time to time I hear people criticize MySQL’s authentication as too rudimentary. This is no longer a problem in Drizzle.

StorageEngine plugins will no doubt play a central role in Drizzle’s future just as it does in MySQL’s present. I won’t say any more about this because, for one, it’s already well understood (i.e. the benefits of pluggable storage engines), and, two, Drizzle is new and there’s a lot of storage engines so it remains to be seen which storage engines become the most influential (like how MyISAM was influential in MySQL’s early history, then gave way to InnoDB).

Another important, pluggable aspect of Drizzle that is not well represented by this list but hinted to by TransactionReplicator is replication. Unlike MySQL, replication in Drizzle is pluggable. Like storage engines, pluggable replication opens the door new to innovations like RabbitReplication, but also like storage engines, Drizzle is new so it remains to be seen which system of replication will become the most influential. Or, perhaps, no one system will dominate and users will need to choose the best replication plugin for their particular environment.

Lastly, the plugin type that interests me since I’ve written a couple of log parsers and I’m the primary coder for mk-query-digest: Logging. I’ll be the one to say it: original/order versions of the MySQL statement logs (i.e. the general log and the slow log) sucked. The general log is a nightmare to parse and it’s really nothing more than queries and timestamps–no “actionable” information. The slow log is better since it has information like query execution time, etc., but it wasn’t really until Percona extensively extended the slow log (both in configurability and output) that the slow log allowed us to do a lot of valuable analyses of query events inside MySQL. Therefore I’m very happy that logging in Drizzle is pluggable so that logging can be done well from the start. Let’s see what logging awesomeness Drizzle has for us.

Query Logging with logging_query

Sadly the Drizzle plugin documentation doesn’t yet mention any logging plugins. The Drizzle logging documentation lists Unix Syslog and File logging, but at the time of writing, both of those pages are blank. From the list of plugins at the start of this blog post, we see logging_query which, by its description, seems to act like a general log since it logs “all” queries, not just slow queries. It’s our only option so we’ll try to enable and use it.

I first tried to simply enable the plugin by starting drizzled with --plugin-add. The server started but there was no logging to be found anywhere, and the plugin did not even show as enabled in the DATA_DICTIONARY. A little confused, I consulted:

$ drizzled --plugin-add=logging_query --help
...
Options used by logging-query:
  --logging-query.enable                Enable logging to CSV file
  --logging-query.filename arg          File to log to
  --logging-query.pcre arg              PCRE to match the query against
  --logging-query.threshold-slow arg (=0)
                                        Threshold for logging slow queries, in 
                                        microseconds
  --logging-query.threshold-big-resultset arg (=0)
                                        Threshold for logging big queries, for 
                                        rows returned
  --logging-query.threshold-big-examined arg (=0)
                                        Threshold for logging big queries, for 
                                        rows examined

Perhaps I must “doubly” enable the plugin: add it with --plugin-add and turn it on with --logging-query.enable. I try that but still no query log can be found anywhere and Drizzle throws no errors. So, based on reading Ronald’d old blog post about this very topic, Drizzle query logging, I add a third option: --logging-query.filename=$BASEDIR/var/query-log.csv. That makes the plugin come alive: the plugin is listed as enabled in the DATA_DICTIONARY and var/query-log.csv exists:

$ cat var/query-log.csv 
1303070901062988,1,1,"","select @@version_comment limit 1","Query",1077,243,243,1,0,0,0,1,"beatrice.local"
1303070902100576,1,2,"","select * from data_dictionary.plugins where plugin_name='logging_query'","Query",1038665,2498,575,2,170,0,0,1,"beatrice.local"

I won’t repeat what Ronald already said in his aforementioned blog post, but he makes the important note: “Unfortunately the log does not yet provide a header.” Yes, that’s very unfortunate because there is no official documentation so the current query log format is essentially useless to anyone but hackers like Ronald and people like me who care enough to search out the answers that I need.

Conclusion

The Drizzle 7 GA tarball comes with roughly 70 usable plugins in its plugin/ directory. More plugins, over 100, are shown in the DATA_DICTIONARY.plugins table. These plugins represent the majority of features provided by Drizzle 7, from query logging, to user authentication, to storage engines, to replication. At present, many of the provided plugins are either bare-bones implementations of features or not sufficiently documented to be usable by a normal user.

It’s easy to criticize, but I make these observations to help bring attention to areas that need work because I want Drizzle to develop and mature quickly. The pluggable storage engine feature of MySQL helped spur the development of storage engines resulting in innovation and competition that still, today, provides users great, free choices. The pluggable design has been taken to the extreme in Drizzle, extending far beyond just storage engines, to nearly everything the database server can do. I hope this spurs similar advancements in features that are lacking in Drizzle, as well as cool new features no one has imagined yet.

From this brief look at the plugins provided with Drizzle 7, I suggest the follow two things to the Drizzle developers:

  1. Document everything about Drizzle at docs.drizzle.org. Now that Drizzle has a GA release, I think good, complete documentation is the single most important task that will allow average users to actually use Drizzle, thereby giving Drizzle traction in the market, thereby creating the need for further development based on user feedback and bugs.
  2. Develop a standard for how plugins should behave. For example: auth_file has a nice error message, but logging_query does nothing silently until you enable it “just right”.

Thanks to those who did the hard work of making Drizzle pluggable and writing the first generation of plugins.

Written by Daniel Nichter

April 17th, 2011 at 3:05 pm