Hack MySQL
Hack MySQL is my personal blog about MySQL, the Go programming language, and software design.
The content of this site is inspired by two things. First is my work at Block. Working with thousands of databases, hundreds of microservices, and engineers across the company, I’m fortunate to always have new challenges and opportunities to both learn and mentor. Many of the posts and pages began as conversations with colleagues. Second is “the literature”: books, other blogs, articles, white papers, tech manuals, etc. I think tech careers are built upon the literature, people sharing knowledge—hacking. I hope this site contributes to the literature.
Under the hood, this static web site uses:
No AI was used to write or edit text on this website. Although AI is a useful tool, all content on this website (excpet where cited or linked) is original work written and edited by Daniel Nichter.
Book
I am the author of Efficient MySQL Performance, published by O’Reilly.
Contact
Email “public” at this domain.
I am zero social media: no Facebook, no Twitter, no LinkedIn, no Instagram—nothing. Anyone with my name on social media is not me.
Output
Andy Grove wrote about activity versus output in High Output Management. I am driven by (and have a healthy obsession with) output that contributes value to other people. I’m happiest at work when building tools and services that help others work more efficiently. And fortunately I’ve had many opportunities to do that in my career:
✓ Efficient MySQL Performance
I wrote Efficient MySQL Performance for software engineers using MySQL, not DBAs. That’s a huge audience; I bet users outnumber DBAs 1,000 to 1. If a software engineer reads and applies everything in this book, I’m certain they will achieve amazing MySQL performance—a skill that will help propel their career.
✓ Percona Toolkit
I’m commonly known in the MySQL industry for the 8 years I spent working on Percona Toolkit (PT). Anyone remotely familiar with MySQL knows that PT is to MySQL as a mechanic’s tools are to a car: indispensable and practically inseparable. PT is so popular, I bet that every MySQL DBA in the world has used at least one PT tool.
✓ Percona Cloud Tools
Not many will know or remember Percona Cloud Tools (PCT) because the project was shut down after 2 years of development. PCT was the first commercial query metrics SaaS platform. In short, it was Datadog Deep Database Monitoring or SolarWinds Database Performance Monitor (formerly VividCortex) but created years before these two.
✓ Percona Monitoring and Management
Many parts of PCT were turned into Percona Monitoring and Management (PMM) v1.0. I worked on PMM during my final year at Percona. Today, many years later, PMM is still a core Percona product, and a few (but not many) of my original v1.0 code contributions remain.
✓ HackMySQL Tools
Years before Percona Toolkit, I got my start in the MySQL industry with this domain and a few tools: mysqlreport, mysqlsla, mysqlsniffer, and others.
These tools were retired a long time ago (replaced by PT and other products), but for several years they were the de facto standard in the industry.
Amazingly, this site still gets hits for these tools, which is why stub pages like Archive / mysqlreport exist: to let people know that whatever link they followed is seriously out of date.
✓ HackMySQL Content
When my tools became obsolete, the content of this domain—the blog posts and pages—became the focus. The current version of this site has only the last 6 years of content, but I’ve been writing here for almost 20 years (since 2005). This is a niche site, but it has a few thousand visitors every month. More importantly for me, many of the posts and pages here allow me to “write once, teach many” since I spend a good portion of my time training new engineers.
✓ Finch
Finch is my latest open source contribution: a modern MySQL benchmarking tool. Like most long-time MySQL users, I’ve used sysbench for almost 20 years, but I needed (and wanted) a much easier and more flexible benchmark tool. At work, I need to evaluate (benchmark) many different types of databases and workloads. COMMIT Latency: Aurora vs. RDS MySQL 8.0 is a good example. I made Finch to make this work exceptionally easy so I could fiddle less with the tool and focus more on the business results. Since I had the time, I spent 8 months polishing Finch into an incredibly powerful yet easy and flexible tool.
✓ Blip
Blip is my second to latest open source contribution: the most advanced MySQL monitor ever written. Check out the Blip docs. I hope Blip will become the MySQL monitor that everyone uses. That’ll take a long time, so let’s check back in a few years.
✓ Etre
Etre is a small but mighty service: an API for tracking assets with user-defined labels. My colleagues and I developed it at Square to track MySQL nodes and associated metadata (as labels). Other back end services query Etre to find particular database nodes. There are other solutions for tracking assets with metadata, but Etre is simpler to use and operate—it just works.
✓ Spin Cycle
Unlike Etre, Spin Cycle is large and necessarily complex.
My colleagues and I developed it at Square to coordinate doing A + B + C + … = Z, where Z was the back end product we needed to create by calling many different back end systems—everything from REST and gRPC APIs owned by other teams, to remote system commands via rce-agent.
This was years before workflow services like Temporal.
But even today, I’m not sure a workflow service would work for various reasons that don’t matter here.
Spin Cycle is a powerful, reliable, and reusable workhorse that coordinates most of the Square database fleet at impressive scale.
✓ square/rce-agent
rce-agent is a highly secure remote command execution (RCE) client-server developed by me and my colleagues at Square.
At a fintech company, even ssh isn’t secure enough, so this little but important service takes RCE security to the extreme.
✓ go-test/deep
Used by over 20,000 (!) other projects, go-test/deep is a surprisingly popular little package, probably because it makes Go testing so much easier, which is why I developed it. It’s a tiny, single function package, but the internal complexity is 🤯 because it requires plumbing nearly the full depth of Go reflection.
✓ go-mysql
With go-mysql I had hoped to create the go-to project for MySQL-related Go modules. Basically, I wanted to reproduce the PT modules in Go but with even better, more reusable interfaces. But this project didn’t take off—it’s really difficult creating widely used reusable packages. The best outcome was go-mysql/hotswap-dsn-driver, which is novel and very useful.
✓ go-metrics
go-metrics is a beautifully designed metric package for counters, gauges, and histograms. It’s also rigorously researched and implemented. However, it’s not popular, probably because engineers use whatever metrics package is already in use and rarely shop around for a better one. Maybe some day more engineers will discover and use it.
✓ go-cmd
go-cmd wraps Go os/exec to provide all the little (but important) details that the low-level Go package doesn’t provide. It has garnered a decent user base and many contributors, but it’s a nice, so I don’t expect it to be wildly popular. Regardless, it provides a lot of value by handling a lot of fine details with respect to correctly running (and terminating) external commands.
✓ Square Query Metrics
When I joined Square in 2016, my first project was to create an internal query metrics system like Datadog Deep Database Monitoring or SolarWinds Database Performance Monitor (formerly VividCortex). For various reasons, neither PMM nor VividCortex worked for Square at that time. But I had developed or helped develop most query metric tools in the MySQL industry, so it took me only 3 months (with the help of an intern) to develop the internal query metrics system that Square still uses today. In the years since, it has provided countless value to hundreds (perhaps thousands) of engineers and saved Square a lot of money (because commercial products are expensive at scale).
✓ Cash DBA Team
In 2021, I switch from Square to Cash (both a part of Block) to bootstrap a new Cash DBA team. After 5 years on the Square DBA team, I was no longer needed for a good reason: I successfully helped build that team and infrastructure up and out to the point where they were successful without me. Then I switch to Cash and did the same in two years: a new manager and a handful of engineers who successfully manage the Cash database fleet without me. I’m still at Cash, but currently I work on various aspects of the whole Cash data strategy.
✓ Conference Presentations
Over the course of my career with MySQL, I have often presented at the yearly MySQL conference (which changed owners and names over the years). For example, you can hear me drone on about Efficient MySQL Performance at Percona Live 2022.
✓ BA Philosophy
I have a bachelors degree in philosophy. As it turns out, logic and reason are quite useful with databases and software engineering. And in a business sense, being able to reason and communicate very clearly about large, complex issues is a super power. At work, a large part of my job is bring clarity and focus to projects, questions, and issues; and this helps engineers work efficiently and happily by knowing what they’re doing and why.
✓ MA French
I have a masters degree in French. Never mind why. Like my BA in philosophy, my MA in French taught me how to work hard—really hard—and how to teach others. As any graduate student knows, graduate studies require a lot more work than undergraduate studies. The same is true in business the more one’s career advances. And as some graduate students know, teaching others is its own field of study: pedagogy. In my graduate program, I took a course on pedagogy and taught several French classes (as the full “professor of record”, not a TA). Now I teach MySQL, software engineering, and related subjects, but the pedagogical principles are the same.
✓ French in the History of Wyoming
I consider my masters thesis as one of my greatest accomplishments and contributions to the world: French influences in the history of Wyoming from La Salle to Arland. Years later, I adapted it for a book called French in the History of Wyoming that I hope to have published. Since I don’t have a Ph.D. in history, it’s difficult to find a publisher even though I am absolutely certain the material is qualified because the work was vetted and approved by five professors (three of whom from prestigious schools: Yale, Harvard, and Stanford). But since isn’t computer-related, I won’t digress on this web site.