Sunday, December 30, 2012

Don't forget to check your partitions!

As it's the end of the year it might be time to check your partition definitions.

If you forget to add a new partition in time partitions with no MAXVALUE might start to throw errors:
mysql> create table nye (`event_id` int not null auto_increment,`edate` year(4) not null, description varchar(200), 
    -> primary key(`event_id`,`edate`)) 
    -> partition by range( edate ) (
    -> partition p2010 VALUES LESS THAN (2011), 
    -> partition p2011 VALUES LESS THAN (2012), 
    -> partition p2012 VALUES LESS THAN (2013) );
Query OK, 0 rows affected (0.04 sec)

mysql> INSERT INTO nye(edate,description) VALUES('2010','twenty ten'); 
Query OK, 1 row affected (0.00 sec)

mysql> INSERT INTO nye(edate,description) VALUES('2011','twenty eleven');
Query OK, 1 row affected (0.00 sec)

mysql> INSERT INTO nye(edate,description) VALUES('2012','twenty twelve');
Query OK, 1 row affected (0.00 sec)

mysql> INSERT INTO nye(edate,description) VALUES('2013','twenty thirdteen');
ERROR 1526 (HY000): Table has no partition for value 2013

And partitions with a MAXVALUE might start to contain data of multiple years, and that might not be how it was intended.

So check your partitioning if you partition by year. It might also be a good moment to think about archiving old data.

You could use the common_schema.sql_range_partitions view to create the next partition.

mysql> select sql_add_next_partition from common_schema.sql_range_partitions where table_name='nye';
+-------------------------------------------------------------------------------------+
| sql_add_next_partition                                                              |
+-------------------------------------------------------------------------------------+
| alter table `test`.`nye` add partition (partition `p_2014` values less than (2014)) |
+-------------------------------------------------------------------------------------+
1 row in set, 22 warnings (0.08 sec)

mysql> call eval("select sql_add_next_partition from common_schema.sql_range_partitions where table_name='nye'");
Query OK, 0 rows affected (0.12 sec)

mysql> INSERT INTO nye(edate,description) VALUES('2013','twenty thirdteen');
Query OK, 1 row affected (0.00 sec)

O, and your sharding definitions might also contain a reference to a year.

And with the right partition definitions it's just:

mysql> SELECT TIMEDIFF(NOW(),'2013-01-01 00:00:00') AS 'Countdown';
+-----------+
| Countdown |
+-----------+
| -27:19:07 |
+-----------+
1 row in set (0.00 sec)

Sunday, December 9, 2012

A difficult XtraBackup restore

There was one MySQL server with a Adaptec Raid controller and 4 disks. One of the disks was having media errors and caused the whole SCSI bus to become unavailable.

This resulted in a corrupted InnoDB table.

Luckily we did have backups. A full backup and incrementals.

So to restore the backups I installed XtraBackup and MySQL 5.5 on another server.

Then the first step was to 'prepare' the backup. This worked okay for the full backup (redo only).

The second step to add the incremantals failed for the first incremental. This was easily resolved by specifying the full paths instead of relative paths.

Then the backup was fully prepared using the redo logs and undo logs.

As XtraBackup doesn't backup your my.cnf we copied the my.cnf from another server and adjusted it for this server. The my.cnf in your backup only contains everything needed for a restore, and some of those settings are Percona Server specific and will result in an error when used with MySQL.

So far everything went as expected.

Then we started MySQL with the newly restored backup by executing "service mysql start", this failed.

Then I tried to start it with a "mysqld_safe &", this worked. So I expected the init script to use wrong parameters. So I executed "bash -x /etc/init.d/mysql start" and expected the error to show, but MySQL started okay this time. Using "/etc/init.d/mysql stop" and then "/etc/init.d/mysql start" resulted in an error again.
Then I added some echo statements to the script to check which parameters it used for mysqld_safe. Then I tried to start mysqld_safe with the same parameters, this worked.

So mysqld_safe failed to start and didn't log anything to the error log when executed from the init script. It does work when called directly.

Then it hit me. The SELinux context for the MySQL files was wrong (ls -lZ).

This is something that must not be changed with chcon as that won't survive a relabel action. I changed the context for the datadir (not /var/lib/mysql in this case) and location for the socket and pid files with semanage. The I used restorecon to apply it to the files. And then MySQL started fine. This is explained in in detail in this blog post.

Conclusions:
  • XtraBackup works great (and fast, disk I/O is quickly a bottleneck for restores)
  • Use full paths instead of relative paths for XtraBackup
  • Make sure to backup your my.cnf (or put in a CM tool like puppet)
  • If you use SELinux, don't forget to make sure the file context is correct after a restore.

Sunday, December 2, 2012

MySQL Zeroday's

SANS ISC reported a number of zeroday's for MySQL today.

* CVE-2012-5611 MySQL (Linux) Stack based buffer overrun PoC Zeroday
http://seclists.org/fulldisclosure/2012/Dec/4
https://bugzilla.redhat.com/show_bug.cgi?id=882599

* CVE-2012-5612 MySQL (Linux) Heap Based Overrun PoC Zeroday
http://seclists.org/fulldisclosure/2012/Dec/5
https://bugzilla.redhat.com/show_bug.cgi?id=882600

* CVE-2012-5613 MySQL (Linux) Database Privilege Elevation Zeroday
Exploit
http://seclists.org/fulldisclosure/2012/Dec/6
https://bugzilla.redhat.com/show_bug.cgi?id=882606

* CVE-2012-5614 MySQL Denial of Service Zeroday PoC
http://seclists.org/fulldisclosure/2012/Dec/7
https://bugzilla.redhat.com/show_bug.cgi?id=882607

* CVE-2012-5615 MySQL Remote Preauth User Enumeration Zeroday
http://seclists.org/fulldisclosure/2012/Dec/9
https://bugzilla.redhat.com/show_bug.cgi?id=882608
 
Source: http://seclists.org/oss-sec/2012/q4/387 

Wednesday, November 7, 2012

Scale with MySQL

Today there was the 'Scale with MySQL' event in the new Oracle building in Utrecht. There were sessions about the MySQL 5.6 RC and about MySQL Cluster. It was nice to meet so many other MySQL users. It was interesting too hear about what MySQL is used for and in which kind of environments.

Visit the MySQL Events page to see all other location for the 'Scale with MySQL' sessions.

And there are more options for meeting other MySQL users in the Netherlands: The first meetup for the MySQL User Group NL is on Friday November 16th.

Sunday, October 28, 2012

First meeting for MySQL User Group NL announced


Kinderdijk windmills 


With almost 40 member the MySQL User Group NL is already a success.

I've now scheduled the first meeting. It will be on Friday November 16th in Amsterdam. Please signup using the meetup.com page.

Agenda (See meetup.com for latest updates)
18:00 Welcome
18:30 First presentation
19:30 Food
20:00 Second presentation

I'm looking for speakers, so feel free to suggest a speaker or to volunteer to speak.

More resources for this user group:


Saturday, October 20, 2012

MySQL AutoTuner

After reading a blog post about MySQL Tuning scripts I thought about the possibility of a fully Automatic MySQL Tuner.

This is how it would work:
A daemon which would connect to your database server and then fetch status variables, just like mysqltuner and such. Then the daemon could decide that a parameter would need to be adjusted and then run "SET GLOBAL …" and write a /etc/mysql/autotuner.cf file which should be included in your my.cnf.

It should have a min/max setting for each option and some thresholds.

Why?
  • Not everyone is a DBA
  • It's could better than the default settings is most cases. Luckily many defaults are updated in 5.6.
  • You're not using my-huge.cf, are you?
  • It could help when there are changing workloads
  • It might be sufficient for a developer environment
  • MySQL might be embedded in a 'virtual appliance' which can be deployed on may different kinds of hardware.
Why not?
  • The risk of it taking a wrong decision is too high
  • It might be better then the defaults, but still worse than a trained DBA
  • A few important settings (InnoDB Buffer Pool Size and InnoDB Logs) require a restart. I don't like anything to restart MySQL automatically (does your puppet config do automatic restarts?)
  • It might generates differences between servers with the same role. This might not be an issue when the autotuner would use P2P/multicast to communicate with other autotuners, but that makes it even more scary
  • I don't like bolted-on solutions
The conclusion is that I don't think this is a good solution. 

Thursday, September 20, 2012

Automated MySQL Master Failover

After the GitHub MySQL Failover incident a lot of blogs/people have explained that fully automated failover might not be the most optimal solution.

Fully automated failover is indeed dangerous, and should  be avoided if possible. But a complete manual failover is also dangerous. A fully automated manually triggered failover is probably a better solution.

A synchronous replication solution is also not a complete solution. A split-brain situation is a good example of a failure which could happen. Of course most clusters have all kinds of safe guard to prevent that, but unfortunately also safe guards can fail.

Every failover/cluster should be considered broken unless:
  1. You've tested the failover scripts and procedures
  2. You've tested the failover scripts and procedures under normal load
  3. You've tested the failover scripts and procedures under high load
  4. You've tested it since the last change in the setup and/or application
  5. Someone else tested the failover scripts and procedures
  6. Everyone working with the cluster/failover setup is properly trained
Just like they do on the Mythbusters show: first verify if it's true and then do everything you can to make it go BOOM!

And a failover/cluster solution is not a backup nor a complete disaster recovery solution.

Just to name a few of the failures with HA setups I've seen:
  • Master and Standy both failing with a full disk because of binlogs
  • Network hicups causing failovers to a standby with a cold cache
  • Failovers triggered by human mistakes (quite often)
  • Failovers because of bugs in the cluster software
  • Failovers due to someone removing the table which was monitored by the loadbalancer
  • Failovers due to a run-away monitoring check from the loadbalancers
  • Failovers due to failover software on a lower level taking 'smart' decisions (Veritas Cluster Server on VMware ESX with VMware HA)