Category: MySQL & Other Databases

  • MySQL Replication Setup

    Mysql uses a Master-slave/Publisher-Subscriber model for Replication. MySQL replication is an asynchronous replication. In MySQL replication master keeps a log of all of the updates performed on the database. Then, one or more slaves connect to the Master(Publisher Server), read each log entry, and perform the indicated update on the slave (Subscriber) server databases. The master server is responsible for the track of log rotation and access control.

    Each slave server has to keep the track of current position within the server’s transaction log. As new transactions occur on the server, they get logged on the master server and downloaded by each slave. Once the transaction has been committed by each slave, the slaves update their position in the server’s transaction log and wait for the next transaction.

    In this article, I will show you the steps to configure the Master/Slave replication between two servers.

    Step 1: Create a user on Master server which Slave server can use to connect. I have created the user named “repl_user”.

    --Connect to MySQL Master server
    mysql -u root -proot
    --Execute the below code
    GRANT REPLICATION SLAVE ON *.* TO 'repl_user'@'%' IDENTIFIED BY 'password';
    FLUSH PRIVILEGES;
    

    Step 2: We have to change the MySQL configuration file usually in the /etc/mysql.cnf location. Here we will add the replication configuration parameters.

    log-bin – will be used to write a log on the desired location
    binlog-do-db – will be used to enabled the database for writing log. I have used Publisher_Database, you have to specify your database name.
    server-id – Specify the ID of the Master server

    log-bin = /home/mysql/logs/mysql-bin.log
    binlog-do-db=publisher_database
    server-id=1
    

    Step 3: Once you have added the above configuration parameters into the My.cnf, next step is restart the MySQL Master Instance.
    You can use below command to restart the MySQl service.

    /etc/init.d/mysqld restart
    service mysqld restart
    

    Step 4: We have to configure the /etc/my.cnf file on the slave server. Here we will add the below parameters in the configuration file.

    server-id – gives the Slave its unique ID
    master-host – tells the Slave the I.P address of the Master server for connection. You can get the IP address using IPConfig command.
    master-connect-retry – Here we will specify the connection retry interval.
    master-user – Specify the user which has permission access the Master server
    master-password – Specify the password of the replication user mentioned above
    replicate-do-db – Specify the subscriber database name
    relay-log – direct slave to use relay log

    server-id=2
    master-host=128.20.30.1
    master-connect-retry=60
    master-user=repl_user
    master-password=password
    replicate-do-db=subscriber_slave
    relay-log = /var/lib/mysql/slave-relay.log
    relay-log-index = /var/lib/mysql/slave-relay-log.index
    

    Step 5: Restart the slave MySQl instance

    /etc/init.d/mysqld restart
    service mysqld restart
    

    Step 6: If your Master MySQL instance is live instance, you have to do the backup/restore using MySQLDump utility.

    --Connect to MySQL Master server
    mysql -u root -proot
    
    --Stop the write operation
    FLUSH TABLES WITH READ LOCK;
    
    --Generate the dump of the database (backup)
    --gzip command will compress the file and create the zip file name backup.sql.gz
    mysqldump publisher_master -u root -p > /home/my_home_dir/backup.sql;
    gzip /home/my_home_dir/backup.sql;
    
    --execute below copy command on slave to copy the backup file
    scp root@128.20.30.1:/home/my_home_dir/database.sql.gz /home/my_home_dir/
    
    --Once copied, extract the file using gunzip
    gunzip /home/my_home_dir/backup.sql.gz
    
    --restore the databsae
    mysql -u root -p subscriber_slave  </home/my_home_dir/backup.sql
    
    

    Step 7: Execute the SHOW MASTER STATUS command on Master server. It will give you the bin log file name and position which we will use specify the slave.

    SHOW MASTER STATUS;
    


    +------------------+----------+--------------+------------------+
    | File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
    +------------------+----------+--------------+------------------+
    | mysql-bin.000001 | 707 | exampledb | |
    +------------------+----------+--------------+------------------+

    Step 8: Execute the below commands on slave.

    --Connect to MySQL Slave server
    mysql -u root -proot
    --stop the slave
    slave stop;
    -- Execute the below command
    CHANGE MASTER TO MASTER_HOST='128.20.30.1', MASTER_USER='repl_user', MASTER_PASSWORD='password', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=707;
    --Start the Slave
    slave start;
    

    Step 9: Login to Master MySQL instance and unlock the tables.

    --Connect to MySQL Master server
    mysql -u root -proot
    -- unlock the tables if you have executed lock tables command
    unlock tables;
    

    You are all set. Master to Slave replication has been started. Make sure while configuring the my.cnf file.
    1. Take the copy of my.cnf file before starting the replication configuration.
    2. Make sure skip-networking parameter is not enabled in the my.cnf file.

  • Scripts which make you Database Hero


    — Create databsae SQLDBPool
    CREATE DATABASE [sqldbpool] ON PRIMARY
    ( NAME = N’sqldbpool’, FILENAME = N’C:\sqldbpool.mdf’ , SIZE = 2048KB , MAXSIZE = UNLIMITED, FILEGROWTH = 1024KB )
    LOG ON
    ( NAME = N’sqldbpool_log’, FILENAME = N’C:\sqldbpool_log.ldf’ , SIZE = 1024KB , MAXSIZE = 2048GB , FILEGROWTH = 10%)
    COLLATE SQL_Latin1_General_CP1_CI_AS
    GO

    –Script to create schema
    USE [sqldbpool]
    GO
    CREATE SCHEMA [mySQLDBPool] AUTHORIZATION [dbo]

    — Script to create table with constraints
    create table mySQLDBPool.Emp
    (
    EmpID int Primary key identity(100,1),
    EmpName Varchar(20) Constraint UK1 Unique,
    DOB datetime Not Null,
    JoinDate datetime default getdate(),
    Age int Constraint Ck1 Check (Age > 18)
    )

    — Script to change the recovery model of the databsae
    USE [master]
    GO
    ALTER DATABASE [SQLDBPool] SET RECOVERY FULL WITH NO_WAIT
    GO

    ALTER DATABASE [SQLDBPool] SET RECOVERY FULL
    GO

    — Script to take the full backup of database
    BACKUP DATABASE [SQLDBPool] TO DISK = N’D:\SQLDBPool.bak’
    WITH NOFORMAT, INIT, NAME = N’SQLDBPool-Full Database Backup’,
    NOREWIND, SKIP, NOUNLOAD, STATS = 10
    GO

    –Script to take the Differential Database backup
    BACKUP DATABASE [SQLDBPool] TO DISK = N’D:\SQLDBPool.diff.bak’
    WITH DIFFERENTIAL , NOFORMAT, INIT, NAME = N’SQLDBPool-Diff Backup’,
    NOREWIND, SKIP, NOUNLOAD, STATS = 10
    GO

    –Script to take the Transaction Log backup that truncates the log
    BACKUP LOG [SQLDBPool] TO DISK = N’D:\SQLDBPoolTlog.trn’
    WITH NOFORMAT, INIT, NAME = N’SQLDBPool-Transaction Log Backup’,SKIP, NOREWIND, NOUNLOAD, STATS = 10
    GO

    — Backup the tail of the log (not normal procedure)
    BACKUP LOG [SQLDBPool] TO DISK = N’D:\SQLDBPoolLog.tailLog.trn’
    WITH NO_TRUNCATE , NOFORMAT, INIT, NAME = N’SQLDBPool-Transaction Log Backup’,NOREWIND,SKIP, NOUNLOAD, NORECOVERY , STATS = 10
    GO

    — Script to Get the backup file properties
    RESTORE FILELISTONLY FROM DISK = ‘D:\SQLDBPool.bak’

    — Script to Restore Full Database Backup
    RESTORE DATABASE [SQLDBPool1] FROM DISK = N’D:\SQLDBPool.bak’
    WITH FILE = 1, MOVE N’sqldbpool’ TO N’D:\SQLDBPooldata.mdf’,
    MOVE N’sqldbpool_log’ TO N’D:\SQLDBPoollog_1.ldf’,
    NOUNLOAD, STATS = 10
    GO

    — Script to delete the backup history of the specific databsae
    EXEC msdb.dbo.sp_delete_database_backuphistory @database_name = N’SQLDBPool1′
    GO

    — Full restore with no recovery (status will be Restoring)
    RESTORE DATABASE [SQLDBPool1] FROM DISK = N’D:\SQLDBPool.bak’
    WITH FILE = 1, MOVE N’SQLDBPool’ TO N’D:\SQLDBPooldata.mdf’,
    MOVE N’SQLDBPool_Log’ TO N’D:\SQLDBPoolLog_1.ldf’,
    NORECOVERY, NOUNLOAD, STATS = 10
    GO

    — Restore transaction log with recovery
    RESTORE LOG [SQLDBPool1] FROM DISK = N’D:\SQLDBPoolLog.trn’
    WITH FILE = 1, NOUNLOAD, RECOVERY STATS = 10
    GO

    –Script to bring the database online without restoring log backup
    restore database sqldbpool with recovery

    –Script to detach database
    USE [master]
    GO
    EXEC master.dbo.sp_detach_db @dbname = N’SQLDBPool’
    GO

    — Script to get the database information
    sp_helpdb ‘SQLDBPOOL’

    –to Attach database
    USE [master]
    GO

    CREATE DATABASE [SQLDBPool1] ON
    ( FILENAME = N’C:\SQLDBPool.mdf’ ),
    ( FILENAME = N’C:\SQLDBPool_Log.ldf’ )
    FOR ATTACH
    GO

    USE SQLDBPool
    GO

    — Get Fragmentation info for each non heap table in SQLDBPool database
    — Avg frag.in percent is External Fragmentation (above 10% is bad)
    — Avg page space used in percent is Internal Fragmention (below 75% is bad)

    SELECT OBJECT_NAME(dt.object_id) AS ‘Table Name’ , si.name AS ‘Index Name’,
    dt.avg_fragmentation_in_percent, dt.avg_page_space_used_in_percent
    FROM
    (SELECT object_id, index_id, avg_fragmentation_in_percent, avg_page_space_used_in_percent
    FROM sys.dm_db_index_physical_stats (DB_ID(‘SQLDBPool’), NULL, NULL, NULL, ‘DETAILED’)
    WHERE index_id <> 0) AS dt
    INNER JOIN sys.indexes AS si
    ON si.object_id = dt.object_id
    AND si.index_id = dt.index_id
    ORDER BY OBJECT_NAME(dt.object_id)

    — Script to Get Fragmention information for a single table
    SELECT TableName = object_name(object_id), database_id, index_id, index_type_desc, avg_fragmentation_in_percent, fragment_count, page_count
    FROM sys.dm_db_index_physical_stats (DB_ID(N’SQLDBPool’), OBJECT_ID(N’mySQLDBPool.Emp’), NULL, NULL , ‘LIMITED’);

    –script to get the index information
    exec sp_helpindex [mySQLDBPool.Emp]

    –Script to Reorganize an index
    ALTER INDEX PK_ProductPhoto_ProductPhotoID ON Production.ProductPhoto
    REORGANIZE
    GO

    — Rebuild an index (offline mode)
    ALTER INDEX ALL ON Production.Product
    REBUILD WITH (FILLFACTOR = 80, SORT_IN_TEMPDB = ON,STATISTICS_NORECOMPUTE = ON);

    –Script to find which columns don’t have statistics
    SELECT c.name AS ‘Column Name’
    FROM sys.columns AS c
    LEFT OUTER JOIN sys.stats_columns AS sc
    ON sc.[object_id] = c.[object_id]
    AND sc.column_id = c.column_id
    WHERE c.[object_id] = OBJECT_ID(‘mySQLDBPool.Emp’)
    AND sc.column_id IS NULL
    ORDER BY c.column_id

    — Create Statistics on DOB column
    CREATE STATISTICS st_BirthDate
    ON mySQLDBPool.Emp(DOB)
    WITH FULLSCAN

    — When were statistics on indexes last updated
    SELECT ‘Index Name’ = i.name, ‘Statistics Date’ = STATS_DATE(i.object_id, i.index_id)
    FROM sys.objects AS o WITH (NOLOCK)
    JOIN sys.indexes AS i WITH (NOLOCK)
    ON o.name = ‘Emp’
    AND o.object_id = i.object_id
    ORDER BY STATS_DATE(i.object_id, i.index_id);

    — Update statistics on all indexes in the table
    UPDATE STATISTICS mySQLDBPool.Emp
    WITH FULLSCAN


    — Script to shrink database
    DBCC SHRINKDATABASE(N’SQLDBPool’ )
    GO

    — Shrink data file (truncate only)
    DBCC SHRINKFILE (N’SQLDBPool_Data’ , 0, TRUNCATEONLY)
    GO

    — Script to shrink Shrink data file – Very Slow and Enhances the fragmentation
    DBCC SHRINKFILE (N’SQLDBPool_Data’ , 10)
    GO
    — Script Shrink transaction log file
    DBCC SHRINKFILE (N’SQLDBPool_Log’ , 0, TRUNCATEONLY)
    GO

    — Script to create view
    CREATE VIEW emp_view
    AS
    SELECT *
    FROM mySQLDBPool.emp

  • MySQL Replication

    Problem/Error

    Could not parse relay log event entry. The possible reasons are: the master’s binary log is corrupted (you can check this by running ‘mysqlbinlog’ on the binary log), the slave’s relay log is corrupted (you can check this by running ‘mysqlbinlog’ on the relay log), a network problem, or a bug in the master’s or slave’s MySQL code. If you want to check the master’s binary log or slave’s relay log, you will be able to know their names by issuing ‘SHOW SLAVE STATUS’ on this slave

    Resolution Steps

    You have to follow below steps to troubleshoot the error.

    Execute the below command

    SHOW MASTER STATUS

    SHOW SLAVE STATUS

    Check the error log for replication and its position.

    Ideally there are three sets of file/position coordinates in SHOW SLAVE STATUS to identify the correct file

    1) The position, ON THE MASTER, from which the I/O thread is reading: Master_Log_File/Read_Master_Log_Pos.

    2) The position, IN THE RELAY LOGS, at which the SQL thread is executing: Relay_Log_File/Relay_Log_Pos

    3) The position, ON THE MASTER, at which the SQL thread is executing: Relay_Master_Log_File/Exec_Master_Log_Pos

    Next you have to check the error log for the log position to identify the correct binary log file and set the correct log file using below command.

    CHANGE MASTER TO MASTER_LOG_FILE=’mysql-bin.000480′

    Once the problem is resolved you can use Maatkit tool to sync table to multiple slaves.

    mk-table-checksum command is used to check what tables are out of sync and when use mk-table-sync command is used to resync them.

  • DRBD, Heartbeat and MySQL

    The easiest solution to implement clustering in MySQL is DRBD and Heartbeat.

    DRBD: The Distributed Replicated Block Device (DRBD) is a software-based, shared-nothing, replicated storage solution mirroring the content of block devices (hard disks, partitions, logical volumes etc.) between servers.

    DRBD mirrors data

    • In real time. Replication occurs continuously, while applications modify the data on the device.
    • Transparently. The applications that store their data on the mirrored device are oblivious of the fact that the data is in fact stored on several computers.
    • Synchronously or asynchronously. With synchronous mirroring, a writing application is notified of write completion only after the write has been carried out on both computer systems. Asynchronous mirroring means the writing application is notified of write completion when the write has completed locally, but before the write has propagated to the peer system

    You can download DRDB from below site

    http://www.drbd.org/download/packages/

  • Memcached & MySQL

    memcached (pronunciation: mem-cash-dee.) is a general-purpose distributed memory caching system that was originally developed by Danga Interactive for LiveJournal, but is now used by many other sites. It is often used to speed up dynamic database-driven websites by caching data and objects in memory to reduce the number of times an external data source (such as a database or API) must be read. Memcached is distributed under a permissive free software license. Memcached lacks authentication and security features, meaning it should only be used on servers with a firewall set up appropriately. By default, memcached uses the port 11211. Among other technologies, it uses libevent. Memcached’s APIs provides a giant hash table distributed across multiple machines. When the table is full, subsequent inserts cause older data to be purged in least recently used (LRU) order. Applications using memcached typically layer memcached requests and additions into core before falling back on a slower backing store, such as a database. You can download memcached API from http://www.danga.com/memcached/