Home |  MySQL Buzz |  FAQ |  Feeds |  Submit your blog feed |  Feedback |  Archive |  Aggregate feed RSS 2.0 English Deutsch Español Français Italiano 日本語 Русский Português 中文
Showing entries 1 to 30 of 41 Next 11 Older Entries

Displaying posts with tag: bash (reset)

MySQL backup and cleanup bash scripts with mydumper
+1 Vote Up -0Vote Down

1. Backup script

#!/bin/sh
# Fri Jun 27 10:44:49 2014
# done by dragkh
# usage: 
# cat /etc/cron.d/backupmysql 
# 0  3  *  *  *       root    /root/bin/clean.backup.hyperion.mysql.mydumper.daily.sh >>  /var/log/clean.backup.${HOSTNAME}.mysql.mydumper.daily.log 2>&1
# 35  3  *  *  *       root    /root/bin/backup.hyperion.mysql.mydumper.daily.sh >> /var/log/backup.${HOSTNAME}.mysql.mydumper.daily.log 2>&1

ROOT_BACKUP_DIR="/home/mydumper"

seik_date () {
if [ -z $1 ]
then
# cdate=`date +%Y-%m-%d\ %H:%M:%S\ %Z`; export cdate; echo $cdate
cdate=`date -R`; export cdate; echo $cdate
else

if [ -z ${2} ]
then
cdate=`date +%Y-%m-%d.%H.%M.%S`; export cdate; echo $cdate
else
cdate=`date "+%Y-%m-%d %H:%M:%S"`; export cdate; echo $cdate
fi

fi
}


function check_dir {
 test ! -d
  [Read more...]
Compile at CentOS 6.5 the new MySQL webscalesql-5.6.17 branch by Facebook, Google, LinkedIn, and Twitter
+1 Vote Up -0Vote Down

http://webscalesql.org/

yeah , big buzz around that one :)

So I decided to check the install process:

1. Clone the repo from

root@webscalesql-5.6.clean:[Mon Mar 31 11:37:11][~]$ cd /opt/
root@webscalesql-5.6.clean:[Mon Mar 31 11:37:15][/opt]$ mkdir installs
root@webscalesql-5.6.clean:[Mon Mar 31 11:37:17][/opt]$ cd installs/
root@webscalesql-5.6.clean:[Mon Mar 31 11:37:19][/opt/installs]$ git clone https://github.com/webscalesql/webscalesql-5.6.git
Initialized empty Git repository in /opt/installs/webscalesql-5.6/.git/
remote: Counting objects: 30397, done.
remote: Compressing objects: 100% (12678/12678), done.
remote: Total 30397 (delta 18716), reused
  [Read more...]
How to store MySQL innobackupex backups at Google Cloud Storage
+0 Vote Up -0Vote Down

In general, I chose Google Cloud Storage to store web sites MySQL backups due to its price and speed of upload/download in real time

I used the Google native tool – gsutil , innobackupex and some bash

in short : the /etc and local MySQL  backup

#!/bin/sh
# Barcelona Tue Nov 22 17 16:30:36 CEST 2013

days_to_keep=3
NFS=/home/mysql.backups/
exportDate=`date +%Y-%m-%d.%H.%M.%S`
export_DIR=${NFS}/${HOSTNAME}.${exportDate}
test ! -d "${export_DIR}" && echo "$(date) : creating ${export_DIR}" && mkdir -p
  [Read more...]
Daily backup OpenStack single MySQL with Percona innobackupex including the /etc directory
+0 Vote Up -0Vote Down

this is a short script for daily backup of the OpenStack MySQL and the /etc direcotry of the control node

[root@dev-epg-rhos-01 BACKUP]# cat /root/bin/epg.innobackupex.openstack.sh
#!/bin/sh
# done for epgmad4@tid.es
# Barcelona Thu Oct 17 16:30:36 CEST 2013

days_to_keep=7
NFS=/BACKUP
exportDate=`date +%Y-%m-%d.%H.%M.%S`
export_DIR=${NFS}/${HOSTNAME}.${exportDate}
test ! -d "${export_DIR}" && echo "$(date) : creating ${export_DIR}" && mkdir -p "${export_DIR}"
export_MySQL_DIR=${export_DIR}/mysql.bckp
export_ETC_DIR=${export_DIR}/etc.bckp
rsync -avh /etc ${export_ETC_DIR}
echo "=========================================================================================================" >> ${export_DIR}/README.restore.with.innobackupex
  [Read more...]
Start
+0 Vote Up -0Vote Down

So lets see how this goes.

Migrating MySQL 5.5.25a jiradb ERROR 2013 (HY000) on huge single db import
+0 Vote Up -0Vote Down

well, I incremented max_allowed_packet from 16M to 512M
anyway, I got the same error on the next clean import.
so decided to find a workaround.
so, how to get the data separated from the ddl statements:

# get the tables names into the insert statement, its better to have that in file for future usage
sed '/^INSERT INTO/!d;s/ VALUES.*$//' jiradb.20130118.sql | sort | uniq > tablas.como.nombres.txt

then how to get the data separated:

root@jiragg:[Fri Jan 18 15:26:33]:[/usr/local/BACKUP]$ cat make.inserts.sh
#!/bin/sh
# trim function thank to http://stackoverflow.com/questions/369758/how-to-trim-whitespace-from-bash-variable
# and http://codesnippets.joyent.com/posts/show/1816
trim() {
    local var=$1
    var="${var#"${var%%[![:space:]]*}"}"   # remove



  [Read more...]
Install MySQL at non default data directory on different drive on CentOS6.3/RHEL 6
+0 Vote Up -0Vote Down

We assume the new disk is /dev/sdb1 formatted as ext3
and it will be mounted as /data

# 0. make sure there is no mysqlm mysql data directory :
yum remove mysql mysql-server -y 
test -d /data/mysql/ && rm -rf /data/mysql/
test -d /var/lib/mysql/ && rm -rf /var/lib/mysql/

# 1. install Mysql
yum install mysql mysql-server -y

# 2. check the mysql  status
service mysqld status

# 3. start the mysqld if not started
service mysqld start

# 4. check the mysql status again
service mysqld status

# 5. stop mysqld in case its started, and check thre is n mysql process:
service mysqld stop
ps axu | grep mysql

# 6. make sure the /data partition is added to the /etc/fstab. If not add it:
test  `cat  /etc/fstab | grep /data | wc -l ` -eq 0  && echo "/dev/sdb1 /data ext3

  [Read more...]
Compile MariaDB 5.5.28 MySQL on Slackware i486-slackware-linux-gcc-3.3.4 with Cmake 2.8.10.1
+0 Vote Up -0Vote Down

1. install Cmake

wget  http://www.cmake.org/files/v2.8/cmake-2.8.10.1.tar.gz
tar xvfz cmake-2.8.10.1.tar.gz
cd cmake-2.8.10.1/
./configure
gmake -j3


2. install libaio

gmake -wget  http://www.kernel.org/pub/linux/kernel/people/bcrl/aio/libaio-0.3.92.tar.gz
tar xvfz libaio-0.3.92.tar.gz
cd libaio-0.3.92/
make prefix=/usr/
make prefix=/usr/ installj3 install

3. install MariaDB 5.5.28
go to https://downloads.mariadb.org/mariadb/5.5.28/ OR

wget https://downloads.mariadb.org/f/mariadb-5.5.28/kvm-tarbake-jaunty-x86/mariadb-5.5.28.tar.gz/from/http:/mariadb.ulak.net.tr/
tar xvf mariadb-5.5.28.tar.gz
cd mariadb-5.5.28/

the build will fail :

root@bubu:[Tue Nov 27

  [Read more...]
OpenCode: MySQL procedures + python + shell code repositories now public
+1 Vote Up -0Vote Down

I write a fair number of scripts on this site and have posted a lot of code over the years. Generally if I am not pasting the code to be viewed on the webpage then I link to a file that a user can download; which leads to a lot of mish-mash code that doesn’t have a home. I’ve always kept the code files in a private SVN repo over the years but have recently moved them all to BitBucket Git repositories. So here they are: lots of code samples and useful bits of programming to save time.

Generic Shell Scripts: https://bitbucket.org/themattreid/generic-bash-scripts/src
Generic Python Scripts: https://bitbucket.org/themattreid/generic-python-scripts/src
Generic MySQL Stored Procs:

  [Read more...]
Simple bash shell script for running batch MySQL jobs
+1 Vote Up -0Vote Down

The other day I needed to run a simple mysql job to backup and delete some database records on a live server. Being a live server, it is important to make sure you aren't asking the database to take on jobs that could potentially lock it up. Better to run a batch job. Running a batch is simple. You can call it right from the mysql console with:

source [path_to]/[the_batch_script].sql

But what if there are millions of records that need deleting? Bash shell script to the rescue.

Here is the idea of the SQL job that needed to get run a few times:

START TRANSACTION;

/* Find what you want to delete and put a LIMIT on your batch size */
CREATE TEMPORARY TABLE records_to_delete_temp SELECT id from `records` where ..... limit 1000;

/* Creating backup table to archive spam orders */
CREATE TABLE IF NOT EXISTS `records_backup` LIKE
  [Read more...]
Mailbox conversion
+0 Vote Up -2Vote Down

Converting from uw-mailboxes (mbx) to Unix format (dovecot)

It took me by surprise how the mailbox formats had changed, when I switched to Dovecot which is the best IMAP/POP3 mail program, in my opinion. It handles large (enormous) mailboxes with many (hundreds of) users. Caching makes things go fast again.

Here is a simple bash utility, to take all users, and convert all mailboxes & folders.

cd /home
for u in *
do
if [ -d /home/$u/Mail ]; then
echo "User: $u"
cd /home/$u/Mail

read more

Water the lawn if temperature exceeded 80 degrees
+0 Vote Up -0Vote Down

I read somewhere that you should do extra watering of lawn (after sunset), if the temperature exceeded 80° F during the day.

So, a really quick hack works like this:

A bash script
#!/bin/bash

TEMP="`/usr/bin/mysql -ss open2300 -e \"SELECT FLOOR(MAX(temp_out)) FROM weather GROUP BY (rec_date) ORDER BY rec_date DESC LIMIT 1;\"`"
if [ "$TEMP" -ge 80 ]; then
/root/water-on.exp
/root/water2.exp 120 1
/root/water3.exp 120 1
fi

read more

Scripting continued
+0 Vote Up -0Vote Down

Since I have been discussing scripting lately I thought I would continue with another topic I touched on briefly - backups.

I have written and modified the following script over the last few years. I have used it (and continue to use it) with multiple clients. It uses Percona's Xtrabackup to take the backup (although it can be easily modified to use mysqldump instead).

First the script

-----------------------------------------------------------------------------------------------------------------

#!/bin/bash
SAVEIFS=$IFS
IFS=$""
day_of_week=`date +%a`
backup_dir=/mysql-backup/
logfile=/root/backup_log.txt
report=/tmp/report.txt
servername=slave1
email=bmurphy@paragon-cs.com
password=`cat /root/.ssh/.backup_password`

# run backup
echo ' ' > $report
echo 'The backup is now beginning:'












  [Read more...]
Interviewing tip..
+0 Vote Up -0Vote Down

I've been involved in a number of interviews over the last few weeks as a client has been loking for a MySQL DBA. When you are looking for position as a DBA in a large scale environement there are some very important things you have to know.

You absolutely must know a scripting language. In a smaller environment this often isn't necessary. You will live and die by this in a large environment. I asked every applicant one specific question..if you had to change a mysql server variable on a pool of 100 mysql servers how would you do this? It's easy when it's one,two or even a dozen servers. just log in, change the my.cnf and change it "on the fly" if you can. Restart mysql if you can't.

You going to do that to 100 servers? It will take all day and be prone to failures. Scripting is the key here. Even just bash shell scripting can be very powerful. In another

  [Read more...]
Simple and efficient MongoDB Backup using script
+0 Vote Up -1Vote Down

MongoDB Backup types and strategies are neatly explained in its documentation, which you can check here. In case you are not familiar with MongoDB backup types and strategies, please have a look at its documentation.

What I am describing here is a simple script which we are using since months to take MongoDB backup and transfer it over to our Backup server. Here are few things its doing:

  • As we have multiple MongoDB Replica Sets, the script identify current replica set and check whether current server is Master or Slave, exit if its Master. We take backup only from Slave host.
  • Take Backup using mongodump command.
  • Upon successful completion of dump, transfer that to our Backup server. Ensure that ssh key based authentication is setup between both servers to implement seamless and
  [Read more...]
Fun with Bash :: one liners
+1 Vote Up -0Vote Down

Here are some quick and easy bash commands to solve every day problems I run into. Comment and leave some of your own if you like. I might update this post with new ones over time. These are just some common ones.

Iterate through directory listing and remove the file extension from each file
ls -1 | while read each; do new=`echo $each |sed 's/\(.*\)\..*/\1/'` && echo $new && mv "$each" "$new"; done

Output relevant process info, and nothing else
ps axo "user,pid,ppid,%cpu,%mem,tty,stime,state,command"| grep -v "grep" | grep $your-string-here

Setup a SOCKS5 proxy on localhost port 5050, to tunnel all traffic through a destination server
ssh -N -D 5050 username@destination_server'

Setup a SOCKS5 proxy via a remote TOR connection, using local port 5050 and remote TOR port 9050
ssh -L 5050:127.0.0.1:9050 username@destination_server'

Display text or code file
  [Read more...]
How do we control MySQL daemon in Linux, part1
+0 Vote Up -0Vote Down

As you may expect from open source world thingy, almost every Linux distribution has developed it’s own way to manage our favourite RDBMS service. Yet none is perfect, or even some of them seems to not work in real server scenario1.

In this post I’m trying to compare and point out most annoying aspects of initialization scripts that I had to face in production.

In ‘old days’ probably all Linux distributions used to start and stop services using so called init scripts usually written in Unix shell (sh or Bash). But situation is not so simple these days anymore.

Folks started to think about improving things, like making

  [Read more...]
Alias shortcuts to MySQL CLI
+3 Vote Up -1Vote Down

Do you get write laziness in the command line everything what you need to connect to a MySQL server every time?

It may take less than minute, but sometimes one minute is vital (especially if we’re near the end of the world):

ivancp@ubuntu$ mysql -u root -p -h mysqlhost database

When we are hurry, these commands often fail several times per minute.

The solution: we can create shortcuts with bash alias commands in file ~/.bashrc :

# File ~ /. Bashrc
 
# Command "my" to connect to a local server
alias my='mysql -u root -p'
 
# Command "my2" to connect to a remote server
alias my2='mysql -u root -h 192.168.1.56 -p'

Next time if you want to access the local server just type the command my [database

  [Read more...]
Linux bash: check the MySQL database disk usage
+0 Vote Up -0Vote Down
#!/bin/sh
# get.database.sizes.sh
# by dragkh
# Wed, 02 Mar 2011 17:59:44 +0100
# it will work on linux boxes with working mysql 
ps axu \
| grep datadir \
| grep mysql \
| grep -v grep \
| grep var \
| sed 's/^.*--datadir=//; s/ .*$//'   \
| sort \
| uniq  \
| while read crap
do 
echo ""
sized=$(du -s $crap | awk '{print $1}')
avail=$(df | grep $(dirname $crap) | awk '{print $2}')
echo -e  "$crap\t$sized\t$avail" | awk '{printf ("%s:\tUsed: %6.2fG \tDISK: %6.2fG\t Usage: %6.2f%% \n",$1,$2/1024/1024,$3/1024/1024,($2/$3)*100)}'
find $crap/ -maxdepth  1 -type d ! -type l  ! -path "$crap/"   -printf '"%p"\n'  | xargs  --no-run-if-empty  du -s | sort -nk 1,9 | awk -vavail=$avail '{crapy="";for(i=2; i<=NF; i++) {crapy=crapy" "$i;};printf (" -- %s\tUsed: %6.2fG
  [Read more...]
Fun with Bash: aliases make your live easier… share your favorites
+2 Vote Up -0Vote Down

I’ve always been a big fan of having a customized .bashrc file. The one I distribute to all of my servers has aliases for quick commands to save me time on the command line, functions that get work done when aliases are too simplistic, reporting for the server for each cli login, and of course a formatted and colored prompt (for terms that support colors). I also change certain aspects and commands based on the operating system since I’m not always on a redhat box or linux at all. Here’s my bashrc file – maybe you have some fun additions that you’d like to share. What saves you time on the command line?

[Bash] Performing array intersection with Bash
+0 Vote Up -0Vote Down

I am currently working on a project to deploy new website builds to a
small number of servers. I needed something simple and reliable that could
be built in a very short period of time. I decided to whip something up in
bash with the intent of refining it in Python later.

As I began to write this code, I realized that it probably would have been
quicker to do it in Python from the start. I decided to stick with bash as
somewhat of an academic exercise. The vast majority of these deployment
scripts were trivial; check the code out of git, create a manifest, package
it up, spew it to the servers, etc, etc. The problem came during the last
step. We decided to use a symlink to point to the active build out of a
number of builds that could be available on the server at any given time.
Since all of our servers should be









  [Read more...]
Easy MySQL: how to backup databases to a remote machine
+3 Vote Up -1Vote Down

Here’s a simple answer to a simple question. “How do I run a backup of MySQL to another machine without writing to the local server’s filesystem?” – this is especially useful if you are running out of space on the local server and cannot write a temporary file to the filesystem during backups.

Method one – this writes a remote file.
mysqldump [options] [db_name|--all-databases]| gzip -c | ssh user@host.com "cat > /path/to/new/file.sql.gz"

Method two – this writes directly into a remote mysql server
mysqldump [options] [db_name|--all-databases]| mysql --host=[remote host] –user=root –password=[pass] [db_name]

How to: rotate wordpress posts into headline/feature status
+0 Vote Up -0Vote Down

If you’re using the new Arthemia theme for WordPress you might notice that there are two areas of the theme that can have articles promoted to; namely Headline and Featured sections. This is controlled by category association. Basically you have a post and if you want it in the Headline area of the theme you attach the category “headline” to it, similarly for the featured section. Now, let’s say you don’t want to manually change this all the time since it can be time consuming to promote posts to those categories if you want rotating content.

Here’s a simple solution. In this bash script I connect to MySQL and remove the current associations from posts and then randomly choose posts to be promoted to the Headline and Featured categories. This can be

  [Read more...]
N900 – control all of your accounts with this script
+1 Vote Up -1Vote Down

If you own a Nokia N900 cellular device you might be interested in the ability to control all of your IM accounts from the command line. For those that do not know, the N900 runs Maemo Linux and is capable of running MySQL embedded if you so choose. Here’s a quick script I wrote to provide that functionality for IM accounts. It’s at the bottom of the page, called “im-connections”.

wiki: http://wiki.maemo.org/N900_Mission_Control#Set_all_SIP_accounts_to_online_or_offline
pastebin: http://pastebin.com/qAC57E1N

Backup large databases with mysqldump by splitting up the backup files
+0 Vote Up -0Vote Down
The primary responsibility of MySQL professionals is to establish and run proper backup and recovery plans. The most used method to backup a MySQL database is the mysqldump utility. This mysqldump utility creates a backup file for one or more MySQL databases that consists of DDL/DML statements needed to recreate the databases with their data. To [...]
How to get colored output from ‘ls’ on Solaris10
+0 Vote Up -3Vote Down

For all of those linux users out there that have moved over to, or tried out, Solaris10 or OpenSolaris because they heard the tales of how MySQL is faster on Solaris… or perhaps you wanted to learn how to use Sol10 for the great features of Zones or the ZFS filesystem? Regardless of why you’re on it you are probably wondering why Linux has colored output of filenames and directories but Solaris does not. The question of ‘why?’ isn’t important, but how to enable colors is. It’s very simple, and here’s how I fixed it. This is a result of digging through multiple semi-related links on Google.

  • Download all packages from SunFreeware.com
    • dependency: libintl-3.4.0-sol10-x86-local
    • dependency: libiconv-1.13.1-sol10-x86-local
    • dependency: gmp-4.2.1-sol10-x86-local
    • dependency: gcc-3.4.6-sol10-x86-local or
      [Read more...]
  • Kontrollkit – new version is available for download!
    +0 Vote Up -0Vote Down
    Just a quick notice to let everyone know that there is a new version of Kontrollkit available. There are two new scripts included as well as some good updates to the my.cnf files. You can download the new version here: http://kontrollsoft.com/software-downloads kt-mysql-systemcheck – generates a report for point-in-time system status that is useful for troubleshooting MySQL [...]
    Automating MySQL access with expect and bash scripting
    +2 Vote Up -2Vote Down

    If you have multiple database servers with strange names, or if you have to hop over multiple machines to connect to any mysql database server, then you know what a pain it can be to administer such a setup. Thanks to some scripting, you can automate such tasks as follows:

    Create an expect script:
    /path/to/sshmysql.exp

    #!/usr/bin/expect -f
    #script by darren cassar
    #mysqlpreacher.com

    set machine [lindex $argv 0]

    set timeout -1

    spawn ssh username@$machine
    match_max 100000
    expect -exact “assword: “
    send — “password\r”
    send — “sudo -k; sudo su – mysql\r”
    expect -exact “sudo -k; sudo su – mysql”
    expect -exact “assword:”
    send — “password\r”











      [Read more...]
    Hidden Features Of Perl, PHP, Javascript, C, C++, C#, Java, Ruby, Python, And Others [Collection Of Incredibly Useful Lists]
    +2 Vote Up -0Vote Down

    Introduction

    StackOverflow is an amazing site for coding questions. It was created by Joel Spolsky of joelonsoftware.com, Jeff Atwood of codinghorror.com, and some other incredibly smart guys who truly care about user experience. I have been a total fan of SO since it went mainstream and it's now a borderline addiction (you can see my StackOverflow badge on the right sidebar).

    The Story

    Update 6/21/09: This server is currently under very heavy load (10-200), even with caching plugins enabled. Please bear with me as I try to

      [Read more...]
    Replicating from MySQL to *
    +3 Vote Up -0Vote Down
    Recently I needed to replicate between MySQL and another database technology. You might say, why on earth would you want to do something like that, but believe me there are reasons and definitely not (to go away from MySQL to some other DB technology like Oracle or SQL server). Unsurprisingly there are quite a few different tools to do it from any platform towards MySQL but very few which do it the other way round, just to name a couple: Golden Gate and DSCallards.
    Showing entries 1 to 30 of 41 Next 11 Older Entries

    Planet MySQL © 1995, 2014, Oracle Corporation and/or its affiliates   Legal Policies | Your Privacy Rights | Terms of Use

    Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.