Here are some observations from writing web-crawling robots.

Intro

At some point, many of us (in the IT security industry) will need to write a robot which scrapes lots of web sites. By "lots", I mean a very large number, run by arbitrary parties. Not just a few run by well-behaved, cooperative entities.

Most owners of web servers try to make them compatible - but this is not guaranteed. Even with the best of intentions, we'll probably find things which go wrong.

Behaviour observed
Faulty DNS
* Returns too large responses
* Returns private addresses in "A" responses

Server hangs / timeout
* Connection timeout
* Timeout waiting for response
* Connection hang during headers or response

Bad responses
* Connection closed after request
* Connection closed while transmitting headers
* Connection closed …

The first section in the exam topics for both the MySQL 5.6 Developer and DBA exam deals is titled MySQL Architecture, but each has a different emphasis.  This blog will focus on those objectives listed for the Developer exam:

• Use MySQL client programs to interface with the MySQL Server interactively and in batch
• Describe SQL Modes and their impact on behavior of MySQL
• Identify characteristics which have session scope

For those of us who have spent time pulling both DBA and developer duties, this section is likely to be relatively straight-forward – perhaps even easy.  I’ll walk through the exercises and documents I’ve used to review below to kick off my Exam Cram blog series.

Using MySQL Client Programs

My first step …

this is a short script for daily backup of the OpenStack MySQL and the /etc direcotry of the control node

[root@dev-epg-rhos-01 BACKUP]# cat /root/bin/epg.innobackupex.openstack.sh
#!/bin/sh
# done for epgmad4@tid.es
# Barcelona Thu Oct 17 16:30:36 CEST 2013

days_to_keep=7
NFS=/BACKUP
exportDate=`date +%Y-%m-%d.%H.%M.%S`
export_DIR=${NFS}/${HOSTNAME}.${exportDate}
test ! -d "${export_DIR}" && echo "$(date) : creating ${export_DIR}" && mkdir -p "${export_DIR}"
export_MySQL_DIR=${export_DIR}/mysql.bckp
export_ETC_DIR=${export_DIR}/etc.bckp
rsync -avh /etc ${export_ETC_DIR}
echo "=========================================================================================================" >> ${export_DIR}/README.restore.with.innobackupex
echo "HOW to restore this FULL mysql backup" >> ${export_DIR}/README.restore.with.innobackupex
echo "=========================================================================================================" >> …

We are looking for community feedback on the use-cases for the Federated Storage Engine in MySQL.

Specifically, I would be interesting to hear which users find that the introduction of multi-source replication meets (or does not meet) their requirements.

For a bit of background:

The Federated Storage Engine was introduced in 5.0, but has long been disabled by default. It is useful for ad-hoc queries across MySQL servers, but it misses some of MySQL’s newer optimizations, and does not perform as well as we would like. Among it’s limitations it also does not support transactions.

By using multi-source replication, many of the downsides mentioned above are negated, since …

When I came from Linux world to Mac OS, I’ve got frustrated in not having a package management system as there are in Ubuntu (with Apt-get) or in CentOS ( YUM ). The world was so fantastic, if I need to install MySQL Database on Ubuntu, I’d just type: $ sudo apt-get install mysql-server On Mac, the […]

The SQL language is a bit like a toolkit for data. It consists of lots of little fiddly bits of syntax that, taken together, allow you to build complex edifices and return powerful results. For the uninitiated, the many tools can be quite confusing, and it's sometimes difficult to decide how to go about the process of building non-trivial queries, that is, queries that are more than a simple SELECT a, b FROM c;

A System for Building Queries

When you're building queries, you could use a system like the following: 

1. Decide which fields contain the values you want to use in our output, and how you wish to alias those fields
   1. Values you want to see in your output
   2. Values you want to use in calculations . For example, to calculate margin on a product, you could calculate price - cost and give it the alias margin.
   3. Values you want …

Checking throughput with async MySQL replication

Replication throughput is the measure of just how fast the slaves can apply replication (at least by my definition).  In MySQL async replication this is important to know because the single-threaded apply nature of async replication can be a write performance bottleneck.  In a production system, we can tell how fast the slave is currently running (applying writes), and we might have historical data to check for the most throughput ever seen, but that doesn’t give us a solid way of determining where we stand right NOW().

An old consulting trick to answer this question is to simply stop replicating on your slave for a minute, (usually just the SQL_THREAD), restart it and watch how long it takes to catch up.  We can also watch the slave thread apply rate during this interval to get a sense of just how many writes per second we can do and compare that with the normal rate …

The MySQL Windows Experience Team is proud to announce the release of MySQL for Excel version 1.1.3, the  latest addition to the MySQL Installer for Windows.

MySQL for Excel 1.1.3 is a maintenance release containing a handful of bug fixes and the following minor features:

• Upon saving a Workbook containing Worksheets in Edit Mode, the user is asked if he wants to exit the Edit Mode on all Worksheets before their parent Workbook is saved so the Worksheets are saved unprotected, otherwise the Worksheets will remain protected and the users will be able to unprotect them later retrieving the passkeys from the application log after closing MySQL for Excel.
• Added background coloring to the column names header row of an Import Data operation to have the same look as the one in an Edit Data operation (i.e. gray-ish background).
• Connection passwords can be …

Devart is glad to announce the release of the new versions of dotConnect ADO.NET data providers, LinqConnect ORM solution, and our ORM designer - Entity Developer. New versions of dotConnect data providers offer you Sync Framework support, support for Oracle 12c, improved PgSqlDump component, load balancing support in dotConnect for MySQL, and other features. Entity Developer and LinqConnect now support Oracle 12c.

Devart is glad to announce the release of the new versions of dotConnect ADO.NET data providers, LinqConnect ORM solution, and our ORM designer - Entity Developer. New versions of dotConnect data providers offer you Sync Framework support, support for Oracle 12c, improved PgSqlDump component, load balancing support in dotConnect for MySQL, and other features. Entity Developer and LinqConnect now support Oracle 12c.