As I mentioned on my last post, where I compared the default configurations options in 5.6 and 5.7, I have been doing some testing for a particular load in several versions of MySQL. What I have been checking is different ways to load a CSV file (the same file I used for testing the compression tools) into MySQL. For those seasoned MySQL DBAs and programmers, you probably know the answer, so you can jump over to my 5.6 versus 5.7 results. However, the first part of this post is dedicated for developers and MySQL beginners that want to know the answer to the title question, in a step-by-step fashion. I must say I also learned something, as I under- and over-estimated some of the effects of certain …[Read more]
This week we are talking about size, which is a subject that should matter to any system administrator in charge of the backup system of any project, and in particular database backups.
I sometimes get questions about what should be the best compression tool to apply during a particular backup system: gzip? bzip2? any other?
The testing environment
In order to test several formats and tools, I created a .csv file (comma-separated values) that was 3,700,635,579 bytes in size by transforming a recent dump of all the OpenStreetMap nodes of the European portion of Spain. It had a total of 46,741,126 rows and looked like this:
171773 38.6048402 -0.0489871 4 2012-08-25 00:37:46 12850816 472193 rubensd 171774 38.6061981 -0.0496867 2 2008-01-19 10:23:21 666916 9250 …[Read more]
Ever come accoss a situation where you have to load a flat text file into MySQL with data provided in single column! Say, your data to be loaded in table looks something like this: Company1 Street Name City, MyState 8582 (999)999-999 another_Company Another Street 2 New City, NWSTATE 8582 (111)111-1111 So how’d you go about […]
Okay! So here we will use Load Data syntax to load file into MySQL Server in a Stored procedure. Yep! It’s a workaround. Download MySQL UDF: [root@localhost kedar]# wget http://www.mysqludf.org/lib_mysqludf_sys/lib_mysqludf_sys_0.0.3.tar.gz [refer: http://www.mysqludf.org/] Extract and Install: [root@localhost kedar]# tar -xzvf lib_mysqludf_sys_0.0.3.tar.gz install.sh lib_mysqludf_sys.c lib_mysqludf_sys.html lib_mysqludf_sys.so lib_mysqludf_sys.sql Makefile [root@localhost kedar]# sh install.sh Compiling the MySQL UDF gcc […]
When you are importing large amounts of data from other sources
LOAD DATA is a common method of inserting data
into a table.
It is one of the old commands implemented in MySQL. As such it is very fast, and it has been optimized for both MyISAM and InnoDB.
All is well when you are loading data into a standalone server. All is almost well when you are using replication. LOAD DATA used to be a problem in old versions of MYSQL, prior to 4.1. With recent versions of MySQL, it is replicated correctly, and sometimes efficiently.
The trouble starts when the data file is big. The definition of big and the amount of trouble that you can get depends on many factors. That's why users may not realize that this problem exists, even with fairly large files, and then being hit by this disaster when the file is only a little larger than the previous …
Loading csv to MySQL or any delimited data files to MySQL database is a very common task frequently questioned about and almost every-time LOAD DATA INFILE come into rescue. In this article I will explain you about all different scenarios and explain you how to get the data loaded directly from csv to MySQL database […]
Having already blogged about loading data from flat files to MySQL, it's time to post a similar case for PostgreSQL, as the manual seems to lack a real life example ...First of all the table to be loadedCREATE TABLE target(code character(3) NOT NULL,"name" character varying(50) NOT NULL,amount numeric,CONSTRAINT pk_1 PRIMARY KEY (code))WITH (OIDS=FALSE);as you can see one of the column names is a