今日ご紹介するのはひらがなカタカナを判別できる新しい「かなセンシティブ」なCollation utf8mb4_ja_0900_as_cs_ksです。 …[Read more]
MySQL 8.0.1では、utf8mb4の大文字小文字およびアクセント記号付きの文字を判別するas_cs collationに加えて、日本語用のCollation(文字照合順)を追加しました。
日本語に関する文字照合およびソートのルールは複雑です。日本語ではひらがな、カタカナ、漢字、アルファベット(ラテン文字)を混在させて利用しています。さらに、全角と半角が存在する文字もあります。では、‘あ’, ‘ア’, ‘a’, ‘ｱ’はどのようにソートされるのでしょうか？
Unicode照合アルゴリズム(UCA / Unicode Collation …[Read more]
In my previous post, I wrote about the new Japanese collation for utf8mb4 introduced in MySQL 8.0.1! This collation (utf8mb4_ja_0900_as_cs) implements accent / case sensitivity for Japanese as defined by CLDR 30.
Today, I am writing about our new utf8mb4_ja_0900_as_cs_ks collation which includes support for kana sensitivity.…
In MySQL 8.0 our plan is to drastically improve support for utf8. While utf8 support itself dates back to MySQL 4.1, there exist some limitations. The “sushi = beer” problem in the title refers to Bug #76553. Sushi and beer don’t even go well together, at least not to my taste:-) I will use this bug as an example to explain issues with utf8 collations in the past and our plans for utf8 support going forward.…
기본 InnoDB 전문 검색(Full Text) 파서는 공백이나 단어 분리자가 토큰인 라틴 기반 언어들에서는 이상적이지만 개별 단어에 대한 고정된 구분자가 없는 중국어, 일본어, 한국어(CJK)같은 언어들에서는 각 단어는 여러개의 문자들의 조합으로 이루어집니다. 그래서 이경우엔 단어 토큰들을 처리할 수 있는 다른 방법이 필요합니다.[Read more]
I have seen a few people now ask about using MySQL's FULLTEXT
indexing with asian languages such as Chinese, Japanese and
Korean (herein referred to as CJK.), however, there doesn't seem
to be a good centralised article that covers it.
The information is out there, I just don't think it has been well presented yet.
As I have recently done a bunch of research on this topic for a customer, I figured it might be a good opportunity to make my debut in the MySQL blogosphere.
So here we go...
I'll open by saying that attempting to use FULLTEXT with CJK text in MySQL 5.0 will be unsuccessful.
From the CJK FAQ in the MySQL manual:
"For FULLTEXT searches, we need to know where words begin and end. With Western languages, this is rarely a problem because most (if not all) of these use an easy-to-identify word boundary — the space character. However, this is not …