A code point is one entry in the Unicode character database. A code point is not the same as a character, depending on the meaning you give to “character.” What appears as a character on screen is called a grapheme in Unicode.

The Unicode code point U+2122 represents the “trademark sign” character. You can match this with ‹\u2122›, ‹\u{2122}›, or ‹\x{2122}›, depending on the regex flavor you’re working with.

The ‹\u› syntax requires exactly four hexadecimal digits. This means you can only use it for Unicode code points U+0000 through U+FFFF.

‹\u{⋯}› and ‹\x{⋯}› allow between one and six hexadecimal digits between the braces, supporting all code points U+000000 through U+10FFFF. You can match U+00E0 with ‹\x{E0}› or ‹\x{00E0}›. Code points U+100000 and above are used very infrequently. They are poorly supported by fonts and operating systems.

Python’s regular expression engine has no support for Unicode code points. Literal Unicode strings in Python 2.x and literal text strings in Python 3.x do have escapes for Unicode code points. \u0000 through \uFFFF represent Unicode code points U+0000 through U+FFFF. \U00000000 through \U0010FFFF represent all Unicode code points. You have to specify eight hexadecimal numbers after \U, even though there are no Unicode code points beyond U+10FFFF.

When hard-coding regular expressions as literal strings in your Python code, you can directly use ‹\u2122› and ‹\U00002122› in your regexes. When reading regexes from a file or receiving them from user input, these Unicode escapes will not work if you pass the string you read or received directly to re.compile(). In Python 2.x, you can decode the Unicode escapes by calling string.decode('unicode-escape'). In Python 3.x you can call string.encode('utf-8').decode('unicode-escape').

Code points can be used inside and outside character classes.

Each Unicode code point fits into a single Unicode category. There are 30 Unicode categories, specified with a code consisting of two letters. These are grouped into 7 super-categories that are specified with a single letter.

‹\p{Ll}› matches a single code point that is in the Ll, or “lowercase letter,” category. ‹\p{L}› is a quick way of writing ‹[\p{Ll}\p{Lu}\p{Lt}\p{Lm}\p{Lo}]› that matches a single code point in any of the “letter” categories.

‹\P› is the negated version of ‹\p›. ‹\P{Ll}› matches a single code point that is not in the Ll category. ‹\P{L}› matches a single code point that does not have any of the “letter” properties. This is not the same as ‹[\P{Ll}\P{Lu}\P{Lt}\P{Lm}\P{Lo}]›, which matches all code points. ‹\P{Ll}› matches the code points in the Lu category (and every other category except Ll), whereas ‹\P{Lu}› includes the Ll code points. Combining just these two in a code point class already matches all possible code points.

The Unicode character database divides all the code points into blocks. Each block consists of a single range of code points. The code points U+0000 through U+FFFF are divided into 156 blocks in version 6.1 of the Unicode standard:

A Unicode block is a single, contiguous range of code points. Although many blocks have the names of Unicode scripts and Unicode categories, they do not correspond 100% with them. The name of a block only indicates its primary use.

The Currency block does not include the dollar and yen symbols. Those are found in the BasicLatin and Latin-1Supplement blocks, for historical reasons. Both are in the Currency Symbol category. To match any currency symbol, use ‹\p{Sc}› instead of ‹\p{InCurrency}›.

Most blocks include unassigned code points, which are in the category ‹\p{Cn}›. None of the other Unicode categories, and none of the Unicode scripts, include unassigned code points.

The ‹\p{InBlockName}› syntax works with .NET, XRegExp, and Perl. Java uses the ‹\p{IsBlockName}› syntax.

Perl also supports the Is variant, but we recommend you stick with the In syntax, to avoid confusion with Unicode scripts. For scripts, Perl supports ‹\p{Script}› and ‹\p{IsScript}›, but not ‹\p{InScript}›.

The Unicode standard stipulates that block names should be case insensitive, and that any differences in spaces, hyphens, or underscores should be ignored. Most regex flavors are not this flexible, unfortunately. All versions of .NET and Java 4 require the block names to be capitalized as shown in the preceding list. Perl 5.8 and later and Java 5 and later allow any mixture of case. Perl, Java, and .NET all support the notation with hyphens and without spaces used in the preceding list. We recommend you use this notation. Of the flavors discussed in this book, only XRegExp and Perl 5.12 and later are fully flexible with regard to spaces, hyphens, and underscores in Unicode block names.

Each Unicode code point, except unassigned ones, is part of exactly one Unicode script. Unassigned code points are not part of any script. The assigned code points up to U+FFFF are assigned to these 72 scripts in version 6.1 of the Unicode standard:

A script is a group of code points used by a particular human writing system. Some scripts, such as Thai, correspond with a single human language. Other scripts, such as Latin, span multiple languages. Some languages are composed of multiple scripts. For instance, there is no Japanese Unicode script; instead, Unicode offers the Hiragana, Katakana, Han, and Latin scripts that Japanese documents are usually composed of.

We listed the Common script first, out of alphabetical order. This script contains all sorts of characters that are common to a wide range of scripts, such as punctuation, whitespace, and miscellaneous symbols.

Java requires the name of the script to be prefixed with Is, as in ‹\p{IsYi}›. Perl allows the Is prefix, but doesn’t require it. XRegExp, PCRE, and Ruby do not allow the Is prefix.

The Unicode standard stipulates that script names should be case insensitive, and that any differences in spaces, hyphens, or underscores should be ignored. Most regex flavors are not this flexible, unfortunately. The notation with the words in the script names capitalized and with underscores between the words works with all flavors in this book that support Unicode scripts.

The difference between code points and characters comes into play when there are combining marks. The Unicode code point U+0061 is “Latin small letter a,” whereas U+00E0 is “Latin small letter a with grave accent.” Both represent what most people would describe as a character.

U+0300 is the “combining grave accent” combining mark. It can be used sensibly only after a letter. A string consisting of the Unicode code points U+0061 U+0300 will be displayed as à, just like U+00E0. The combining mark U+0300 is displayed on top of the character U+0061.

The reason for these two different ways of displaying an accented letter is that many historical character sets encode “a with grave accent” as a single character. Unicode’s designers thought it would be useful to have a one-on-one mapping with popular legacy character sets, in addition to the Unicode way of separating marks and base letters, which makes arbitrary combinations not supported by legacy character sets possible.

What matters to you as a regex user is that all regex flavors discussed in this book operate on code points rather than graphical characters. When we say that the regular expression ‹.› matches a single character, it really matches just a single code point. If your subject text consists of the two code points U+0061 U+0300, which can be represented as the string literal "\u0061\u0300" in a programming language such as Java, the dot will match only the code point U+0061, or a, without the accent U+0300. The regex ‹..› will match both.

Perl and PCRE offer a special regex token ‹\X›, which matches any single Unicode grapheme. Essentially, it is the Unicode version of the venerable dot. ‹\X› will find two matches in the text àà, regardless of how it is encoded. If it is encoded as \u00E0\u0061\u0300 the first match is \u00E0, and the second \u0061\u0300. The dot, which matches any single Unicode code point, would find three matches as it matches \u00E0, \u0061, and \u0300 separately.

The rules for exactly which combinations of Unicode code points are considered graphemes are quite complicated.^[2] Generally speaking, to match a grapheme we need to match any character that is not a mark and all the marks that follow it, if any. We can match this with the regex ‹(?>\P{M}\p{M}*)› in all regex flavors that support Unicode but not the ‹\X› token for graphemes. ‹\P{M}› matches any character that is not in the Mark category. ‹\p{M}*› matches all the marks, if any, that follow it.

We put these two regex tokens in an atomic group to make sure the ‹\p{M}*› won’t backtrack if any following regex tokens fail to match. ‹\X{2}.› does not match àà, because there is nothing left for the dot to match after ‹\X{2}› has matched the two accented letters. ‹(?>\P{M}\p{M}*){2}.› does not match àà for the same reason. But ‹(?:\P{M}\p{M}*){2}.› with an non-capturing group does match àà if it is encoded as \u00E0\u0061\u0300. Upon the second iteration of the group, ‹\p{M}*› will match \u0300. The dot will then fail to match. This causes the regex to backtrack, forcing ‹\p{M}*› to give up its match, allowing the dot to match \u0300.

JavaScript’s regex engine does not support atomic grouping. This is not a feature that could be added by XRegExp, because XRegExp still relies on JavaScript’s regex engine for the actual pattern matching. So when using XRegExp, ‹(?:\P{M}\p{M}*)› is the closest we can get to emulating ‹\X›. Without the atomic group, you’ll have to keep in mind that ‹\p{M}*› may backtrack if whatever follows ‹(?:\P{M}\p{M}*)› in your regex can match characters in the Mark category.

The uppercase ‹\P› is the negated variant of the lowercase ‹\p›. For instance, ‹\P{Sc}› matches any character that does not have the “Currency Symbol” Unicode property. ‹\P› is supported by all flavors that support ‹\p›, and for all the properties, block, and scripts that they support.

All flavors allow all the ‹\u›, ‹\x›, ‹\p›, and ‹\P› tokens they support to be used inside character classes. The character represented by the code point, or the characters in the category, block, or script, are then added to the character class. For instance, you could match a character that is either an opening quote (initial punctuation property), a closing quote (final punctuation property), or the trademark symbol (U+2122) with:

[\p{Pi}\p{Pf}\u2122]

[\p{Pi}\p{Pf}\x{2122}]

If your regular expression flavor does not support Unicode categories, blocks, or scripts, you can list the characters that are in the category, block, or script in a character class. For blocks this is very easy: each block is simply a range between two code points. The Greek Extended block comprises the characters U+1F00 to U+1FFF:

[\u1F00-\u1FFF]

[\x{1F00}-\x{1FFF}]

[\u0370-\u0373\u0375-\u0377\u037A-\u037D\u0384\u0386\u0388-\u038A↵
\u038C\u038E-\u03A1\u03A3-\u03E1\u03F0-\u03FF\u1D26-\u1D2A\u1D5D-\u1D61↵
\u1D66-\u1D6A\u1DBF\u1F00-\u1F15\u1F18-\u1F1D\u1F20-\u1F45\u1F48-\u1F4D↵
\u1F50-\u1F57\u1F59\u1F5B\u1F5D\u1F5F-\u1F7D\u1F80-\u1FB4\u1FB6-\u1FC4↵
\u1FC6-\u1FD3\u1FD6-\u1FDB\u1FDD-\u1FEF\u1FF2-\u1FF4\u1FF6-\u1FFE\u2126↵
\U00010140-\U0001018A\U0001D200-\U0001D245]

[\u0370-\u0373\u0375-\u0377\u037A-\u037D\u0384\u0386\u0388-\u038A↵
\u038C\u038E-\u03A1\u03A3-\u03E1\u03F0-\u03FF\u1D26-\u1D2A\u1D5D-\u1D61↵
\u1D66-\u1D6A\u1DBF\u1F00-\u1F15\u1F18-\u1F1D\u1F20-\u1F45\u1F48-\u1F4D↵
\u1F50-\u1F57\u1F59\u1F5B\u1F5D\u1F5F-\u1F7D\u1F80-\u1FB4\u1FB6-\u1FC4↵
\u1FC6-\u1FD3\u1FD6-\u1FDB\u1FDD-\u1FEF\u1FF2-\u1FF4\u1FF6-\u1FFE\u2126]

[\x{0370}-\x{0373}\x{0375}-\x{0377}\x{037A}-\x{037D}\x{0384}\x{0386}↵
\x{0388}-\x{038A}\x{038C}\x{038E}-\x{03A1}\x{03A3}-\x{03E1}↵
\x{03F0}-\x{03FF}\x{1D26}-\x{1D2A}\x{1D5D}-\x{1D61}\x{1D66}-\x{1D6A}↵
\x{1DBF}\x{1F00}-\x{1F15}\x{1F18}-\x{1F1D}\x{1F20}-\x{1F45}↵
\x{1F48}-\x{1F4D}\x{1F50}-\x{1F57}\x{1F59}\x{1F5B}\x{1F5D}\x{1F5F}-↵
\x{1F7D}\x{1F80}-\x{1FB4}\x{1FB6}-\x{1FC4}\x{1FC6}-\x{1FD3}\x{1FD6}-↵
\x{1FDB}\x{1FDD}-\x{1FEF}\x{1FF2}-\x{1FF4}\x{1FF6}-\x{1FFE}\x{2126}↵
\x{10140}-\x{10178}\x{10179}-\x{10189}\x{1018A}\x{1D200}-\x{1D245}]