Table of Contents for
Regular Expressions Cookbook, 2nd Edition

Cover image for bash Cookbook, 2nd Edition

Regular Expressions Cookbook, 2nd Edition by Steven Levithan Published by O'Reilly Media, Inc., 2012

Hexadecimal character

[a-fA-F0-9]
Regex options: None
Regex flavors: .NET, Java, JavaScript, PCRE, Perl, Python, Ruby

Nonhexadecimal character

[^a-fA-F0-9]
Regex options: None
Regex flavors: .NET, Java, JavaScript, PCRE, Perl, Python, Ruby

Discussion

The notation using square brackets is called a character class. A character class matches a single character out of a list of possible characters. The three classes in the first regex match either an a or an e. They do so independently. When you test calendar against this regex, the first character class matches a, the second e, and the third a.

Inside a character class, only four characters have a special function: \, ^, -, and ]. If you’re using Java or .NET, the opening bracket [ is also a metacharacter inside character classes.

A backslash always escapes the character that follows it, just as it does outside character classes. The escaped character can be a single character, or the start or end of a range. The other four metacharacters get their special meanings only when they’re placed in a certain position. It is possible to include them as literal characters in a character class without escaping them, by positioning them in a way that they don’t get their special meaning. ‹[][^-]› pulls off this trick. This works with all flavors in this book, except JavaScript. JavaScript treats ‹[]› as an empty character class that always fails to match. But we recommend that you always escape these metacharacters, so the previous regex should be ‹[\]\[\^\-]›. Escaping the metacharacters makes your regular expression easier to understand.

All other characters are literals and simply add themselves to the character class. The regular expression ‹[$()*+.?{|]› matches any one of the nine characters between the square brackets. These nine characters only have special meanings outside character classes. Inside character classes they are just literal text. Escaping them would only make your regular expression harder to read.

Alphanumeric characters cannot be escaped with a backslash. Doing so may be an error or may create a regular expression token (something with a special meaning in a regular expression). In our discussions of certain other regex tokens, such as in Recipe 2.2, we mention that they can be used inside character classes. All these tokens consist of a backslash and a letter, sometimes followed by a bunch of other characters. Thus, ‹[\r\n]› matches a carriage return (\r) or line feed (\n).

A caret (^) negates the character class if you place it immediately after the opening bracket. It makes the character class match any character that is not in the list.

Caution

In all the regex flavors discussed in this book, a negated character class matches line break characters, unless you add them to the negated character class. Make sure that you don’t accidentally allow your regex to span across lines.

A hyphen (-) creates a range when it is placed between two characters. The range includes the character before the hyphen, the character after the hyphen, and all characters that lie between them in numerical order. To know which characters those are, you have to look at the ASCII or Unicode character table. ‹[A-z]› includes all characters in the ASCII table between the uppercase A and the lowercase z. The range includes some punctuation, so ‹[A-Z\[\\\]\^_`a-z]› matches the same characters more explicitly. We recommend that you create ranges only between two digits or between two letters that are both upper- or lowercase.

Tip

Reversed ranges, such as ‹[z-a]›, are not permitted.

Variations

Shorthands

Six regex tokens that consist of a backslash and a letter form shorthand character classes: ‹\d›, ‹\D›, ‹\w›, ‹\W›, ‹\s› and ‹\S›. You can use these both inside and outside character classes. Each lowercase shorthand character has an associated uppercase shorthand character with the opposite meaning.

‹\d› and ‹[\d]› both match a single digit. ‹\D› matches any character that is not a digit, and is equivalent to ‹[^\d]›.

Here is how we can use the ‹\d› shorthand to rewrite the “hexadecimal character” regex from earlier in this recipe:

[a-fA-F\d]
Regex options: None
Regex flavors: .NET, Java, PCRE, Perl, Python, Ruby
‹\w› matches a single word character. A word character is a character that can occur as part of a word. That includes letters, digits, and the underscore. The particular choice of characters here may seem odd, but it was chosen because these are the characters that are typically allowed in identifiers in programming languages. ‹\W› matches any character that is not part of such a propellerhead word.
In Java 4 to 6, JavaScript, PCRE, and Ruby, ‹\w› is always identical to ‹[a-zA-Z0-9_]›. In .NET, it includes letters and digits from all other scripts (Cyrillic, Thai, etc.). In Java 7, the other scripts are included only if you set the UNICODE_CHARACTER_CLASS flag. In Python 2.x, the other scripts are included only if you pass the UNICODE or U flag when creating the regex. In Python 3.x the other scripts are included by default, but you can make ‹\w› ASCII-only with the ASCII or A flag. In Perl 5.14, the /a (ASCII) flag makes ‹\w› identical to ‹[a-zA-Z0-9_]›, while /u (Unicode) adds all Unicode scripts, and /l (locale) makes ‹\w› depend on the locale. Prior to Perl 5.14, or when using /d (default) or none of the /adlu flags in Perl 5.14, ‹\w› automatically includes Unicode scripts if the subject string or the regex are encoded as UTF-8, or the regex includes a code point above 255 such as ‹\x{100}› or a Unicode property such as ‹\p{L}›. If not, the default for ‹\w› is pure ASCII.
‹\d› follows the same rules as ‹\w› in all these flavors. In .NET, digits from other scripts are always included. In Python it depends on the UNICODE and ASCII flags, and whether you’re using Python 2.x or 3.x. In Perl 5.14, it depends on the /adlu flags. In earlier versions of Perl, it depends on the encoding of the subject and regex, and whether the regex has any Uncicode tokens.
‹\s› matches any whitespace character. This includes spaces, tabs, and line breaks. ‹\S› matches any character not matched by ‹\s› In .NET and JavaScript, ‹\s› also matches any character defined as whitespace by the Unicode standard. In Java, Perl, and Python, ‹\s› follows the same rules as ‹\w› and ‹\d›.
Notice that JavaScript uses Unicode for ‹\s› but ASCII for ‹\d› and ‹\w›. Further inconsistency arises when we add ‹\b› to the mix. ‹\b› is not a shorthand character class, but a word boundary. Though you’d expect ‹\b› to support Unicode when ‹\w› does and to be ASCII-only when ‹\w› is ASCII-only, this isn’t always the case. The subsection Word Characters in Recipe 2.6 has the details.

Case insensitivity

(?i)[A-F0-9]
Regex options: None
Regex flavors: .NET, Java, XRegExp, PCRE, Perl, Python, Ruby
(?i)[^A-F0-9]
Regex options: None
Regex flavors: .NET, Java, XRegExp, PCRE, Perl, Python, Ruby
Case insensitivity, whether set with an external flag (see Recipe 3.4) or a mode modifier inside the regex (see Case-insensitive matching in Recipe 2.1), also affects character classes. The two regexes just shown are equivalent to the ones in the original solution.
JavaScript follows the same rule, but it doesn’t support ‹(?i)›. To make a regular expression case-insensitive in JavaScript, set the /i flag when creating it. Or use the XRegExp library for JavaScript, which adds support for mode modifiers at the start of the regex.

Flavor-Specific Features

.NET character class subtraction

[a-zA-Z0-9-[g-zG-Z]]
Regex options: None
Regex flavors: .NET 2.0 or later
This regular expression matches a single hexadecimal character, but in a roundabout way. The base character class matches any alphanumeric character, and a nested class then subtracts the letters g through z. This nested class must appear at the end of the base class, preceded by a hyphen, as shown here: ‹[class-[subtract]]›.
Character class subtraction is particularly useful when working with Unicode categories, blocks, and scripts. As an example, ‹\p{IsThai}› matches any character in the Thai block. ‹\P{N}› matches any character that is not in the Number category. Combining them with subtraction, ‹[\p{IsThai}-[\P{N}]]› matches any of the 10 Thai digits using character class subtraction. Recipe 2.7 has all the details on working with Unicode properties.

Java character class union, intersection, and subtraction

Java allows one character class to be nested inside another. If the nested class is included directly, the resulting class is the union of the two. You can nest as many classes as you like. The regexes ‹[a-f[A-F][0-9]]› and ‹[a-f[A-F[0-9]]]› use character class union. They match a hexadecimal digit just like the original regex without the extra square brackets.

The regex ‹[\w&&[a-fA-F0-9\s]]› uses character class intersection to match a hexadecimal digit. It could win a prize in a regex obfuscation contest. The base character class ‹[\w]› matches any word character. The nested class ‹[a-fA-F0-9\s]› matches any hexadecimal digit and any whitespace character. The resulting class is the intersection of the two, matching hexadecimal digits and nothing else. Because the base class does not match whitespace and the nested class does not match ‹[g-zG-Z_]›, those are dropped from the final character class, leaving only the hexadecimal digits.

‹[a-zA-Z0-9&&[^g-zG-Z]]› uses character class subtraction to match a single hexadecimal character in a roundabout way. The base character class ‹[a-zA-Z0-9]› matches any alphanumeric character. The nested class ‹[^g-zG-Z]› then subtracts the letters g through z. This nested class must be a negated character class, preceded by two ampersands, as shown here: ‹[class&&[^subtract]]›.

Character class intersection and subtraction are particularly useful when working with Unicode properties, blocks, and scripts. Thus, ‹\p{InThai}› matches any character in the Thai block, whereas ‹\p{N}› matches any character that is in the Number category. In consequence, ‹[\p{InThai}&&[\p{N}]]› matches any of the 10 Thai digits using character class intersection.

If you’re wondering about the subtle differences in the ‹\p› regex tokens, you’ll find those all explained in Recipe 2.7. Recipe 2.7 has all the details on working with Unicode properties.

Table of Contents for
Regular Expressions Cookbook, 2nd Edition

2.3. Match One of Many Characters

Problem

Solution

Calendar with misspellings

Hexadecimal character

Nonhexadecimal character

Discussion

Caution

Tip

Variations

Shorthands

Case insensitivity

Flavor-Specific Features

.NET character class subtraction

Java character class union, intersection, and subtraction

See Also

Table of Contents for Regular Expressions Cookbook, 2nd Edition

2.3. Match One of Many Characters

Problem

Solution

Calendar with misspellings

Hexadecimal character

Nonhexadecimal character

Discussion

Caution

Tip

Variations

Shorthands

Case insensitivity

Flavor-Specific Features

.NET character class subtraction

Java character class union, intersection, and subtraction

See Also

Table of Contents for
Regular Expressions Cookbook, 2nd Edition