RegexPlanet is a website developed by Andrew Marcuse. Its claim to fame is that it allows you to test your regexes against a larger variety of regular expression libraries than any other regex tester we are aware of. On the home page you’ll find links to testers for Java, JavaScript, .NET, Perl, PHP, Python, and Ruby. They all use the same basic interface. Only the list of options is adapted to those of each programming language. Figure 1-4 shows the .NET version.

Figure 1-4. RegexPlanet

Type or paste your regular expression into the “regular expression” box. If you want to test a search-and-replace, paste the replacement text into the “replacement” box. You can test your regex against as many different subject strings as you like. Paste your subject strings into the “input” boxes. Click “more inputs” if you need more than five. The “regex” and “input” boxes allow you to type or paste in multiple lines of text, even though they only show one line at a time. The arrows at the right are the scrollbar.

When you’re done, click the “test” button to send all your strings to the regexplanet.com server. The resulting page, as shown in Figure 1-4, lists the test results at the top. The first two columns repeat your input. The remaining columns show the results of various function calls. These columns are different for the various programming languages that the site supports.

Lars Olav Torvik has put a great little regular expression tester online at http://regex.larsolavtorvik.com (see Figure 1-5).

Figure 1-5. regex.larsolavtorvik.com

To start, select the regular expression flavor you’re working with by clicking on the flavor’s name at the top of the page. Lars offers PHP PCRE, PHP POSIX, and JavaScript. PHP PCRE, the PCRE regex flavor discussed in this book, is used by PHP’s preg functions. POSIX is an old and limited regex flavor used by PHP’s ereg functions, which are not discussed in this book. If you select JavaScript, you’ll be working with your browser’s JavaScript implementation.

Type your regular expression into the Pattern field and your subject text into the Subject field. A moment later, the Matches field displays your subject text with highlighted regex matches. The Code field displays a single line of source code that applies your regex to your subject text. Copying and pasting this into your code editor saves you the tedious job of manually converting your regex into a string literal. Any string or array returned by the code is displayed in the Result field. Because Lars used Ajax technology to build his site, results are updated in just a few moments for all flavors. To use the tool, you have to be online, as PHP is processed on the server rather than in your browser.

The second column displays a list of regex commands and regex options. These depend on the regex flavor. The regex commands typically include match, replace, and split operations. The regex options consist of common options such as case insensitivity, as well as implementation-specific options. These commands and options are described in Chapter 3.

http://www.nregex.com (Figure 1-6) is a straightforward online regex tester built on .NET technology by David Seruyange. It supports the .NET 2.0 regex flavor, which is also used by .NET 3.0, 3.5, and 4.0.

Figure 1-6. Nregex

The layout of the page is somewhat confusing. Enter your regular expression into the field under the Regular Expression label, and set the regex options using the checkboxes below that. Enter your subject text in the large box at the bottom, replacing the default If I just had $5.00 then "she" wouldn't be so @#$! mad.. If your subject is a web page, type the URL in the Load Target From URL field, and click the Load button under that input field. If your subject is a file on your hard disk, click the Browse button, find the file you want, and then click the Load button under that input field.

Your subject text will appear duplicated in the “Matches & Replacements” field at the center of the web page, with the regex matches highlighted. If you type something into the Replacement String field, the result of the search-and-replace is shown instead. If your regular expression is invalid, ... appears.

The regex matching is done in .NET code running on the server, so you need to be online for the site to work. If the automatic updates are slow, perhaps because your subject text is very long, tick the Manually Evaluate Regex checkbox above the field for your regular expression to show the Evaluate button. Click that button to update the “Matches & Replacements” display.

Michael Lovitt put a minimalistic regex tester online at http://www.rubular.com (Figure 1-7). At the time of writing, it lets you choose between Ruby 1.8.7 and Ruby 1.9.2. This allows you to test both the Ruby 1.8 and Ruby 1.9 regex flavors used in this book.

Figure 1-7. Rubular

Enter your regular expression in the box between the two forward slashes under “Your regular expression.” You can turn on case insensitivity by typing an i in the small box after the second slash. Similarly, if you like, turn on the option “the dot matches line breaks” by typing an m in the same box. im turns on both options. Though these conventions may seem a bit user-unfriendly if you’re new to Ruby, they conform to the /regex/im syntax used to specify a regex in Ruby source code.

Type or paste your subject text into the “Your test string” box, and wait a moment. A new “Match result” box appears to the right, showing your subject text with all regex matches highlighted.

Sergey Evdokimov created several regular expression testers for Java developers. The home page at http://www.myregexp.com (Figure 1-8) offers an online regex tester. It’s a Java applet that runs in your browser. The Java 4 (or later) runtime needs to be installed on your computer. The applet uses the java.util.regex package to evaluate your regular expressions, which is new in Java 4. In this book, the “Java” regex flavor refers to this package.

Figure 1-8. myregexp.com

Type your regular expression into the Regular Expression box. Use the Flags menu to set the regex options you want. Three of the options also have direct checkboxes.

If you want to test a regex that already exists as a string in Java code, copy the whole string to the clipboard. In the myregexp.com tester, click on the Edit menu, and then “Paste Regex from Java String.” In the same menu, pick “Copy Regex for Java Source” when you’re done editing the regular expression. The Edit menu has similar commands for JavaScript and XML as well.

Below the regular expression, there are four tabs that run four different tests:

At the top of the page at http://www.myregexp.com, you can click the link to get Sergey’s regex tester as a plug-in for Eclipse.

Expresso (not to be confused with caffeine-laden espresso) is a .NET application for creating and testing regular expressions. You can download it at http://www.ultrapico.com/Expresso.htm. The .NET Framework 2.0 or later must be installed on your computer.

The download is a free 60-day trial. After the trial, you have to register or Expresso will (mostly) stop working. Registration is free, but requires you to give the Ultrapico folks your email address. The registration key is sent by email.

Expresso displays a screen like the one shown in Figure 1-9. The Regular Expression box where you type in your regular expression is permanently visible. No syntax highlighting is available. The Regex Analyzer box automatically builds a brief English-language analysis of your regular expression. It too is permanently visible.

Figure 1-9. Expresso

In Design Mode, you can set matching options such as “Ignore Case” at the bottom of the screen. Most of the screen space is taken up by a row of tabs where you can select the regular expression token you want to insert. If you have two monitors or one large monitor, click the Undock button to float the row of tabs. Then you can build up your regular expression in the other mode (Test Mode) as well.

In Test Mode, type or paste your sample text in the lower-left corner. Then, click the Run Match button to get a list of all matches in the Search Results box. No highlighting is applied to the sample text. Click on a match in the results to select that match in the sample text.

The Expression Library shows a list of sample regular expressions and a list of recent regular expressions. Your regex is added to that list each time you press Run Match. You can edit the library through the Library menu in the main menu bar.

The Regulator, which you can download from http://sourceforge.net/projects/regulator/, is not safe for SCUBA diving or cooking-gas canisters; it is another .NET application for creating and testing regular expressions. The latest version requires .NET 2.0 or later. Older versions for .NET 1.x can still be downloaded. The Regulator is open source, and no payment or registration is required.

The Regulator does everything in one screen (Figure 1-10). The New Document tab is where you enter your regular expression. Syntax highlighting is automatically applied, but syntax errors in your regex are not made obvious. Right-click to select the regex token you want to insert from a menu. You can set regular expression options via the buttons on the main toolbar. The icons are a bit cryptic. Wait for the tool tip to see which option you’re setting with each button.

Figure 1-10. The Regulator

Below the area for your regex and to the right, click on the Input button to display the area for pasting in your sample text. Click the “Replace with” button to type in the replacement text, if you want to do a search-and-replace. Below the regex and to the left, you can see the results of your regex operation. Results are not updated automatically; you must click the Match, Replace, or Split button in the toolbar to update the results. No highlighting is applied to the input. Click on a match in the results to select it in the subject text.

The Regex Analyzer panel shows a simple English-language analysis of your regular expression, but it is not automatic or interactive. To update the analysis, select Regex Analyzer in the View menu, even if it is already visible. Clicking on the analysis only moves the text cursor.

SDL Regex Fuzzer’s fuzzy name does not make its purpose obvious. Microsoft bills it as “a tool to help test regular expressions for potential denial of service vulnerabilities.” You can download it for free at http://www.microsoft.com/en-us/download/details.aspx?id=20095. It requires .NET 3.5 to run.

What SDL Regex Fuzzer really does is to check whether there exists a subject string that causes your regular expression to execute in exponential time. In our book we call this “catastrophic backtracking.” We explain this in detail along with potential solutions in Recipe 2.15. Basically, a regex that exhibits catastrophic backtracking will cause your application to run forever or to crash. If your application is a server, that could be exploited in a denial-of-service attack.

Figure 1-11. SDL Regex Fuzzer

Figure 1-11 shows the results of a test in SDL Regex Fuzzer. In Step 1 we pasted in a regular expression from Recipe 2.15. Since this regex can never match non-ASCII characters, there’s no need to select that option in Step 2. Otherwise, we should have. We left Step 3 set to the default of 100 iterations. About five seconds after clicking the Start button in Step 4, SDL Regex Fuzzer showed a sample string that will cause our regex to fail in .NET 3.5.

Unfortunately, the usefulness of this tool is greatly limited because it only supports a small subset of the .NET regex syntax. When we tried to test the naïve solution from Recipe 2.15, which would definitely fail this test, we received the error message shown in Figure 1-12. Proper understanding of the concepts discussed in Recipe 2.15 is still the only way to make sure you don’t bring down your applications with overly complex regular expressions.

Figure 1-12. SDL Regex Fuzzer Limitations

PowerGREP, developed by Jan Goyvaerts, one of this book’s authors, is probably the most feature-rich grep tool available for the Microsoft Windows platform (Figure 1-13). PowerGREP uses a custom regex flavor that combines the best of the flavors discussed in this book. This flavor is labeled “JGsoft” in RegexBuddy.

Figure 1-13. PowerGREP

To run a quick regular expression search, simply select Clear in the Action menu and type your regular expression into the Search box on the Action panel. Click on a folder in the File Selector panel, and select “Include File or Folder” or “Include Folder and Subfolders” in the File Selector menu. Then, select Execute in the Action menu to run your search.

To run a search-and-replace, select “search-and-replace” in the “action type” drop-down list at the top-left corner of the Action panel after clearing the action. A Replace box will appear below the Search box. Enter your replacement text there. All the other steps are the same as for searching.

PowerGREP has the unique ability to use up to five lists of regular expressions at the same time, with any number of regular expressions in each list. While the previous two paragraphs provide all you need to run simple searches like you can in any grep tool, unleashing PowerGREP’s full potential will take a bit of reading through the tool’s comprehensive documentation.

PowerGREP runs on Windows 2000, XP, Vista, 7, and 8. You can download a free evaluation copy at http://www.powergrep.com/PowerGREPCookbook.exe. Except for saving results and libraries, the trial is fully functional for 15 days of actual use. Though the trial won’t save the results shown on the Results panel, it will modify all your files for search-and-replace actions, just like the full version does.

Figure 1-14. Windows Grep

Windows Grep (http://www.wingrep.com) is one of the oldest grep tools for Windows. Its age shows a bit in its user interface (Figure 1-14), but it does what it says on the tin just fine. It supports a limited regular expression flavor called POSIX ERE. For the features that it supports, it uses the same syntax as the flavors in this book. Windows Grep is shareware, which means you can download it for free, but payment is expected if you want to keep it.

To prepare a search, select Search in the Search menu. The screen that appears differs depending on whether you’ve selected Beginner Mode or Expert Mode in the Options menu. Beginners get a step-by-step wizard, whereas experts get a tabbed dialog.

When you’ve set up the search, Windows Grep immediately executes it, presenting you with a list of files in which matches were found. Click once on a file to see its matches in the bottom panel, and double-click to open the file. Select “All Matches” in the View menu to make the bottom panel show everything.

To run a search-and-replace, select Replace in the Search menu.

RegexRenamer (Figure 1-15) is not really a grep tool. Instead of searching through the contents of files, it searches and replaces through the names of files. You can download it at http://regexrenamer.sourceforge.net. RegexRenamer requires version 2.0 or later of the Microsoft .NET Framework.

Figure 1-15. RegexRenamer

Type your regular expression into the Match box and the replacement text into the Replace box. Click /i to turn on case insensitivity, and /g to replace all matches in each filename rather than just the first. /x turns on free-spacing syntax, which isn’t very useful, since you have only one line to type in your regular expression.

Use the tree at the left to select the folder that holds the files you want to rename. You can set a file mask or a regex filter in the top-right corner. This restricts the list of files to which your search-and-replace regex will be applied. Using one regex to filter and another to replace is much handier than trying to do both tasks with just one regex.

We can make our solution easier to read when using a regex flavor that supports a feature called block escape:

The●punctuation●characters●in●the●ASCII●table●are:●↵
\Q!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~\E

Perl, PCRE and Java support the regex tokens ‹\Q› and ‹\E›. ‹\Q› suppresses the meaning of all metacharacters, including the backslash, until ‹\E›. If you omit ‹\E›, all characters after the ‹\Q› until the end of the regex are treated as literals.

The only benefit of ‹\Q...\E› is that it is easier to read than ‹\.\.\.›.

By default, regular expressions are case sensitive. ‹regex› matches regex but not Regex, REGEX, or ReGeX. To make ‹regex› match all of those, you need to turn on case insensitivity.

In most applications, that’s a simple matter of marking or clearing a checkbox. All programming languages discussed in the next chapter have a flag or property that you can set to make your regex case insensitive. Recipe 3.4 in the next chapter explains how to apply the regex options listed with each regular expression solution in this book in your source code.

ascii

If you cannot turn on case insensitivity outside the regex, you can do so within by using the ‹(?i)› mode modifier, such as ‹(?i)regex›. This works with the .NET, Java, PCRE, Perl, Python, and Ruby flavors. It works with JavaScript when using the XRegExp library.

(?i)ascii

.NET, Java, PCRE, Perl, and Ruby support local mode modifiers, which affect only part of the regular expression. ‹sensitive(?i)caseless(?-i)sensitive› matches sensitiveCASELESSsensitive but not SENSITIVEcaselessSENSITIVE. ‹(?i)› turns on case insensitivity for the remainder of the regex, and ‹(?-i)› turns it off for the remainder of the regex. They act as toggle switches.

Recipe 2.9 shows how to use local mode modifiers with groups instead of toggles.

Here’s another way to match the same seven ASCII control characters matched by the regexes earlier in this recipe:

\cG\x1B\cL\cJ\cM\cI\cK

Using ‹\cA› through ‹\cZ›, you can match one of the 26 control characters that occupy positions 1 through 26 in the ASCII table. The c must be lowercase. The letter that follows the c is case insensitive in most flavors. We recommend that you always use an uppercase letter. Java requires this.

This syntax can be handy if you’re used to entering control characters on console systems by pressing the Control key along with a letter. On a terminal, Ctrl-H sends a backspace. In a regex, ‹\cH› matches a backspace.

Python and the classic Ruby engine in Ruby 1.8 do not support this syntax. The Oniguruma engine in Ruby 1.9 does.

The escape control character, at position 27 in the ASCII table, is beyond the reach of the English alphabet, so we leave it as ‹\x1B› in our regular expression.

Following is yet another way to match our list of seven commonly used control characters:

\x07\x1B\x0C\x0A\x0D\x09\x0B

A lowercase \x followed by two uppercase hexadecimal digits matches a single character in the ASCII set. Figure 2-1 shows which hexadecimal combinations from ‹\x00› through ‹\x7F› match each character in the entire ASCII character set. The table is arranged with the first hexadecimal digit going down the left side and the second digit going across the top.

Figure 2-1. ASCII table

The characters that ‹\x80› through ‹\xFF› match depends on how your regex engine interprets them, and which code page your subject text is encoded in. We recommend that you not use ‹\x80› through ‹\xFF›. Instead, use the Unicode code point token described in Recipe 2.7.

c[ae]l[ae]nd[ae]r

[a-fA-F0-9]

[^a-fA-F0-9]

Six regex tokens that consist of a backslash and a letter form shorthand character classes: ‹\d›, ‹\D›, ‹\w›, ‹\W›, ‹\s› and ‹\S›. You can use these both inside and outside character classes. Each lowercase shorthand character has an associated uppercase shorthand character with the opposite meaning.

‹\d› and ‹[\d]› both match a single digit. ‹\D› matches any character that is not a digit, and is equivalent to ‹[^\d]›.

Here is how we can use the ‹\d› shorthand to rewrite the “hexadecimal character” regex from earlier in this recipe:

[a-fA-F\d]

‹\w› matches a single word character. A word character is a character that can occur as part of a word. That includes letters, digits, and the underscore. The particular choice of characters here may seem odd, but it was chosen because these are the characters that are typically allowed in identifiers in programming languages. ‹\W› matches any character that is not part of such a propellerhead word.

In Java 4 to 6, JavaScript, PCRE, and Ruby, ‹\w› is always identical to ‹[a-zA-Z0-9_]›. In .NET, it includes letters and digits from all other scripts (Cyrillic, Thai, etc.). In Java 7, the other scripts are included only if you set the UNICODE_CHARACTER_CLASS flag. In Python 2.x, the other scripts are included only if you pass the UNICODE or U flag when creating the regex. In Python 3.x the other scripts are included by default, but you can make ‹\w› ASCII-only with the ASCII or A flag. In Perl 5.14, the /a (ASCII) flag makes ‹\w› identical to ‹[a-zA-Z0-9_]›, while /u (Unicode) adds all Unicode scripts, and /l (locale) makes ‹\w› depend on the locale. Prior to Perl 5.14, or when using /d (default) or none of the /adlu flags in Perl 5.14, ‹\w› automatically includes Unicode scripts if the subject string or the regex are encoded as UTF-8, or the regex includes a code point above 255 such as ‹\x{100}› or a Unicode property such as ‹\p{L}›. If not, the default for ‹\w› is pure ASCII.

‹\d› follows the same rules as ‹\w› in all these flavors. In .NET, digits from other scripts are always included. In Python it depends on the UNICODE and ASCII flags, and whether you’re using Python 2.x or 3.x. In Perl 5.14, it depends on the /adlu flags. In earlier versions of Perl, it depends on the encoding of the subject and regex, and whether the regex has any Uncicode tokens.

‹\s› matches any whitespace character. This includes spaces, tabs, and line breaks. ‹\S› matches any character not matched by ‹\s› In .NET and JavaScript, ‹\s› also matches any character defined as whitespace by the Unicode standard. In Java, Perl, and Python, ‹\s› follows the same rules as ‹\w› and ‹\d›.

Notice that JavaScript uses Unicode for ‹\s› but ASCII for ‹\d› and ‹\w›. Further inconsistency arises when we add ‹\b› to the mix. ‹\b› is not a shorthand character class, but a word boundary. Though you’d expect ‹\b› to support Unicode when ‹\w› does and to be ASCII-only when ‹\w› is ASCII-only, this isn’t always the case. The subsection Word Characters in Recipe 2.6 has the details.

(?i)[A-F0-9]

(?i)[^A-F0-9]

Case insensitivity, whether set with an external flag (see Recipe 3.4) or a mode modifier inside the regex (see Case-insensitive matching in Recipe 2.1), also affects character classes. The two regexes just shown are equivalent to the ones in the original solution.

JavaScript follows the same rule, but it doesn’t support ‹(?i)›. To make a regular expression case-insensitive in JavaScript, set the /i flag when creating it. Or use the XRegExp library for JavaScript, which adds support for mode modifiers at the start of the regex.

[a-zA-Z0-9-[g-zG-Z]]

This regular expression matches a single hexadecimal character, but in a roundabout way. The base character class matches any alphanumeric character, and a nested class then subtracts the letters g through z. This nested class must appear at the end of the base class, preceded by a hyphen, as shown here: ‹[class-[subtract]]›.

Character class subtraction is particularly useful when working with Unicode categories, blocks, and scripts. As an example, ‹\p{IsThai}› matches any character in the Thai block. ‹\P{N}› matches any character that is not in the Number category. Combining them with subtraction, ‹[\p{IsThai}-[\P{N}]]› matches any of the 10 Thai digits using character class subtraction. Recipe 2.7 has all the details on working with Unicode properties.

Java allows one character class to be nested inside another. If the nested class is included directly, the resulting class is the union of the two. You can nest as many classes as you like. The regexes ‹[a-f[A-F][0-9]]› and ‹[a-f[A-F[0-9]]]› use character class union. They match a hexadecimal digit just like the original regex without the extra square brackets.

The regex ‹[\w&&[a-fA-F0-9\s]]› uses character class intersection to match a hexadecimal digit. It could win a prize in a regex obfuscation contest. The base character class ‹[\w]› matches any word character. The nested class ‹[a-fA-F0-9\s]› matches any hexadecimal digit and any whitespace character. The resulting class is the intersection of the two, matching hexadecimal digits and nothing else. Because the base class does not match whitespace and the nested class does not match ‹[g-zG-Z_]›, those are dropped from the final character class, leaving only the hexadecimal digits.

‹[a-zA-Z0-9&&[^g-zG-Z]]› uses character class subtraction to match a single hexadecimal character in a roundabout way. The base character class ‹[a-zA-Z0-9]› matches any alphanumeric character. The nested class ‹[^g-zG-Z]› then subtracts the letters g through z. This nested class must be a negated character class, preceded by two ampersands, as shown here: ‹[class&&[^subtract]]›.

Character class intersection and subtraction are particularly useful when working with Unicode properties, blocks, and scripts. Thus, ‹\p{InThai}› matches any character in the Thai block, whereas ‹\p{N}› matches any character that is in the Number category. In consequence, ‹[\p{InThai}&&[\p{N}]]› matches any of the 10 Thai digits using character class intersection.

If you’re wondering about the subtle differences in the ‹\p› regex tokens, you’ll find those all explained in Recipe 2.7. Recipe 2.7 has all the details on working with Unicode properties.

'.'

'.'

'[\s\S]'

The dot is one of the oldest and simplest regular expression features. Its meaning has always been to match any single character.

There is, however, some confusion as to what any character truly means. The oldest tools for working with regular expressions processed files line by line, so there was never an opportunity for the subject text to include a line break. The programming languages discussed in this book process the subject text as a whole, no matter how many line breaks you put into it. If you want true line-by-line processing, you have to write a bit of code that splits the subject into an array of lines and applies the regex to each line in the array. Recipe 3.21 in the next chapter shows how to do this.

Larry Wall, the developer of Perl, wanted Perl to retain the traditional behavior of line-based tools, in which the dot never matched a line break. All the other flavors discussed in this book followed suit. ‹.› thus matches any single character except line break characters.

If you do want to allow your regular expression to span multiple lines, turn on the “dot matches line breaks” option. This option masquerades under different names. Perl and many others confusingly call it “single line” mode, whereas Java calls it “dot all” mode. Recipe 3.4 in the next chapter has all the details. Whatever the name of this option in your favorite programming language is, think of it as “dot matches line breaks” mode. That’s all the option does.

An alternative solution is needed for JavaScript, which doesn’t have a “dot matches line breaks” option. As Recipe 2.3 explains, ‹\s› matches any whitespace character, whereas ‹\S› matches any character that is not matched by ‹\s›. Combining these into ‹[\s\S]› results in a character class that includes all characters, including line breaks. ‹[\d\D]› and ‹[\w\W]› have the same effect.

The dot is the most abused regular expression feature. ‹\d\d.\d\d.\d\d› is not a good way to match a date. It does match 05/16/08 just fine, but it also matches 99/99/99. Worse, it matches 12345678.

A proper regex for matching only valid dates is a subject for a later chapter (see Recipe 4.5). But replacing the dot with a more appropriate character class is very easy. ‹\d\d[/.\-]\d\d[/.\-]\d\d› allows a forward slash, dot, or hyphen to be used as the date separator. This regex still matches 99/99/99, but not 12345678.

Use the dot only when you really want to allow any character. Use a character class or negated character class in any other situation.

^alpha

\Aalpha

omega$

omega\Z

^begin

end$

The regular expression tokens ‹^›, ‹$›, ‹\A›, ‹\Z›, and ‹\z› are called anchors. They do not match any characters. Instead, they match at certain positions, effectively anchoring the regular expression match at those positions.

A line is the part of the subject text that lies between the start of the subject and a line break, between two line breaks, or between a line break and the end of the subject. If there are no line breaks in the subject, then the whole subject is considered to be one line. Thus, the following text consists of four lines, one each for one, two, an empty string, and four:

one
two

four

The text could be represented in a program as oneLFtwoLFLFfour.

The anchor ‹\A› always matches at the very start of the subject text, before the first character. That is the only place where it matches. Place ‹\A› at the start of your regular expression to test whether the subject text begins with the text you want to match. The “A” must be uppercase.

JavaScript does not support ‹\A›.

The anchor ‹^› is equivalent to ‹\A›, as long as you do not turn on the “^ and $ match at line breaks” option. This option is off by default for all regex flavors except Ruby. Ruby does not offer a way to turn this option off.

Unless you’re using JavaScript, we recommend that you always use ‹\A› instead of ‹^›. The meaning of ‹\A› never changes, avoiding any confusion or mistakes in setting regex options.

The anchors ‹\Z› and ‹\z› always match at the very end of the subject text, after the last character. Place ‹\Z› or ‹\z› at the end of your regular expression to test whether the subject text ends with the text you want to match.

.NET, Java, PCRE, Perl, and Ruby support both ‹\Z› and ‹\z›. Python supports only ‹\Z›. JavaScript does not support ‹\Z› or ‹\z› at all.

The difference between ‹\Z› and ‹\z› comes into play when the last character in your subject text is a line break. In that case, ‹\Z› can match at the very end of the subject text, after the final line break, as well as immediately before that line break. The benefit is that you can search for ‹omega\Z› without having to worry about stripping off a trailing line break at the end of your subject text. When reading a file line by line, some tools include the line break at the end of the line, whereas others don’t; ‹\Z› masks this difference. ‹\z› matches only at the very end of the subject text, so it will not match text if a trailing line break follows.

The anchor ‹$› is equivalent to ‹\Z›, as long as you do not turn on the “^ and $ match at line breaks” option. This option is off by default for all regex flavors except Ruby. Ruby does not offer a way to turn this option off. Just like ‹\Z›, ‹$› matches at the very end of the subject text, as well as before the final line break, if any.

To help clarify this subtle and somewhat confusing situation, let’s look at an example in Perl. Assuming that $/ (the current record separator) is set to its default \n, the following Perl statement reads a single line from the terminal (standard input):

$line = <>;

Perl leaves the newline on the content of the variable $line. Therefore, an expression such as ‹end●of●input.\z› will not match the variable. But ‹end●of●input.\Z› and ‹end●of●input.$› will both match, because they ignore the trailing newline.

To make processing easier, Perl programmers often strip newlines with:

chomp $line;

After that operation is performed, all three anchors will match. (Technically, chomp strips a string of the current record separator.)

Unless you’re using JavaScript, we recommend that you always use ‹\Z› instead of ‹$›. The meaning of ‹\Z› never changes, avoiding any confusion or mistakes in setting regex options.

By default, ‹^› matches only at the start of the subject text, just like ‹\A›. Only in Ruby does ‹^› always match at the start of a line. All the other flavors require you to turn on the option to make the caret and dollar sign match at line breaks. This option is typically referred to as “multiline” mode.

Do not confuse this mode with “single line” mode, which would be better known as “dot matches line breaks” mode. “Multiline” mode affects only the caret and dollar sign; “single line” mode affects only the dot, as Recipe 2.4 explains. It is perfectly possible to turn on both “single line” and “multiline” mode at the same time. By default, both options are off.

With the correct option set, ‹^› will match at the start of each line in the subject text. Strictly speaking, it matches before the very first character in the file, as it always does, and also after each line break character in the subject text. The caret in ‹\n^› is redundant because ‹^› always matches after ‹\n›.

By default, ‹$› matches only at the end of the subject text or before the final line break, just like ‹\Z›. Only in Ruby does ‹$› always match at the end of each line. All the other flavors require you to turn on the “multiline” option to make the caret and dollar match at line breaks.

With the correct option set, ‹$› will match at the end of each line in the subject text. (Of course, it also matches after the very last character in the text because that is always the end of a line as well.) The dollar in ‹$\n› is redundant because ‹$› always matches before ‹\n›.

It is perfectly valid for a regular expression to consist of nothing but one or more anchors. Such a regular expression will find a zero-length match at each position where the anchor can match. If you place several anchors together, all of them need to match at the same position for the regex to match.

You could use such a regular expression in a search-and-replace. Replace ‹\A› or ‹\Z› to prepend or append something to the whole subject. Replace ‹^› or ‹$›, in “^ and $ match at line breaks” mode, to prepend or append something in each line in the subject text.

Combine two anchors to test for blank lines or missing input. ‹\A\Z› matches the empty string, as well as the string that consists of a single newline. ‹\A\z› matches only the empty string. ‹^$›, in “^ and $ match at line breaks” mode, matches each empty line in the subject text.

\bcat\b

\Bcat\B

The regular expression token ‹\b› is called a word boundary. It matches at the start or the end of a word. By itself, it results in a zero-length match. ‹\b› is an anchor, just like the tokens introduced in the previous section.

Strictly speaking, ‹\b› matches in these three positions:

To run a “whole words only” search using a regular expression, simply place the word between two word boundaries, as we did with ‹\bcat\b›. The first ‹\b› requires the ‹c› to occur at the very start of the string, or after a nonword character. The second ‹\b› requires the ‹t› to occur at the very end of the string, or before a nonword character.

Line break characters are nonword characters. ‹\b› will match after a line break if the line break is immediately followed by a word character. It will also match before a line break immediately preceded by a word character. So a word that occupies a whole line by itself will be found by a “whole words only” search. ‹\b› is unaffected by “multiline” mode or ‹(?m)›, which is one of the reasons why this book refers to “multiline” mode as “^ and $ match at line breaks” mode.

None of the flavors discussed in this book have separate tokens for matching only before or only after a word. Unless you wanted to create a regex that consists of nothing but a word boundary, these aren’t needed. The tokens before or after the ‹\b› in your regular expression will determine where ‹\b› can match. The ‹\b› in ‹\bx› and ‹!\b› could match only at the start of a word. The ‹\b› in ‹x\b› and ‹\b!› could match only at the end of a word. ‹x\bx› and ‹!\b!› can never match anywhere.

If you really want to match only the position before a word or only after a word, you can do so with lookahead and lookbehind. Recipe 2.16 explains lookahead and lookbehind. This method does not work with JavaScript and Ruby 1.8 because these flavors do not support lookbehind. The regex ‹(?<!\w)(?=\w)› matches the start of a word by checking that the character before the match position is not a word character, and that the character after the match position is a word character. ‹(?<=\w)(?!\w)› does the opposite: it matches the end of the word by checking that the preceding character is a word character, and that the following character is not a word character. It’s important to use negative lookaround with ‹\w› rather than positive lookaround with ‹\W› to check for the absence of a word character. ‹(?<!\w)› matches at the start of the string because there is no word character (or any character at all) before the start of the string. But ‹(?<=\W)› never matches at the start of the string. ‹(?!\w)› matches at the end of the string for the same reason. So our two lookaround constructs will correctly match the start of the string if the string begins with a word and the end of the string if it ends with a word.

‹\B› matches at every position in the subject text where ‹\b› does not match. ‹\B› matches at every position that is not at the start or end of a word.

Strictly speaking, ‹\B› matches in these five positions:

‹\Bcat\B› matches cat in staccato, but not in My cat is brown, category, or bobcat.

To do the opposite of a “whole words only” search (i.e., excluding My cat is brown and including staccato, category, and bobcat), you need to use alternation to combine ‹\Bcat› and ‹cat\B› into ‹\Bcat|cat\B›. ‹\Bcat› matches cat in staccato and bobcat. ‹cat\B› matches cat in category (and staccato if ‹\Bcat› hadn’t already taken care of that). Recipe 2.8 explains alternation.

\u2122

\U00002122

These regexes work in Python 2.x only when quoted as Unicode strings: u"\u2122" or u"\U00002122".

\x{2122}

PCRE must be compiled with UTF-8 support; in PHP, turn on UTF-8 support with the /u pattern modifier.

\u{2122}

Ruby 1.8 does not support Unicode regular expressions.

\p{Sc}

PCRE must be compiled with UTF-8 support; in PHP, turn on UTF-8 support with the /u pattern modifier. JavaScript and Python do not support Unicode properties. XRegExp adds support for Unicode properties to JavaScript. Ruby 1.8 does not support Unicode regular expressions.

\p{IsGreekExtended}

\p{InGreekExtended}

JavaScript, PCRE, Python, and Ruby 1.9 do not support Unicode blocks. They do support Unicode code points, which you can use to match blocks as shown in the section in this recipe. XRegExp adds support for Unicode blocks to JavaScript.

\p{Greek}

\p{IsGreek}

Unicode script support requires PCRE 6.5 or later, and PCRE must be compiled with UTF-8 support. In PHP, turn on UTF-8 support with the /u pattern modifier. .NET, JavaScript, and Python do not support Unicode properties. XRegExp adds support for Unicode properties to JavaScript. Ruby 1.8 does not support Unicode regular expressions.

\X

PCRE and Perl have a dedicated token for matching graphemes. PCRE must be compiled with UTF-8 support; in PHP, turn on UTF-8 support with the /u pattern modifier.

(?>\P{M}\p{M}*)

(?:\P{M}\p{M}*)

.NET, Java, XRegExp, and Ruby 1.9 do not have a token for matching graphemes. But they do support Unicode categories, which we can use to emulate matching graphemes.

JavaScript (without XRegExp) and Python do not support Unicode properties. Ruby 1.8 does not support Unicode regular expressions.

A code point is one entry in the Unicode character database. A code point is not the same as a character, depending on the meaning you give to “character.” What appears as a character on screen is called a grapheme in Unicode.

The Unicode code point U+2122 represents the “trademark sign” character. You can match this with ‹\u2122›, ‹\u{2122}›, or ‹\x{2122}›, depending on the regex flavor you’re working with.

The ‹\u› syntax requires exactly four hexadecimal digits. This means you can only use it for Unicode code points U+0000 through U+FFFF.

‹\u{⋯}› and ‹\x{⋯}› allow between one and six hexadecimal digits between the braces, supporting all code points U+000000 through U+10FFFF. You can match U+00E0 with ‹\x{E0}› or ‹\x{00E0}›. Code points U+100000 and above are used very infrequently. They are poorly supported by fonts and operating systems.

Python’s regular expression engine has no support for Unicode code points. Literal Unicode strings in Python 2.x and literal text strings in Python 3.x do have escapes for Unicode code points. \u0000 through \uFFFF represent Unicode code points U+0000 through U+FFFF. \U00000000 through \U0010FFFF represent all Unicode code points. You have to specify eight hexadecimal numbers after \U, even though there are no Unicode code points beyond U+10FFFF.

When hard-coding regular expressions as literal strings in your Python code, you can directly use ‹\u2122› and ‹\U00002122› in your regexes. When reading regexes from a file or receiving them from user input, these Unicode escapes will not work if you pass the string you read or received directly to re.compile(). In Python 2.x, you can decode the Unicode escapes by calling string.decode('unicode-escape'). In Python 3.x you can call string.encode('utf-8').decode('unicode-escape').

Code points can be used inside and outside character classes.

Each Unicode code point fits into a single Unicode category. There are 30 Unicode categories, specified with a code consisting of two letters. These are grouped into 7 super-categories that are specified with a single letter.

‹\p{Ll}› matches a single code point that is in the Ll, or “lowercase letter,” category. ‹\p{L}› is a quick way of writing ‹[\p{Ll}\p{Lu}\p{Lt}\p{Lm}\p{Lo}]› that matches a single code point in any of the “letter” categories.

‹\P› is the negated version of ‹\p›. ‹\P{Ll}› matches a single code point that is not in the Ll category. ‹\P{L}› matches a single code point that does not have any of the “letter” properties. This is not the same as ‹[\P{Ll}\P{Lu}\P{Lt}\P{Lm}\P{Lo}]›, which matches all code points. ‹\P{Ll}› matches the code points in the Lu category (and every other category except Ll), whereas ‹\P{Lu}› includes the Ll code points. Combining just these two in a code point class already matches all possible code points.

The Unicode character database divides all the code points into blocks. Each block consists of a single range of code points. The code points U+0000 through U+FFFF are divided into 156 blocks in version 6.1 of the Unicode standard:

A Unicode block is a single, contiguous range of code points. Although many blocks have the names of Unicode scripts and Unicode categories, they do not correspond 100% with them. The name of a block only indicates its primary use.

The Currency block does not include the dollar and yen symbols. Those are found in the BasicLatin and Latin-1Supplement blocks, for historical reasons. Both are in the Currency Symbol category. To match any currency symbol, use ‹\p{Sc}› instead of ‹\p{InCurrency}›.

Most blocks include unassigned code points, which are in the category ‹\p{Cn}›. None of the other Unicode categories, and none of the Unicode scripts, include unassigned code points.

The ‹\p{InBlockName}› syntax works with .NET, XRegExp, and Perl. Java uses the ‹\p{IsBlockName}› syntax.

Perl also supports the Is variant, but we recommend you stick with the In syntax, to avoid confusion with Unicode scripts. For scripts, Perl supports ‹\p{Script}› and ‹\p{IsScript}›, but not ‹\p{InScript}›.

The Unicode standard stipulates that block names should be case insensitive, and that any differences in spaces, hyphens, or underscores should be ignored. Most regex flavors are not this flexible, unfortunately. All versions of .NET and Java 4 require the block names to be capitalized as shown in the preceding list. Perl 5.8 and later and Java 5 and later allow any mixture of case. Perl, Java, and .NET all support the notation with hyphens and without spaces used in the preceding list. We recommend you use this notation. Of the flavors discussed in this book, only XRegExp and Perl 5.12 and later are fully flexible with regard to spaces, hyphens, and underscores in Unicode block names.

Each Unicode code point, except unassigned ones, is part of exactly one Unicode script. Unassigned code points are not part of any script. The assigned code points up to U+FFFF are assigned to these 72 scripts in version 6.1 of the Unicode standard:

A script is a group of code points used by a particular human writing system. Some scripts, such as Thai, correspond with a single human language. Other scripts, such as Latin, span multiple languages. Some languages are composed of multiple scripts. For instance, there is no Japanese Unicode script; instead, Unicode offers the Hiragana, Katakana, Han, and Latin scripts that Japanese documents are usually composed of.

We listed the Common script first, out of alphabetical order. This script contains all sorts of characters that are common to a wide range of scripts, such as punctuation, whitespace, and miscellaneous symbols.

Java requires the name of the script to be prefixed with Is, as in ‹\p{IsYi}›. Perl allows the Is prefix, but doesn’t require it. XRegExp, PCRE, and Ruby do not allow the Is prefix.

The Unicode standard stipulates that script names should be case insensitive, and that any differences in spaces, hyphens, or underscores should be ignored. Most regex flavors are not this flexible, unfortunately. The notation with the words in the script names capitalized and with underscores between the words works with all flavors in this book that support Unicode scripts.

The difference between code points and characters comes into play when there are combining marks. The Unicode code point U+0061 is “Latin small letter a,” whereas U+00E0 is “Latin small letter a with grave accent.” Both represent what most people would describe as a character.

U+0300 is the “combining grave accent” combining mark. It can be used sensibly only after a letter. A string consisting of the Unicode code points U+0061 U+0300 will be displayed as à, just like U+00E0. The combining mark U+0300 is displayed on top of the character U+0061.

The reason for these two different ways of displaying an accented letter is that many historical character sets encode “a with grave accent” as a single character. Unicode’s designers thought it would be useful to have a one-on-one mapping with popular legacy character sets, in addition to the Unicode way of separating marks and base letters, which makes arbitrary combinations not supported by legacy character sets possible.

What matters to you as a regex user is that all regex flavors discussed in this book operate on code points rather than graphical characters. When we say that the regular expression ‹.› matches a single character, it really matches just a single code point. If your subject text consists of the two code points U+0061 U+0300, which can be represented as the string literal "\u0061\u0300" in a programming language such as Java, the dot will match only the code point U+0061, or a, without the accent U+0300. The regex ‹..› will match both.

Perl and PCRE offer a special regex token ‹\X›, which matches any single Unicode grapheme. Essentially, it is the Unicode version of the venerable dot. ‹\X› will find two matches in the text àà, regardless of how it is encoded. If it is encoded as \u00E0\u0061\u0300 the first match is \u00E0, and the second \u0061\u0300. The dot, which matches any single Unicode code point, would find three matches as it matches \u00E0, \u0061, and \u0300 separately.

The rules for exactly which combinations of Unicode code points are considered graphemes are quite complicated.^[2] Generally speaking, to match a grapheme we need to match any character that is not a mark and all the marks that follow it, if any. We can match this with the regex ‹(?>\P{M}\p{M}*)› in all regex flavors that support Unicode but not the ‹\X› token for graphemes. ‹\P{M}› matches any character that is not in the Mark category. ‹\p{M}*› matches all the marks, if any, that follow it.

We put these two regex tokens in an atomic group to make sure the ‹\p{M}*› won’t backtrack if any following regex tokens fail to match. ‹\X{2}.› does not match àà, because there is nothing left for the dot to match after ‹\X{2}› has matched the two accented letters. ‹(?>\P{M}\p{M}*){2}.› does not match àà for the same reason. But ‹(?:\P{M}\p{M}*){2}.› with an non-capturing group does match àà if it is encoded as \u00E0\u0061\u0300. Upon the second iteration of the group, ‹\p{M}*› will match \u0300. The dot will then fail to match. This causes the regex to backtrack, forcing ‹\p{M}*› to give up its match, allowing the dot to match \u0300.

JavaScript’s regex engine does not support atomic grouping. This is not a feature that could be added by XRegExp, because XRegExp still relies on JavaScript’s regex engine for the actual pattern matching. So when using XRegExp, ‹(?:\P{M}\p{M}*)› is the closest we can get to emulating ‹\X›. Without the atomic group, you’ll have to keep in mind that ‹\p{M}*› may backtrack if whatever follows ‹(?:\P{M}\p{M}*)› in your regex can match characters in the Mark category.

The uppercase ‹\P› is the negated variant of the lowercase ‹\p›. For instance, ‹\P{Sc}› matches any character that does not have the “Currency Symbol” Unicode property. ‹\P› is supported by all flavors that support ‹\p›, and for all the properties, block, and scripts that they support.

All flavors allow all the ‹\u›, ‹\x›, ‹\p›, and ‹\P› tokens they support to be used inside character classes. The character represented by the code point, or the characters in the category, block, or script, are then added to the character class. For instance, you could match a character that is either an opening quote (initial punctuation property), a closing quote (final punctuation property), or the trademark symbol (U+2122) with:

[\p{Pi}\p{Pf}\u2122]

[\p{Pi}\p{Pf}\x{2122}]

If your regular expression flavor does not support Unicode categories, blocks, or scripts, you can list the characters that are in the category, block, or script in a character class. For blocks this is very easy: each block is simply a range between two code points. The Greek Extended block comprises the characters U+1F00 to U+1FFF:

[\u1F00-\u1FFF]

[\x{1F00}-\x{1FFF}]

For most categories and many scripts, the equivalent character class is a long list of individual code points and short ranges. The characters that comprise each category and many of the scripts are scattered throughout the Unicode table. This is the Greek script:

[\u0370-\u0373\u0375-\u0377\u037A-\u037D\u0384\u0386\u0388-\u038A↵
\u038C\u038E-\u03A1\u03A3-\u03E1\u03F0-\u03FF\u1D26-\u1D2A\u1D5D-\u1D61↵
\u1D66-\u1D6A\u1DBF\u1F00-\u1F15\u1F18-\u1F1D\u1F20-\u1F45\u1F48-\u1F4D↵
\u1F50-\u1F57\u1F59\u1F5B\u1F5D\u1F5F-\u1F7D\u1F80-\u1FB4\u1FB6-\u1FC4↵
\u1FC6-\u1FD3\u1FD6-\u1FDB\u1FDD-\u1FEF\u1FF2-\u1FF4\u1FF6-\u1FFE\u2126↵
\U00010140-\U0001018A\U0001D200-\U0001D245]

We generated this regular expression using the UnicodeSet web application at http://unicode.org/cldr/utility/list-unicodeset.jsp. We entered \p{Greek} as the input, ticked the “Abbreviate” and “Escape” checkboxes, and clicked the “Show Set” button.

Only Python supports this syntax for Unicode code points as we explained earlier in this recipe in the section Unicode code point. To make this regular expression work with other regex flavors, we need to make some changes.

The regex will work with many more flavors if we remove the code points beyond U+FFFF from the character class:

[\u0370-\u0373\u0375-\u0377\u037A-\u037D\u0384\u0386\u0388-\u038A↵
\u038C\u038E-\u03A1\u03A3-\u03E1\u03F0-\u03FF\u1D26-\u1D2A\u1D5D-\u1D61↵
\u1D66-\u1D6A\u1DBF\u1F00-\u1F15\u1F18-\u1F1D\u1F20-\u1F45\u1F48-\u1F4D↵
\u1F50-\u1F57\u1F59\u1F5B\u1F5D\u1F5F-\u1F7D\u1F80-\u1FB4\u1FB6-\u1FC4↵
\u1FC6-\u1FD3\u1FD6-\u1FDB\u1FDD-\u1FEF\u1FF2-\u1FF4\u1FF6-\u1FFE\u2126]

Perl and PCRE use a different syntax for Unicode code points. In the original regex, we need to replace ‹\uFFFF› with ‹\x{FFFF}› and ‹\U0010FFFF› with ‹\x{10FFFF}›. This regex also works with Java 7.

[\x{0370}-\x{0373}\x{0375}-\x{0377}\x{037A}-\x{037D}\x{0384}\x{0386}↵
\x{0388}-\x{038A}\x{038C}\x{038E}-\x{03A1}\x{03A3}-\x{03E1}↵
\x{03F0}-\x{03FF}\x{1D26}-\x{1D2A}\x{1D5D}-\x{1D61}\x{1D66}-\x{1D6A}↵
\x{1DBF}\x{1F00}-\x{1F15}\x{1F18}-\x{1F1D}\x{1F20}-\x{1F45}↵
\x{1F48}-\x{1F4D}\x{1F50}-\x{1F57}\x{1F59}\x{1F5B}\x{1F5D}\x{1F5F}-↵
\x{1F7D}\x{1F80}-\x{1FB4}\x{1FB6}-\x{1FC4}\x{1FC6}-\x{1FD3}\x{1FD6}-↵
\x{1FDB}\x{1FDD}-\x{1FEF}\x{1FF2}-\x{1FF4}\x{1FF6}-\x{1FFE}\x{2126}↵
\x{10140}-\x{10178}\x{10179}-\x{10189}\x{1018A}\x{1D200}-\x{1D245}]

In the regex ‹\b(Mary|Jane|Sue)\b›, we need the parentheses for grouping only. Instead of using a capturing group, we could use a noncapturing group:

\b(?:Mary|Jane|Sue)\b

The three characters ‹(?:› open the noncapturing group. The parenthesis ‹)› closes it. The noncapturing group provides the same grouping functionality, but does not capture anything.

When counting opening parentheses of capturing groups to determine their numbers, do not count the parenthesis of the noncapturing group. This is the main benefit of noncapturing groups: you can add them to an existing regex without upsetting the references to numbered capturing groups.

Another benefit of noncapturing groups is performance. If you’re not going to use a backreference to a particular group (Recipe 2.10), reinsert it into the replacement text (Recipe 2.21), or retrieve its match in source code (Recipe 3.9), a capturing group adds unnecessary overhead that you can eliminate by using a noncapturing group. In practice, you’ll hardly notice the performance difference, unless you’re using the regex in a tight loop and/or on lots of data.

In the variation of Recipe 2.1, we explain that .NET, Java, PCRE, Perl, and Ruby support local mode modifiers, using the mode toggles: ‹sensitive(?i)caseless(?-i)sensitive›. Although this syntax also involves parentheses, a toggle such as ‹(?i)› does not involve any grouping.

Instead of using toggles, you can specify mode modifiers in a noncapturing group:

\b(?i:Mary|Jane|Sue)\b

sensitive(?i:caseless)sensitive

Adding mode modifiers to a noncapturing group sets that mode for the part of the regular expression inside the group. The previous settings are restored at the closing parenthesis. Since case sensitivity is the default, only the part of the regex inside:

(?i:⋯)

is case insensitive.

You can combine multiple modifiers. ‹(?ism:⋯)›. Use a hyphen to turn off modifiers: ‹(?-ism:⋯)› turns off the three options. ‹(?i-sm)› turns on case insensitivity (i), and turns off both “dot matches line breaks” (s) and “^ and $ match at line breaks” (m). These options are explained in Recipes 2.4 and 2.5.

\b(?<year>\d\d\d\d)-(?<month>\d\d)-(?<day>\d\d)\b

\b(?'year'\d\d\d\d)-(?'month'\d\d)-(?'day'\d\d)\b

\b(?P<year>\d\d\d\d)-(?P<month>\d\d)-(?P<day>\d\d)\b

\b\d\d(?<magic>\d\d)-\k<magic>-\k<magic>\b

\b\d\d(?'magic'\d\d)-\k'magic'-\k'magic'\b

\b\d\d(?P<magic>\d\d)-(?P=magic)-(?P=magic)\b

Recipes 2.9 and 2.10 illustrate capturing groups and backreferences. To be more precise: these recipes use numbered capturing groups and numbered backreferences. Each group automatically gets a number, which you use for the backreference.

Modern regex flavors support named capturing groups in addition to numbered groups. The only difference between named and numbered groups is your ability to assign a descriptive name, instead of being stuck with automatic numbers. Named groups make your regular expression more readable and easier to maintain. Inserting a capturing group into an existing regex can change the numbers assigned to all the capturing groups. Names that you assign remain the same.

Python was the first regular expression flavor to support named capture. It uses the syntax ‹(?P<name>regex)›. The name must consist of word characters matched by ‹\w›. ‹(?P<name>› is the group’s opening bracket, and ‹)› is the closing bracket.

The designers of the .NET Regex class came up with their own syntax for named capture, using two interchangeable variants. ‹(?<name>regex)› mimics Python’s syntax, minus the P. The name must consist of word characters matched by ‹\w›. ‹(?<name>› is the group’s opening bracket, and ‹)› is the closing bracket.

The angle brackets in the named capture syntax are annoying when you’re coding in XML, or writing this book in DocBook XML. That’s the reason for .NET’s alternate named capture syntax: ‹(?'name'regex)›. The angle brackets are replaced with single quotes. Choose whichever syntax is easier for you to type. Their functionality is identical.

Perhaps due to .NET’s popularity over Python, the .NET syntax seems to be the one that other regex library developers prefer to copy. Perl 5.10 and later have it, and so does the Oniguruma engine in Ruby 1.9. Perl 5.10 and Ruby 1.9 support both the syntax using angle brackets and single quotes. Java 7 also copied the .NET syntax, but only the variant using angle brackets. Standard JavaScript does not support named capture. XRegExp adds support for named capture using the .NET syntax, but only the variant with angle brackets.

PCRE copied Python’s syntax long ago, at a time when Perl did not support named capture at all. PCRE 7, the version that adds the new features in Perl 5.10, supports both the .NET syntax and the Python syntax. Perhaps as a testament to the success of PCRE, in a reverse compatibility move, Perl 5.10 also supports the Python syntax. In PCRE and Perl 5.10, the functionality of the .NET syntax and the Python syntax for named capture is identical.

Choose the syntax that is most useful to you. If you’re coding in PHP and you want your code to work with older versions of PHP that incorporate older versions of PCRE, use the Python syntax. If you don’t need compatibility with older versions and you also work with .NET or Ruby, the .NET syntax makes it easier to copy and paste between all these languages. If you’re unsure, use the Python syntax for PHP/PCRE. People recompiling your code with an older version of PCRE are going to be unhappy if the regexes in your code suddenly stop working. When copying a regex to .NET or Ruby, deleting a few Ps is easy enough.

Documentation for PCRE 7 and Perl 5.10 barely mention the Python syntax, but it is by no means deprecated. For PCRE and PHP, we actually recommend it.

With named capture comes named backreferences. Just as named capturing groups are functionally identical to numbered capturing groups, named backreferences are functionally identical to numbered backreferences. They’re just easier to read and maintain.

Python uses the syntax ‹(?P=name)› to create a backreference to the group name. Although this syntax uses parentheses, the backreference is not a group. You cannot put anything between the name and the closing parenthesis. A backreference ‹(?P=name)› is a singular regex token, just like ‹\1›. PCRE and Perl 5.10 also support the Python syntax for named backreferences.

.NET uses the syntax ‹\k<name>› and ‹\k'name'›. The two variants are identical in functionality, and you can freely mix them. A named group created with the bracket syntax can be referenced with the quote syntax, and vice versa. Perl 5.10, PCRE 7, and Ruby 1.9 also support the .NET syntax for named backreferences. Java 7 and XRegExp support only the variant using angle brackets.

We strongly recommend you don’t mix named and numbered groups in the same regex. Different flavors follow different rules for numbering unnamed groups that appear between named groups. Perl 5.10, Ruby 1.9, Java 7, and XRegExp copied .NET’s syntax, but they do not follow .NET’s way of numbering named capturing groups or of mixing numbered capturing groups with named groups. Instead of trying to explain the differences, we simply recommend not mixing named and numbered groups. Avoid the confusion and either give all unnamed groups a name or make them noncapturing.

Perl 5.10, Ruby 1.9, and .NET allow multiple named capturing groups to share the same name. We take advantage of this in the solutions for recipes 4.5, 8.7, and 8.19. When a regular expression uses alternation to find different variations of certain text, using capturing groups with the same name makes it easy to extract parts from the match, regardless of which alternative actually matched the text. The section Pure regular expression in Recipe 4.5 uses alternation to separately match dates in months of different lengths. Each alternative matches the day and the month. By using the same group names “day” and “month” in all the alternatives, we only need to query two capturing groups to retrieve the day and the month after the regular expression finds a match.

All the other flavors in this book that support named capture treat multiple groups with the same name as an error.

\b\d{100}\b

\b[a-f0-9]{1,8}\b

\b[a-f0-9]{1,8}h?\b

\d*\.\d+(e\d+)?

The quantifier ‹{n}›, where n is a nonnegative integer, repeats the preceding regex token n number of times. The ‹\d{100}› in ‹\b\d{100}\b› matches a string of 100 digits. You could achieve the same by typing ‹\d› 100 times.

‹{1}› repeats the preceding token once, as it would without any quantifier. ‹ab{1}c› is the same regex as ‹abc›.

‹{0}› repeats the preceding token zero times, essentially deleting it from the regular expression. ‹ab{0}c› is the same regex as ‹ac›.

For variable repetition, we use the quantifier ‹{n,m}›, where n is a nonnegative integer and m is greater than n. ‹\b[a-f0-9]{1,8}\b› matches a hexadecimal number with one to eight digits. With variable repetition, the order in which the alternatives are attempted comes into play. Recipe 2.13 explains that in detail.

If n and m are equal, we have fixed repetition. ‹\b\d{100,100}\b› is the same regex as ‹\b\d{100}\b›.

The quantifier ‹{n,}›, where n is a nonnegative integer, allows for infinite repetition. Essentially, infinite repetition is variable repetition without an upper limit.

‹\d{1,}› matches one or more digits, and ‹\d+› does the same. A plus after a regex token that’s not a quantifier means “one or more.” Recipe 2.13 shows the meaning of a plus after a quantifier.

‹\d{0,}› matches zero or more digits, and ‹\d*› does the same. The asterisk always means “zero or more.” In addition to allowing infinite repetition, ‹{0,}› and the asterisk also make the preceding token optional.

If we use variable repetition with n set to zero, we’re effectively making the token that precedes the quantifier optional. ‹h{0,1}› matches the ‹h› once or not at all. If there is no h, ‹h{0,1}› results in a zero-length match. If you use ‹h{0,1}› as a regular expression all by itself, it will find a zero-length match before each character in the subject text that is not an h. Each h will result in a match of one character (the h).

‹h?› does the same as ‹h{0,1}›. A question mark after a valid and complete regex token that is not a quantifier means “zero or once.” The next recipe shows the meaning of a question mark after a quantifier.

If you place a quantifier after the closing parenthesis of a group, the whole group is repeated. ‹(?:abc){3}› is the same as ‹abcabcabc›.

Quantifiers can be nested. ‹(e\d+)?› matches an e followed by one or more digits, or a zero-length match. In our floating-point regular expression, this is the optional exponent.

Capturing groups can be repeated. As explained in Recipe 2.9, the group’s match is captured each time the engine exits the group, overwriting any text previously matched by the group. ‹(\d\d){1,3}› matches a string of two, four, or six digits. The engine exits the group three times. When this regex matches 123456, the capturing group will hold 56, because 56 was stored by the last iteration of the group. The other two matches by the group, 12 and 34, cannot be retrieved.

‹(\d\d){3}› captures the same text as ‹\d\d\d\d(\d\d)›. If you want the capturing group to capture all two, four, or six digits rather than just the last two, you have to place the capturing group around the quantifier instead of repeating the capturing group: ‹((?:\d\d){1,3})›. Here we used a noncapturing group to take over the grouping function from the capturing group. We also could have used two capturing groups: ‹((\d\d){1,3})›. When this last regex matches 123456, ‹\1› holds 123456 and ‹\2› holds 56.

.NET’s regular expression engine is the only one that allows you to retrieve all the iterations of a repeated capturing group. If you directly query the group’s Value property, which returns a string, you’ll get 56, as with every other regular expression engine. Backreferences in the regular expression and replacement text also substitute 56, but if you use the group’s CaptureCollection, you’ll get a stack with 56, 34, and 12.

The four kinds of lookaround groups supported by modern regex flavors have the special property of giving up the text matched by the part of the regex inside the lookaround. Essentially, lookaround checks whether certain text can be matched without actually matching it.

Lookaround that looks backward is called lookbehind. This is the only regular expression construct that will traverse the text from right to left instead of from left to right. The syntax for positive lookbehind is ‹(?<=⋯)›. The four characters ‹(?<=› form the opening bracket. What you can put inside the lookbehind, here represented by ‹⋯›, varies among regular expression flavors. But simple literal text, such as ‹(?<=<b>)›, always works.

Lookbehind checks to see whether the text inside the lookbehind occurs immediately to the left of the position that the regular expression engine has reached. If you match ‹(?<=<b>)› against My <b>cat</b> is furry, the lookbehind will fail to match until the regular expression starts the match attempt at the letter c in the subject. The regex engine then enters the lookbehind group, telling it to look to the left. ‹<b>› matches to the left of c. The engine exits the lookbehind at this point, and discards any text matched by the lookbehind from the match attempt. In other words, the match-in-progress is back at where it was when the engine entered the lookbehind. In this case, the match-in-progress is the zero-length match before the c in the subject string. The lookbehind only tests or asserts that ‹<b>› can be matched; it does not actually match it. Lookaround constructs are therefore called zero-length assertions.

After the lookbehind has matched, the shorthand character class ‹\w+› attempts to match one or more word characters. It matches cat. The ‹\w+› is not inside any kind of lookaround or group, and so it matches the text cat normally. We say that ‹\w+› matches and consumes cat, whereas lookaround can match something but can never consume anything.

Lookaround that looks forward, in the same direction that the regular expression normally traverses the text, is called lookahead. Lookahead is equally supported by all regex flavors in this book. The syntax for positive lookahead is ‹(?=⋯)›. The three characters ‹(?=› form the opening bracket of the group. Everything you can use in a regular expression can be used inside lookahead, here represented by ‹⋯›.

When the ‹\w+› in ‹(?<=<b>)\w+(?=</b>)› has matched cat in My <b>cat</b> is furry, the regex engine enters the lookahead. The only special behavior for the lookahead at this point is that the regex engine remembers which part of the text it has matched so far, associating it with the lookahead. ‹</b>› is then matched normally. Now the regex engine exits the lookahead. The regex inside the lookahead matches, so the lookahead itself matches. The regex engine discards the text matched by the lookahead, by restoring the match-in-progress it remembered when entering the lookahead. Our overall match-in-progress is back at cat. Since this is also the end of our regular expression, cat becomes the final match result.

‹(?!⋯)›, with an exclamation point instead of an equals sign, is negative lookahead. Negative lookahead works just like positive lookahead, except that whereas positive lookahead matches when the regex inside the lookahead matches, negative lookahead matches when the regex inside the lookahead fails to match.

The matching process is exactly the same. The engine saves the match-in-progress when entering the negative lookahead, and attempts to match the regex inside the lookahead normally. If the sub-regex matches, the lookahead fails, and the regex engine backtracks. If the sub-regex fails to match, the engine restores the match-in-process and proceeds with the remainder of the regex.

Similarly, (?<!⋯) is negative lookbehind. Negative lookbehind matches when none of the alternatives inside the lookbehind can be found looking backward from the position the regex has reached in the subject text.

Lookahead is easy. All regex flavors discussed in this book allow you to put a complete regular expression inside the lookahead. Everything you can use in a regular expression can be used inside lookahead. You can even nest other lookahead and lookbehind groups inside lookahead. Your brain might get into a twist, but the regex engine will handle everything nicely.

Lookbehind is a different story. Regular expression software has always been designed to search the text from left to right only. Searching backward is often implemented as a bit of a hack: the regex engine determines how many characters you put inside the lookbehind, jumps back that many characters, and then compares the text in the lookbehind with the text in the subject from left to right.

For this reason, the earliest implementations allowed only fixed-length literal text inside lookbehind. Perl and Python still require lookbehind to have a fixed length, but they do allow fixed-length regex tokens such as character classes, and allow alternation as long as all alternatives match the same number of characters.

PCRE and Ruby 1.9 take this one step further. They allow alternatives of different lengths inside lookbehind, as long as the length of each alternative is constant. They can handle something like ‹(?<=one|two|three|forty-two|gr[ae]y)›, but nothing more complex than that.

Internally, PCRE and Ruby 1.9 expand this into six lookbehind tests. First, they jump back three characters to test ‹one|two›, then four characters to test ‹gray|grey›, then five to test ‹three›, and finally nine to test ‹forty-two›.

Java takes lookbehind one step further. Java allows any finite-length regular expression inside lookbehind. This means you can use anything except the infinite quantifiers ‹*›, ‹+›, and ‹{42,}› inside lookbehind. Internally, Java’s regex engine calculates the minimum and maximum length of the text that could possibly be matched by the part of the regex in the lookbehind. It then jumps back the minimum number of characters, and applies the regex in the lookbehind from left to right. If this fails, the engine jumps back one more character and tries again, until either the lookbehind matches or the maximum number of characters has been tried.

If all this sounds rather inefficient, it is. Lookbehind is very convenient, but it won’t break any speed records. Later, we present a solution for JavaScript and Ruby 1.8, which don’t support lookbehind at all. This solution is actually far more efficient than using lookbehind.

The regular expression engine in the .NET Framework is the only one in the world^[5] that can actually apply a full regular expression from right to left. .NET allows you to use anything inside lookbehind, and it will actually apply the regular expression from right to left. Both the regular expression inside the lookbehind and the subject text are scanned from right to left.

If you use lookbehind at the start of the regex or lookahead at the end of the regex, the net effect is that you’re requiring something to appear before or after the regex match, without including it in the match. If you use lookaround in the middle of your regular expression, you can apply multiple tests to the same text.

In Flavor-Specific Features (a subsection of Recipe 2.3), we showed how to use character class subtraction to match a Thai digit. Only .NET and Java support character class subtraction.

A character is a Thai digit if it is both a Thai character (any sort) and a digit (any script). With lookahead, you can test both requirements on the same character:

(?=\p{Thai})\p{N}

This regex works only with the three flavors that support Unicode scripts, as we explain in Recipe 2.7. But the principle of using lookahead to match the same character more than once works with all flavors discussed in this book.

When the regular expression engine searches for ‹(?=\p{Thai})\p{N}›, it starts by entering the lookahead at each position in the string where it begins a match attempt. If the character at that position is not in the Thai script (i.e., ‹\p{Thai}› fails to match), the lookahead fails. This causes the whole match attempt to fail, forcing the regex engine to start over at the next character.

When the regex reaches a Thai character, ‹\p{Thai}› matches. Thus, the ‹(?=\p{Thai})› lookaround matches, too. As the engine exits the lookaround, it restores the match-in-progress. In this case, that’s the zero-length match before the character just found to be Thai. Next up is ‹\p{N}›. Because the lookahead discarded its match, ‹\p{N}› is compared with the same character that ‹\p{Thai}› already matched. If this character has the Unicode property Number, ‹\p{N}› matches. Since ‹\p{N}› is not inside a lookaround, it consumes the character, and we have found our Thai digit.

When the regular expression engine exits a lookaround group, it discards the text matched by the lookaround. Because the text is discarded, any backtracking positions remembered by alternation or quantifiers inside the lookaround are also discarded. This effectively makes lookahead and lookbehind atomic. Recipe 2.14 explains atomic groups in detail.

In most situations, the atomic nature of lookaround is irrelevant. A lookaround is merely an assertion to check whether the regex inside the lookaround matches or fails. How many different ways it can match is irrelevant, as it does not consume any part of the subject text.

The atomic nature comes into play only when you use capturing groups inside lookahead (and lookbehind, if your regex flavor allows you to). While the lookahead does not consume any text, the regex engine will remember which part of the text was matched by any capturing groups inside the lookahead. If the lookahead is at the end of the regex, you will indeed end up with capturing groups that match text not matched by the regular expression itself. If the lookahead is in the middle of the regex, you can end up with capturing groups that match overlapping parts of the subject text.

The only situation in which the atomic nature of lookaround can alter the overall regex match is when you use a backreference outside the lookaround to a capturing group created inside the lookaround. Consider this regular expression:

(?=(\d+))\w+\1

At first glance, you may think that this regex would match 123x12. ‹\d+› would capture 12 into the first capturing group, then ‹\w+› would match 3x, and finally ‹\1› would match 12 again.

But that never happens. The regular expression enters the lookaround and the capturing group. The greedy ‹\d+› matches 123. This match is stored into the first capturing group. The engine then exits the lookahead, resetting the match-in-progress to the start of the string, discarding the backtracking positions remembered by the greedy plus but keeping the 123 stored in the first capturing group.

Now, the greedy ‹\w+› is attempted at the start of the string. It eats up 123x12. ‹\1›, which references 123, fails at the end of the string. ‹\w+› backtracks one character. ‹\1› fails again. ‹\w+› keeps backtracking until it has given up everything except the first 1 in the subject. ‹\1› also fails to match after the first 1.

The final 12 would match ‹\1› if the regex engine could return to the lookahead and give up 123 in favor of 12, but the regex engine doesn’t do that.

The regex engine has no further backtracking positions to go to. ‹\w+› backtracked all the way, and the lookaround forced ‹\d+› to give up its backtracking positions. The match attempt fails.