Mass Deface and L. Perl is widely renowned for excellence in text processing, and regular expressions are one of the big factors behind this fame. Perl regular expressions display an efficiency and flexibility unknown in most other computer languages. Mastering even the basics of regular expressions will allow you to manipulate text with surprising ease. What is a regular expression? A regular expression is simply a string that describes a pattern. Patterns are in common use these days; examples are the patterns typed into a search engine to find web pages and the patterns used to list files in a directory, e.g., C or C. In Perl, the patterns described by regular expressions are used to search strings, extract desired parts of strings, and to do search and replace operations. Regular expressions have the undeserved reputation of being abstract and difficult to understand. Regular expressions are constructed using simple concepts like conditionals and loops and are no more difficult to understand than the corresponding C conditionals and C loops in the Perl language itself. In fact, the main challenge in learning regular expressions is just getting used to the terse notation used to express these concepts. This tutorial flattens the learning curve by discussing regular expression concepts, along with their notation, one at a time and with many examples. The first part of the tutorial will progress from the simplest word searches to the basic regular expression concepts. If you master the first part, you will have all the tools needed to solve about 98% of your needs. The second part of the tutorial is for those comfortable with the basics and hungry for more power tools. It discusses the more advanced regular expression operators and introduces the latest cutting-edge innovations. A note: to save time, 'regular expression' is often abbreviated as regexp or regex. Regexp is a more natural abbreviation than regex, but is harder to pronounce. The Perl pod documentation is evenly split on regexp vs regex; in Perl, there is more than one way to abbreviate it. We'll use regexp in this tutorial. =head1 Part 1: The basics =head2 Simple word matching The simplest regexp is simply a word, or more generally, a string of characters. A regexp consisting of a word matches any string that contains that word: "Hello World" =~ /World/; # matches What is this Perl statement all about? C<"Hello World"> is a simple double-quoted string. C is the regular expression and the C enclosing C tells Perl to search a string for a match. The operator C<=~> associates the string with the regexp match and produces a true value if the regexp matched, or false if the regexp did not match. In our case, C matches the second word in C<"Hello World">, so the expression is true. Expressions like this are useful in conditionals: if ("Hello World" =~ /World/) { print "It matches\n"; } else { print "It doesn't match\n"; } There are useful variations on this theme. The sense of the match can be reversed by using the C operator: if ("Hello World" !~ /World/) { print "It doesn't match\n"; } else { print "It matches\n"; } The literal string in the regexp can be replaced by a variable: $greeting = "World"; if ("Hello World" =~ /$greeting/) { print "It matches\n"; } else { print "It doesn't match\n"; } If you're matching against the special default variable C<$_>, the C<$_ =~> part can be omitted: $_ = "Hello World"; if (/World/) { print "It matches\n"; } else { print "It doesn't match\n"; } And finally, the C default delimiters for a match can be changed to arbitrary delimiters by putting an C<'m'> out front: "Hello World" =~ m!World!; # matches, delimited by '!' "Hello World" =~ m{World}; # matches, note the matching '{}' "/usr/bin/perl" =~ m"/perl"; # matches after '/usr/bin', # '/' becomes an ordinary char C, C, and C all represent the same thing. When, e.g., the quote (C<">) is used as a delimiter, the forward slash C<'/'> becomes an ordinary character and can be used in this regexp without trouble. Let's consider how different regexps would match C<"Hello World">: "Hello World" =~ /world/; # doesn't match "Hello World" =~ /o W/; # matches "Hello World" =~ /oW/; # doesn't match "Hello World" =~ /World /; # doesn't match The first regexp C doesn't match because regexps are case-sensitive. The second regexp matches because the substring S> occurs in the string S>. The space character ' ' is treated like any other character in a regexp and is needed to match in this case. The lack of a space character is the reason the third regexp C<'oW'> doesn't match. The fourth regexp C<'World '> doesn't match because there is a space at the end of the regexp, but not at the end of the string. The lesson here is that regexps must match a part of the string I in order for the statement to be true. If a regexp matches in more than one place in the string, Perl will always match at the earliest possible point in the string: "Hello World" =~ /o/; # matches 'o' in 'Hello' "That hat is red" =~ /hat/; # matches 'hat' in 'That' With respect to character matching, there are a few more points you need to know about. First of all, not all characters can be used 'as is' in a match. Some characters, called I, are reserved for use in regexp notation. The metacharacters are {}[]()^$.|*+?\ The significance of each of these will be explained in the rest of the tutorial, but for now, it is important only to know that a metacharacter can be matched by putting a backslash before it: "2+2=4" =~ /2+2/; # doesn't match, + is a metacharacter "2+2=4" =~ /2\+2/; # matches, \+ is treated like an ordinary + "The interval is [0,1)." =~ /[0,1)./ # is a syntax error! "The interval is [0,1)." =~ /\[0,1\)\./ # matches "#!/usr/bin/perl" =~ /#!\/usr\/bin\/perl/; # matches In the last regexp, the forward slash C<'/'> is also backslashed, because it is used to delimit the regexp. This can lead to LTS (leaning toothpick syndrome), however, and it is often more readable to change delimiters. "#!/usr/bin/perl" =~ m!#\!/usr/bin/perl!; # easier to read The backslash character C<'\'> is a metacharacter itself and needs to be backslashed: 'C:\WIN32' =~ /C:\\WIN/; # matches In addition to the metacharacters, there are some ASCII characters which don't have printable character equivalents and are instead represented by I. Common examples are C<\t> for a tab, C<\n> for a newline, C<\r> for a carriage return and C<\a> for a bell (or alert). If your string is better thought of as a sequence of arbitrary bytes, the octal escape sequence, e.g., C<\033>, or hexadecimal escape sequence, e.g., C<\x1B> may be a more natural representation for your bytes. Here are some examples of escapes: "1000\t2000" =~ m(0\t2) # matches "1000\n2000" =~ /0\n20/ # matches "1000\t2000" =~ /\000\t2/ # doesn't match, "0" ne "\000" "cat" =~ /\o{143}\x61\x74/ # matches in ASCII, but a weird way # to spell cat If you've been around Perl a while, all this talk of escape sequences may seem familiar. Similar escape sequences are used in double-quoted strings and in fact the regexps in Perl are mostly treated as double-quoted strings. This means that variables can be used in regexps as well. Just like double-quoted strings, the values of the variables in the regexp will be substituted in before the regexp is evaluated for matching purposes. So we have: $foo = 'house'; 'housecat' =~ /$foo/; # matches 'cathouse' =~ /cat$foo/; # matches 'housecat' =~ /${foo}cat/; # matches So far, so good. With the knowledge above you can already perform searches with just about any literal string regexp you can dream up. Here is a I emulation of the Unix grep program: % cat > simple_grep #!/usr/bin/perl $regexp = shift; while (<>) { print if /$regexp/; } ^D % chmod +x simple_grep % simple_grep abba /usr/dict/words Babbage cabbage cabbages sabbath Sabbathize Sabbathizes sabbatical scabbard scabbards This program is easy to understand. C<#!/usr/bin/perl> is the standard way to invoke a perl program from the shell. S> saves the first command line argument as the regexp to be used, leaving the rest of the command line arguments to be treated as files. S) >>> loops over all the lines in all the files. For each line, S> prints the line if the regexp matches the line. In this line, both C and C use the default variable C<$_> implicitly. With all of the regexps above, if the regexp matched anywhere in the string, it was considered a match. Sometimes, however, we'd like to specify I in the string the regexp should try to match. To do this, we would use the I metacharacters C<^> and C<$>. The anchor C<^> means match at the beginning of the string and the anchor C<$> means match at the end of the string, or before a newline at the end of the string. Here is how they are used: "housekeeper" =~ /keeper/; # matches "housekeeper" =~ /^keeper/; # doesn't match "housekeeper" =~ /keeper$/; # matches "housekeeper\n" =~ /keeper$/; # matches The second regexp doesn't match because C<^> constrains C to match only at the beginning of the string, but C<"housekeeper"> has keeper starting in the middle. The third regexp does match, since the C<$> constrains C to match only at the end of the string. When both C<^> and C<$> are used at the same time, the regexp has to match both the beginning and the end of the string, i.e., the regexp matches the whole string. Consider "keeper" =~ /^keep$/; # doesn't match "keeper" =~ /^keeper$/; # matches "" =~ /^$/; # ^$ matches an empty string The first regexp doesn't match because the string has more to it than C. Since the second regexp is exactly the string, it matches. Using both C<^> and C<$> in a regexp forces the complete string to match, so it gives you complete control over which strings match and which don't. Suppose you are looking for a fellow named bert, off in a string by himself: "dogbert" =~ /bert/; # matches, but not what you want "dilbert" =~ /^bert/; # doesn't match, but .. "bertram" =~ /^bert/; # matches, so still not good enough "bertram" =~ /^bert$/; # doesn't match, good "dilbert" =~ /^bert$/; # doesn't match, good "bert" =~ /^bert$/; # matches, perfect Of course, in the case of a literal string, one could just as easily use the string comparison S> and it would be more efficient. The C<^...$> regexp really becomes useful when we add in the more powerful regexp tools below. =head2 Using character classes Although one can already do quite a lot with the literal string regexps above, we've only scratched the surface of regular expression technology. In this and subsequent sections we will introduce regexp concepts (and associated metacharacter notations) that will allow a regexp to represent not just a single character sequence, but a I of them. One such concept is that of a I. A character class allows a set of possible characters, rather than just a single character, to match at a particular point in a regexp. Character classes are denoted by brackets C<[...]>, with the set of characters to be possibly matched inside. Here are some examples: /cat/; # matches 'cat' /[bcr]at/; # matches 'bat, 'cat', or 'rat' /item[0123456789]/; # matches 'item0' or ... or 'item9' "abc" =~ /[cab]/; # matches 'a' In the last statement, even though C<'c'> is the first character in the class, C<'a'> matches because the first character position in the string is the earliest point at which the regexp can match. /[yY][eE][sS]/; # match 'yes' in a case-insensitive way # 'yes', 'Yes', 'YES', etc. This regexp displays a common task: perform a case-insensitive match. Perl provides a way of avoiding all those brackets by simply appending an C<'i'> to the end of the match. Then C can be rewritten as C. The C<'i'> stands for case-insensitive and is an example of a I of the matching operation. We will meet other modifiers later in the tutorial. We saw in the section above that there were ordinary characters, which represented themselves, and special characters, which needed a backslash C<\> to represent themselves. The same is true in a character class, but the sets of ordinary and special characters inside a character class are different than those outside a character class. The special characters for a character class are C<-]\^$> (and the pattern delimiter, whatever it is). C<]> is special because it denotes the end of a character class. C<$> is special because it denotes a scalar variable. C<\> is special because it is used in escape sequences, just like above. Here is how the special characters C<]$\> are handled: /[\]c]def/; # matches ']def' or 'cdef' $x = 'bcr'; /[$x]at/; # matches 'bat', 'cat', or 'rat' /[\$x]at/; # matches '$at' or 'xat' /[\\$x]at/; # matches '\at', 'bat, 'cat', or 'rat' The last two are a little tricky. In C<[\$x]>, the backslash protects the dollar sign, so the character class has two members C<$> and C. In C<[\\$x]>, the backslash is protected, so C<$x> is treated as a variable and substituted in double quote fashion. The special character C<'-'> acts as a range operator within character classes, so that a contiguous set of characters can be written as a range. With ranges, the unwieldy C<[0123456789]> and C<[abc...xyz]> become the svelte C<[0-9]> and C<[a-z]>. Some examples are /item[0-9]/; # matches 'item0' or ... or 'item9' /[0-9bx-z]aa/; # matches '0aa', ..., '9aa', # 'baa', 'xaa', 'yaa', or 'zaa' /[0-9a-fA-F]/; # matches a hexadecimal digit /[0-9a-zA-Z_]/; # matches a "word" character, # like those in a Perl variable name If C<'-'> is the first or last character in a character class, it is treated as an ordinary character; C<[-ab]>, C<[ab-]> and C<[a\-b]> are all equivalent. The special character C<^> in the first position of a character class denotes a I, which matches any character but those in the brackets. Both C<[...]> and C<[^...]> must match a character, or the match fails. Then /[^a]at/; # doesn't match 'aat' or 'at', but matches # all other 'bat', 'cat, '0at', '%at', etc. /[^0-9]/; # matches a non-numeric character /[a^]at/; # matches 'aat' or '^at'; here '^' is ordinary Now, even C<[0-9]> can be a bother to write multiple times, so in the interest of saving keystrokes and making regexps more readable, Perl has several abbreviations for common character classes, as shown below. Since the introduction of Unicode, unless the C modifier is in effect, these character classes match more than just a few characters in the ASCII range. =over 4 =item * \d matches a digit, not just [0-9] but also digits from non-roman scripts =item * \s matches a whitespace character, the set [\ \t\r\n\f] and others =item * \w matches a word character (alphanumeric or _), not just [0-9a-zA-Z_] but also digits and characters from non-roman scripts =item * \D is a negated \d; it represents any other character than a digit, or [^\d] =item * \S is a negated \s; it represents any non-whitespace character [^\s] =item * \W is a negated \w; it represents any non-word character [^\w] =item * The period '.' matches any character but "\n" (unless the modifier C is in effect, as explained below). =item * \N, like the period, matches any character but "\n", but it does so regardless of whether the modifier C is in effect. =back The C modifier, available starting in Perl 5.14, is used to restrict the matches of \d, \s, and \w to just those in the ASCII range. It is useful to keep your program from being needlessly exposed to full Unicode (and its accompanying security considerations) when all you want is to process English-like text. (The "a" may be doubled, C, to provide even more restrictions, preventing case-insensitive matching of ASCII with non-ASCII characters; otherwise a Unicode "Kelvin Sign" would caselessly match a "k" or "K".) The C<\d\s\w\D\S\W> abbreviations can be used both inside and outside of character classes. Here are some in use: /\d\d:\d\d:\d\d/; # matches a hh:mm:ss time format /[\d\s]/; # matches any digit or whitespace character /\w\W\w/; # matches a word char, followed by a # non-word char, followed by a word char /..rt/; # matches any two chars, followed by 'rt' /end\./; # matches 'end.' /end[.]/; # same thing, matches 'end.' Because a period is a metacharacter, it needs to be escaped to match as an ordinary period. Because, for example, C<\d> and C<\w> are sets of characters, it is incorrect to think of C<[^\d\w]> as C<[\D\W]>; in fact C<[^\d\w]> is the same as C<[^\w]>, which is the same as C<[\W]>. Think DeMorgan's laws. An anchor useful in basic regexps is the I C<\b>. This matches a boundary between a word character and a non-word character C<\w\W> or C<\W\w>: $x = "Housecat catenates house and cat"; $x =~ /cat/; # matches cat in 'housecat' $x =~ /\bcat/; # matches cat in 'catenates' $x =~ /cat\b/; # matches cat in 'housecat' $x =~ /\bcat\b/; # matches 'cat' at end of string Note in the last example, the end of the string is considered a word boundary. You might wonder why C<'.'> matches everything but C<"\n"> - why not every character? The reason is that often one is matching against lines and would like to ignore the newline characters. For instance, while the string C<"\n"> represents one line, we would like to think of it as empty. Then "" =~ /^$/; # matches "\n" =~ /^$/; # matches, $ anchors before "\n" "" =~ /./; # doesn't match; it needs a char "" =~ /^.$/; # doesn't match; it needs a char "\n" =~ /^.$/; # doesn't match; it needs a char other than "\n" "a" =~ /^.$/; # matches "a\n" =~ /^.$/; # matches, $ anchors before "\n" This behavior is convenient, because we usually want to ignore newlines when we count and match characters in a line. Sometimes, however, we want to keep track of newlines. We might even want C<^> and C<$> to anchor at the beginning and end of lines within the string, rather than just the beginning and end of the string. Perl allows us to choose between ignoring and paying attention to newlines by using the C and C modifiers. C and C stand for single line and multi-line and they determine whether a string is to be treated as one continuous string, or as a set of lines. The two modifiers affect two aspects of how the regexp is interpreted: 1) how the C<'.'> character class is defined, and 2) where the anchors C<^> and C<$> are able to match. Here are the four possible combinations: =over 4 =item * no modifiers (//): Default behavior. C<'.'> matches any character except C<"\n">. C<^> matches only at the beginning of the string and C<$> matches only at the end or before a newline at the end. =item * s modifier (//s): Treat string as a single long line. C<'.'> matches any character, even C<"\n">. C<^> matches only at the beginning of the string and C<$> matches only at the end or before a newline at the end. =item * m modifier (//m): Treat string as a set of multiple lines. C<'.'> matches any character except C<"\n">. C<^> and C<$> are able to match at the start or end of I line within the string. =item * both s and m modifiers (//sm): Treat string as a single long line, but detect multiple lines. C<'.'> matches any character, even C<"\n">. C<^> and C<$>, however, are able to match at the start or end of I line within the string. =back Here are examples of C and C in action: $x = "There once was a girl\nWho programmed in Perl\n"; $x =~ /^Who/; # doesn't match, "Who" not at start of string $x =~ /^Who/s; # doesn't match, "Who" not at start of string $x =~ /^Who/m; # matches, "Who" at start of second line $x =~ /^Who/sm; # matches, "Who" at start of second line $x =~ /girl.Who/; # doesn't match, "." doesn't match "\n" $x =~ /girl.Who/s; # matches, "." matches "\n" $x =~ /girl.Who/m; # doesn't match, "." doesn't match "\n" $x =~ /girl.Who/sm; # matches, "." matches "\n" Most of the time, the default behavior is what is wanted, but C and C are occasionally very useful. If C is being used, the start of the string can still be matched with C<\A> and the end of the string can still be matched with the anchors C<\Z> (matches both the end and the newline before, like C<$>), and C<\z> (matches only the end): $x =~ /^Who/m; # matches, "Who" at start of second line $x =~ /\AWho/m; # doesn't match, "Who" is not at start of string $x =~ /girl$/m; # matches, "girl" at end of first line $x =~ /girl\Z/m; # doesn't match, "girl" is not at end of string $x =~ /Perl\Z/m; # matches, "Perl" is at newline before end $x =~ /Perl\z/m; # doesn't match, "Perl" is not at end of string We now know how to create choices among classes of characters in a regexp. What about choices among words or character strings? Such choices are described in the next section. =head2 Matching this or that Sometimes we would like our regexp to be able to match different possible words or character strings. This is accomplished by using the I metacharacter C<|>. To match C or C, we form the regexp C. As before, Perl will try to match the regexp at the earliest possible point in the string. At each character position, Perl will first try to match the first alternative, C. If C doesn't match, Perl will then try the next alternative, C. If C doesn't match either, then the match fails and Perl moves to the next position in the string. Some examples: "cats and dogs" =~ /cat|dog|bird/; # matches "cat" "cats and dogs" =~ /dog|cat|bird/; # matches "cat" Even though C is the first alternative in the second regexp, C is able to match earlier in the string. "cats" =~ /c|ca|cat|cats/; # matches "c" "cats" =~ /cats|cat|ca|c/; # matches "cats" Here, all the alternatives match at the first string position, so the first alternative is the one that matches. If some of the alternatives are truncations of the others, put the longest ones first to give them a chance to match. "cab" =~ /a|b|c/ # matches "c" # /a|b|c/ == /[abc]/ The last example points out that character classes are like alternations of characters. At a given character position, the first alternative that allows the regexp match to succeed will be the one that matches. =head2 Grouping things and hierarchical matching Alternation allows a regexp to choose among alternatives, but by itself it is unsatisfying. The reason is that each alternative is a whole regexp, but sometime we want alternatives for just part of a regexp. For instance, suppose we want to search for housecats or housekeepers. The regexp C fits the bill, but is inefficient because we had to type C twice. It would be nice to have parts of the regexp be constant, like C, and some parts have alternatives, like C. The I metacharacters C<()> solve this problem. Grouping allows parts of a regexp to be treated as a single unit. Parts of a regexp are grouped by enclosing them in parentheses. Thus we could solve the C by forming the regexp as C. The regexp C means match C followed by either C or C. Some more examples are /(a|b)b/; # matches 'ab' or 'bb' /(ac|b)b/; # matches 'acb' or 'bb' /(^a|b)c/; # matches 'ac' at start of string or 'bc' anywhere /(a|[bc])d/; # matches 'ad', 'bd', or 'cd' /house(cat|)/; # matches either 'housecat' or 'house' /house(cat(s|)|)/; # matches either 'housecats' or 'housecat' or # 'house'. Note groups can be nested. /(19|20|)\d\d/; # match years 19xx, 20xx, or the Y2K problem, xx "20" =~ /(19|20|)\d\d/; # matches the null alternative '()\d\d', # because '20\d\d' can't match Alternations behave the same way in groups as out of them: at a given string position, the leftmost alternative that allows the regexp to match is taken. So in the last example at the first string position, C<"20"> matches the second alternative, but there is nothing left over to match the next two digits C<\d\d>. So Perl moves on to the next alternative, which is the null alternative and that works, since C<"20"> is two digits. The process of trying one alternative, seeing if it matches, and moving on to the next alternative, while going back in the string from where the previous alternative was tried, if it doesn't, is called I. The term 'backtracking' comes from the idea that matching a regexp is like a walk in the woods. Successfully matching a regexp is like arriving at a destination. There are many possible trailheads, one for each string position, and each one is tried in order, left to right. From each trailhead there may be many paths, some of which get you there, and some which are dead ends. When you walk along a trail and hit a dead end, you have to backtrack along the trail to an earlier point to try another trail. If you hit your destination, you stop immediately and forget about trying all the other trails. You are persistent, and only if you have tried all the trails from all the trailheads and not arrived at your destination, do you declare failure. To be concrete, here is a step-by-step analysis of what Perl does when it tries to match the regexp "abcde" =~ /(abd|abc)(df|d|de)/; =over 4 =item 0 Start with the first letter in the string 'a'. =item 1 Try the first alternative in the first group 'abd'. =item 2 Match 'a' followed by 'b'. So far so good. =item 3 'd' in the regexp doesn't match 'c' in the string - a dead end. So backtrack two characters and pick the second alternative in the first group 'abc'. =item 4 Match 'a' followed by 'b' followed by 'c'. We are on a roll and have satisfied the first group. Set $1 to 'abc'. =item 5 Move on to the second group and pick the first alternative 'df'. =item 6 Match the 'd'. =item 7 'f' in the regexp doesn't match 'e' in the string, so a dead end. Backtrack one character and pick the second alternative in the second group 'd'. =item 8 'd' matches. The second grouping is satisfied, so set $2 to 'd'. =item 9 We are at the end of the regexp, so we are done! We have matched 'abcd' out of the string "abcde". =back There are a couple of things to note about this analysis. First, the third alternative in the second group 'de' also allows a match, but we stopped before we got to it - at a given character position, leftmost wins. Second, we were able to get a match at the first character position of the string 'a'. If there were no matches at the first position, Perl would move to the second character position 'b' and attempt the match all over again. Only when all possible paths at all possible character positions have been exhausted does Perl give up and declare S> to be false. Even with all this work, regexp matching happens remarkably fast. To speed things up, Perl compiles the regexp into a compact sequence of opcodes that can often fit inside a processor cache. When the code is executed, these opcodes can then run at full throttle and search very quickly. =head2 Extracting matches The grouping metacharacters C<()> also serve another completely different function: they allow the extraction of the parts of a string that matched. This is very useful to find out what matched and for text processing in general. For each grouping, the part that matched inside goes into the special variables C<$1>, C<$2>, etc. They can be used just as ordinary variables: # extract hours, minutes, seconds if ($time =~ /(\d\d):(\d\d):(\d\d)/) { # match hh:mm:ss format $hours = $1; $minutes = $2; $seconds = $3; } Now, we know that in scalar context, S> returns a true or false value. In list context, however, it returns the list of matched values C<($1,$2,$3)>. So we could write the code more compactly as # extract hours, minutes, seconds ($hours, $minutes, $second) = ($time =~ /(\d\d):(\d\d):(\d\d)/); If the groupings in a regexp are nested, C<$1> gets the group with the leftmost opening parenthesis, C<$2> the next opening parenthesis, etc. Here is a regexp with nested groups: /(ab(cd|ef)((gi)|j))/; 1 2 34 If this regexp matches, C<$1> contains a string starting with C<'ab'>, C<$2> is either set to C<'cd'> or C<'ef'>, C<$3> equals either C<'gi'> or C<'j'>, and C<$4> is either set to C<'gi'>, just like C<$3>, or it remains undefined. For convenience, Perl sets C<$+> to the string held by the highest numbered C<$1>, C<$2>,... that got assigned (and, somewhat related, C<$^N> to the value of the C<$1>, C<$2>,... most-recently assigned; i.e. the C<$1>, C<$2>,... associated with the rightmost closing parenthesis used in the match). =head2 Backreferences Closely associated with the matching variables C<$1>, C<$2>, ... are the I C<\g1>, C<\g2>,... Backreferences are simply matching variables that can be used I a regexp. This is a really nice feature; what matches later in a regexp is made to depend on what matched earlier in the regexp. Suppose we wanted to look for doubled words in a text, like 'the the'. The following regexp finds all 3-letter doubles with a space in between: /\b(\w\w\w)\s\g1\b/; The grouping assigns a value to \g1, so that the same 3-letter sequence is used for both parts. A similar task is to find words consisting of two identical parts: % simple_grep '^(\w\w\w\w|\w\w\w|\w\w|\w)\g1$' /usr/dict/words beriberi booboo coco mama murmur papa The regexp has a single grouping which considers 4-letter combinations, then 3-letter combinations, etc., and uses C<\g1> to look for a repeat. Although C<$1> and C<\g1> represent the same thing, care should be taken to use matched variables C<$1>, C<$2>,... only I a regexp and backreferences C<\g1>, C<\g2>,... only I a regexp; not doing so may lead to surprising and unsatisfactory results. =head2 Relative backreferences Counting the opening parentheses to get the correct number for a backreference is error-prone as soon as there is more than one capturing group. A more convenient technique became available with Perl 5.10: relative backreferences. To refer to the immediately preceding capture group one now may write C<\g{-1}>, the next but last is available via C<\g{-2}>, and so on. Another good reason in addition to readability and maintainability for using relative backreferences is illustrated by the following example, where a simple pattern for matching peculiar strings is used: $a99a = '([a-z])(\d)\g2\g1'; # matches a11a, g22g, x33x, etc. Now that we have this pattern stored as a handy string, we might feel tempted to use it as a part of some other pattern: $line = "code=e99e"; if ($line =~ /^(\w+)=$a99a$/){ # unexpected behavior! print "$1 is valid\n"; } else { print "bad line: '$line'\n"; } But this doesn't match, at least not the way one might expect. Only after inserting the interpolated C<$a99a> and looking at the resulting full text of the regexp is it obvious that the backreferences have backfired. The subexpression C<(\w+)> has snatched number 1 and demoted the groups in C<$a99a> by one rank. This can be avoided by using relative backreferences: $a99a = '([a-z])(\d)\g{-1}\g{-2}'; # safe for being interpolated =head2 Named backreferences Perl 5.10 also introduced named capture groups and named backreferences. To attach a name to a capturing group, you write either C<< (?...) >> or C<< (?'name'...) >>. The backreference may then be written as C<\g{name}>. It is permissible to attach the same name to more than one group, but then only the leftmost one of the eponymous set can be referenced. Outside of the pattern a named capture group is accessible through the C<%+> hash. Assuming that we have to match calendar dates which may be given in one of the three formats yyyy-mm-dd, mm/dd/yyyy or dd.mm.yyyy, we can write three suitable patterns where we use 'd', 'm' and 'y' respectively as the names of the groups capturing the pertaining components of a date. The matching operation combines the three patterns as alternatives: $fmt1 = '(?\d\d\d\d)-(?\d\d)-(?\d\d)'; $fmt2 = '(?\d\d)/(?\d\d)/(?\d\d\d\d)'; $fmt3 = '(?\d\d)\.(?\d\d)\.(?\d\d\d\d)'; for my $d qw( 2006-10-21 15.01.2007 10/31/2005 ){ if ( $d =~ m{$fmt1|$fmt2|$fmt3} ){ print "day=$+{d} month=$+{m} year=$+{y}\n"; } } If any of the alternatives matches, the hash C<%+> is bound to contain the three key-value pairs. =head2 Alternative capture group numbering Yet another capturing group numbering technique (also as from Perl 5.10) deals with the problem of referring to groups within a set of alternatives. Consider a pattern for matching a time of the day, civil or military style: if ( $time =~ /(\d\d|\d):(\d\d)|(\d\d)(\d\d)/ ){ # process hour and minute } Processing the results requires an additional if statement to determine whether C<$1> and C<$2> or C<$3> and C<$4> contain the goodies. It would be easier if we could use group numbers 1 and 2 in second alternative as well, and this is exactly what the parenthesized construct C<(?|...)>, set around an alternative achieves. Here is an extended version of the previous pattern: if ( $time =~ /(?|(\d\d|\d):(\d\d)|(\d\d)(\d\d))\s+([A-Z][A-Z][A-Z])/ ){ print "hour=$1 minute=$2 zone=$3\n"; } Within the alternative numbering group, group numbers start at the same position for each alternative. After the group, numbering continues with one higher than the maximum reached across all the alternatives. =head2 Position information In addition to what was matched, Perl (since 5.6.0) also provides the positions of what was matched as contents of the C<@-> and C<@+> arrays. C<$-[0]> is the position of the start of the entire match and C<$+[0]> is the position of the end. Similarly, C<$-[n]> is the position of the start of the C<$n> match and C<$+[n]> is the position of the end. If C<$n> is undefined, so are C<$-[n]> and C<$+[n]>. Then this code $x = "Mmm...donut, thought Homer"; $x =~ /^(Mmm|Yech)\.\.\.(donut|peas)/; # matches foreach $expr (1..$#-) { print "Match $expr: '${$expr}' at position ($-[$expr],$+[$expr])\n"; } prints Match 1: 'Mmm' at position (0,3) Match 2: 'donut' at position (6,11) Even if there are no groupings in a regexp, it is still possible to find out what exactly matched in a string. If you use them, Perl will set C<$`> to the part of the string before the match, will set C<$&> to the part of the string that matched, and will set C<$'> to the part of the string after the match. An example: $x = "the cat caught the mouse"; $x =~ /cat/; # $` = 'the ', $& = 'cat', $' = ' caught the mouse' $x =~ /the/; # $` = '', $& = 'the', $' = ' cat caught the mouse' In the second match, C<$`> equals C<''> because the regexp matched at the first character position in the string and stopped; it never saw the second 'the'. It is important to note that using C<$`> and C<$'> slows down regexp matching quite a bit, while C<$&> slows it down to a lesser extent, because if they are used in one regexp in a program, they are generated for I regexps in the program. So if raw performance is a goal of your application, they should be avoided. If you need to extract the corresponding substrings, use C<@-> and C<@+> instead: $` is the same as substr( $x, 0, $-[0] ) $& is the same as substr( $x, $-[0], $+[0]-$-[0] ) $' is the same as substr( $x, $+[0] ) As of Perl 5.10, the C<${^PREMATCH}>, C<${^MATCH}> and C<${^POSTMATCH}> variables may be used. These are only set if the C
expressions turn that around by allowing arbitrary Perl code to be a part of a regexp. A code evaluation expression is denoted C<(?{code})>, with I a string of Perl statements. Be warned that this feature is considered experimental, and may be changed without notice. Code expressions are zero-width assertions, and the value they return depends on their environment. There are two possibilities: either the code expression is used as a conditional in a conditional expression C<(?(condition)...)>, or it is not. If the code expression is a conditional, the code is evaluated and the result (i.e., the result of the last statement) is used to determine truth or falsehood. If the code expression is not used as a conditional, the assertion always evaluates true and the result is put into the special variable C<$^R>. The variable C<$^R> can then be used in code expressions later in the regexp. Here are some silly examples: $x = "abcdef"; $x =~ /abc(?{print "Hi Mom!";})def/; # matches, # prints 'Hi Mom!' $x =~ /aaa(?{print "Hi Mom!";})def/; # doesn't match, # no 'Hi Mom!' Pay careful attention to the next example: $x =~ /abc(?{print "Hi Mom!";})ddd/; # doesn't match, # no 'Hi Mom!' # but why not? At first glance, you'd think that it shouldn't print, because obviously the C isn't going to match the target string. But look at this example: $x =~ /abc(?{print "Hi Mom!";})[dD]dd/; # doesn't match, # but _does_ print Hmm. What happened here? If you've been following along, you know that the above pattern should be effectively (almost) the same as the last one; enclosing the C in a character class isn't going to change what it matches. So why does the first not print while the second one does? The answer lies in the optimizations the regex engine makes. In the first case, all the engine sees are plain old characters (aside from the C construct). It's smart enough to realize that the string 'ddd' doesn't occur in our target string before actually running the pattern through. But in the second case, we've tricked it into thinking that our pattern is more complicated. It takes a look, sees our character class, and decides that it will have to actually run the pattern to determine whether or not it matches, and in the process of running it hits the print statement before it discovers that we don't have a match. To take a closer look at how the engine does optimizations, see the section L<"Pragmas and debugging"> below. More fun with C: $x =~ /(?{print "Hi Mom!";})/; # matches, # prints 'Hi Mom!' $x =~ /(?{$c = 1;})(?{print "$c";})/; # matches, # prints '1' $x =~ /(?{$c = 1;})(?{print "$^R";})/; # matches, # prints '1' The bit of magic mentioned in the section title occurs when the regexp backtracks in the process of searching for a match. If the regexp backtracks over a code expression and if the variables used within are localized using C, the changes in the variables produced by the code expression are undone! Thus, if we wanted to count how many times a character got matched inside a group, we could use, e.g., $x = "aaaa"; $count = 0; # initialize 'a' count $c = "bob"; # test if $c gets clobbered $x =~ /(?{local $c = 0;}) # initialize count ( a # match 'a' (?{local $c = $c + 1;}) # increment count )* # do this any number of times, aa # but match 'aa' at the end (?{$count = $c;}) # copy local $c var into $count /x; print "'a' count is $count, \$c variable is '$c'\n"; This prints 'a' count is 2, $c variable is 'bob' If we replace the S> with S>, the variable changes are I undone during backtracking, and we get 'a' count is 4, $c variable is 'bob' Note that only localized variable changes are undone. Other side effects of code expression execution are permanent. Thus $x = "aaaa"; $x =~ /(a(?{print "Yow\n";}))*aa/; produces Yow Yow Yow Yow The result C<$^R> is automatically localized, so that it will behave properly in the presence of backtracking. This example uses a code expression in a conditional to match a definite article, either 'the' in English or 'der|die|das' in German: $lang = 'DE'; # use German ... $text = "das"; print "matched\n" if $text =~ /(?(?{ $lang eq 'EN'; # is the language English? }) the | # if so, then match 'the' (der|die|das) # else, match 'der|die|das' ) /xi; Note that the syntax here is C<(?(?{...})yes-regexp|no-regexp)>, not C<(?((?{...}))yes-regexp|no-regexp)>. In other words, in the case of a code expression, we don't need the extra parentheses around the conditional. If you try to use code expressions with interpolating variables, Perl may surprise you: $bar = 5; $pat = '(?{ 1 })'; /foo(?{ $bar })bar/; # compiles ok, $bar not interpolated /foo(?{ 1 })$bar/; # compile error! /foo${pat}bar/; # compile error! $pat = qr/(?{ $foo = 1 })/; # precompile code regexp /foo${pat}bar/; # compiles ok If a regexp has (1) code expressions and interpolating variables, or (2) a variable that interpolates a code expression, Perl treats the regexp as an error. If the code expression is precompiled into a variable, however, interpolating is ok. The question is, why is this an error? The reason is that variable interpolation and code expressions together pose a security risk. The combination is dangerous because many programmers who write search engines often take user input and plug it directly into a regexp: $regexp = <>; # read user-supplied regexp $chomp $regexp; # get rid of possible newline $text =~ /$regexp/; # search $text for the $regexp If the C<$regexp> variable contains a code expression, the user could then execute arbitrary Perl code. For instance, some joker could search for S> to erase your files. In this sense, the combination of interpolation and code expressions I your regexp. So by default, using both interpolation and code expressions in the same regexp is not allowed. If you're not concerned about malicious users, it is possible to bypass this security check by invoking S>: use re 'eval'; # throw caution out the door $bar = 5; $pat = '(?{ 1 })'; /foo(?{ 1 })$bar/; # compiles ok /foo${pat}bar/; # compiles ok Another form of code expression is the I. The pattern code expression is like a regular code expression, except that the result of the code evaluation is treated as a regular expression and matched immediately. A simple example is $length = 5; $char = 'a'; $x = 'aaaaabb'; $x =~ /(??{$char x $length})/x; # matches, there are 5 of 'a' This final example contains both ordinary and pattern code expressions. It detects whether a binary string C<1101010010001...> has a Fibonacci spacing 0,1,1,2,3,5,... of the C<1>'s: $x = "1101010010001000001"; $z0 = ''; $z1 = '0'; # initial conditions print "It is a Fibonacci sequence\n" if $x =~ /^1 # match an initial '1' (?: ((??{ $z0 })) # match some '0' 1 # and then a '1' (?{ $z0 = $z1; $z1 .= $^N; }) )+ # repeat as needed $ # that is all there is /x; printf "Largest sequence matched was %d\n", length($z1)-length($z0); Remember that C<$^N> is set to whatever was matched by the last completed capture group. This prints It is a Fibonacci sequence Largest sequence matched was 5 Ha! Try that with your garden variety regexp package... Note that the variables C<$z0> and C<$z1> are not substituted when the regexp is compiled, as happens for ordinary variables outside a code expression. Rather, the code expressions are evaluated when Perl encounters them during the search for a match. The regexp without the C modifier is /^1(?:((??{ $z0 }))1(?{ $z0 = $z1; $z1 .= $^N; }))+$/ which shows that spaces are still possible in the code parts. Nevertheless, when working with code and conditional expressions, the extended form of regexps is almost necessary in creating and debugging regexps. =head2 Backtracking control verbs Perl 5.10 introduced a number of control verbs intended to provide detailed control over the backtracking process, by directly influencing the regexp engine and by providing monitoring techniques. As all the features in this group are experimental and subject to change or removal in a future version of Perl, the interested reader is referred to L for a detailed description. Below is just one example, illustrating the control verb C<(*FAIL)>, which may be abbreviated as C<(*F)>. If this is inserted in a regexp it will cause it to fail, just as it would at some mismatch between the pattern and the string. Processing of the regexp continues as it would after any "normal" failure, so that, for instance, the next position in the string or another alternative will be tried. As failing to match doesn't preserve capture groups or produce results, it may be necessary to use this in combination with embedded code. %count = (); "supercalifragilisticexpialidocious" =~ /([aeiou])(?{ $count{$1}++; })(*FAIL)/i; printf "%3d '%s'\n", $count{$_}, $_ for (sort keys %count); The pattern begins with a class matching a subset of letters. Whenever this matches, a statement like C<$count{'a'}++;> is executed, incrementing the letter's counter. Then C<(*FAIL)> does what it says, and the regexp engine proceeds according to the book: as long as the end of the string hasn't been reached, the position is advanced before looking for another vowel. Thus, match or no match makes no difference, and the regexp engine proceeds until the entire string has been inspected. (It's remarkable that an alternative solution using something like $count{lc($_)}++ for split('', "supercalifragilisticexpialidocious"); printf "%3d '%s'\n", $count2{$_}, $_ for ( qw{ a e i o u } ); is considerably slower.) =head2 Pragmas and debugging Speaking of debugging, there are several pragmas available to control and debug regexps in Perl. We have already encountered one pragma in the previous section, S>, that allows variable interpolation and code expressions to coexist in a regexp. The other pragmas are use re 'taint'; $tainted = <>; @parts = ($tainted =~ /(\w+)\s+(\w+)/; # @parts is now tainted The C pragma causes any substrings from a match with a tainted variable to be tainted as well. This is not normally the case, as regexps are often used to extract the safe bits from a tainted variable. Use C when you are not extracting safe bits, but are performing some other processing. Both C and C pragmas are lexically scoped, which means they are in effect only until the end of the block enclosing the pragmas. use re '/m'; # or any other flags $multiline_string =~ /^foo/; # /m is implied The C pragma (introduced in Perl 5.14) turns on the given regular expression flags until the end of the lexical scope. See Lflags' mode"> for more detail. use re 'debug'; /^(.*)$/s; # output debugging info use re 'debugcolor'; /^(.*)$/s; # output debugging info in living color The global C and C pragmas allow one to get detailed debugging info about regexp compilation and execution. C is the same as debug, except the debugging information is displayed in color on terminals that can display termcap color sequences. Here is example output: % perl -e 'use re "debug"; "abc" =~ /a*b+c/;' Compiling REx 'a*b+c' size 9 first at 1 1: STAR(4) 2: EXACT (0) 4: PLUS(7) 5: EXACT (0) 7: EXACT (9) 9: END(0) floating 'bc' at 0..2147483647 (checking floating) minlen 2 Guessing start of match, REx 'a*b+c' against 'abc'... Found floating substr 'bc' at offset 1... Guessed: match at offset 0 Matching REx 'a*b+c' against 'abc' Setting an EVAL scope, savestack=3 0 <> | 1: STAR EXACT can match 1 times out of 32767... Setting an EVAL scope, savestack=3 1 | 4: PLUS EXACT can match 1 times out of 32767... Setting an EVAL scope, savestack=3 2 | 7: EXACT 3 <> | 9: END Match successful! Freeing REx: 'a*b+c' If you have gotten this far into the tutorial, you can probably guess what the different parts of the debugging output tell you. The first part Compiling REx 'a*b+c' size 9 first at 1 1: STAR(4) 2: EXACT (0) 4: PLUS(7) 5: EXACT (0) 7: EXACT
a string of Perl statements. Be warned that this feature is considered experimental, and may be changed without notice. Code expressions are zero-width assertions, and the value they return depends on their environment. There are two possibilities: either the code expression is used as a conditional in a conditional expression C<(?(condition)...)>, or it is not. If the code expression is a conditional, the code is evaluated and the result (i.e., the result of the last statement) is used to determine truth or falsehood. If the code expression is not used as a conditional, the assertion always evaluates true and the result is put into the special variable C<$^R>. The variable C<$^R> can then be used in code expressions later in the regexp. Here are some silly examples: $x = "abcdef"; $x =~ /abc(?{print "Hi Mom!";})def/; # matches, # prints 'Hi Mom!' $x =~ /aaa(?{print "Hi Mom!";})def/; # doesn't match, # no 'Hi Mom!' Pay careful attention to the next example: $x =~ /abc(?{print "Hi Mom!";})ddd/; # doesn't match, # no 'Hi Mom!' # but why not? At first glance, you'd think that it shouldn't print, because obviously the C isn't going to match the target string. But look at this example: $x =~ /abc(?{print "Hi Mom!";})[dD]dd/; # doesn't match, # but _does_ print Hmm. What happened here? If you've been following along, you know that the above pattern should be effectively (almost) the same as the last one; enclosing the C in a character class isn't going to change what it matches. So why does the first not print while the second one does? The answer lies in the optimizations the regex engine makes. In the first case, all the engine sees are plain old characters (aside from the C construct). It's smart enough to realize that the string 'ddd' doesn't occur in our target string before actually running the pattern through. But in the second case, we've tricked it into thinking that our pattern is more complicated. It takes a look, sees our character class, and decides that it will have to actually run the pattern to determine whether or not it matches, and in the process of running it hits the print statement before it discovers that we don't have a match. To take a closer look at how the engine does optimizations, see the section L<"Pragmas and debugging"> below. More fun with C: $x =~ /(?{print "Hi Mom!";})/; # matches, # prints 'Hi Mom!' $x =~ /(?{$c = 1;})(?{print "$c";})/; # matches, # prints '1' $x =~ /(?{$c = 1;})(?{print "$^R";})/; # matches, # prints '1' The bit of magic mentioned in the section title occurs when the regexp backtracks in the process of searching for a match. If the regexp backtracks over a code expression and if the variables used within are localized using C, the changes in the variables produced by the code expression are undone! Thus, if we wanted to count how many times a character got matched inside a group, we could use, e.g., $x = "aaaa"; $count = 0; # initialize 'a' count $c = "bob"; # test if $c gets clobbered $x =~ /(?{local $c = 0;}) # initialize count ( a # match 'a' (?{local $c = $c + 1;}) # increment count )* # do this any number of times, aa # but match 'aa' at the end (?{$count = $c;}) # copy local $c var into $count /x; print "'a' count is $count, \$c variable is '$c'\n"; This prints 'a' count is 2, $c variable is 'bob' If we replace the S> with S>, the variable changes are I undone during backtracking, and we get 'a' count is 4, $c variable is 'bob' Note that only localized variable changes are undone. Other side effects of code expression execution are permanent. Thus $x = "aaaa"; $x =~ /(a(?{print "Yow\n";}))*aa/; produces Yow Yow Yow Yow The result C<$^R> is automatically localized, so that it will behave properly in the presence of backtracking. This example uses a code expression in a conditional to match a definite article, either 'the' in English or 'der|die|das' in German: $lang = 'DE'; # use German ... $text = "das"; print "matched\n" if $text =~ /(?(?{ $lang eq 'EN'; # is the language English? }) the | # if so, then match 'the' (der|die|das) # else, match 'der|die|das' ) /xi; Note that the syntax here is C<(?(?{...})yes-regexp|no-regexp)>, not C<(?((?{...}))yes-regexp|no-regexp)>. In other words, in the case of a code expression, we don't need the extra parentheses around the conditional. If you try to use code expressions with interpolating variables, Perl may surprise you: $bar = 5; $pat = '(?{ 1 })'; /foo(?{ $bar })bar/; # compiles ok, $bar not interpolated /foo(?{ 1 })$bar/; # compile error! /foo${pat}bar/; # compile error! $pat = qr/(?{ $foo = 1 })/; # precompile code regexp /foo${pat}bar/; # compiles ok If a regexp has (1) code expressions and interpolating variables, or (2) a variable that interpolates a code expression, Perl treats the regexp as an error. If the code expression is precompiled into a variable, however, interpolating is ok. The question is, why is this an error? The reason is that variable interpolation and code expressions together pose a security risk. The combination is dangerous because many programmers who write search engines often take user input and plug it directly into a regexp: $regexp = <>; # read user-supplied regexp $chomp $regexp; # get rid of possible newline $text =~ /$regexp/; # search $text for the $regexp If the C<$regexp> variable contains a code expression, the user could then execute arbitrary Perl code. For instance, some joker could search for S> to erase your files. In this sense, the combination of interpolation and code expressions I your regexp. So by default, using both interpolation and code expressions in the same regexp is not allowed. If you're not concerned about malicious users, it is possible to bypass this security check by invoking S>: use re 'eval'; # throw caution out the door $bar = 5; $pat = '(?{ 1 })'; /foo(?{ 1 })$bar/; # compiles ok /foo${pat}bar/; # compiles ok Another form of code expression is the I. The pattern code expression is like a regular code expression, except that the result of the code evaluation is treated as a regular expression and matched immediately. A simple example is $length = 5; $char = 'a'; $x = 'aaaaabb'; $x =~ /(??{$char x $length})/x; # matches, there are 5 of 'a' This final example contains both ordinary and pattern code expressions. It detects whether a binary string C<1101010010001...> has a Fibonacci spacing 0,1,1,2,3,5,... of the C<1>'s: $x = "1101010010001000001"; $z0 = ''; $z1 = '0'; # initial conditions print "It is a Fibonacci sequence\n" if $x =~ /^1 # match an initial '1' (?: ((??{ $z0 })) # match some '0' 1 # and then a '1' (?{ $z0 = $z1; $z1 .= $^N; }) )+ # repeat as needed $ # that is all there is /x; printf "Largest sequence matched was %d\n", length($z1)-length($z0); Remember that C<$^N> is set to whatever was matched by the last completed capture group. This prints It is a Fibonacci sequence Largest sequence matched was 5 Ha! Try that with your garden variety regexp package... Note that the variables C<$z0> and C<$z1> are not substituted when the regexp is compiled, as happens for ordinary variables outside a code expression. Rather, the code expressions are evaluated when Perl encounters them during the search for a match. The regexp without the C modifier is /^1(?:((??{ $z0 }))1(?{ $z0 = $z1; $z1 .= $^N; }))+$/ which shows that spaces are still possible in the code parts. Nevertheless, when working with code and conditional expressions, the extended form of regexps is almost necessary in creating and debugging regexps. =head2 Backtracking control verbs Perl 5.10 introduced a number of control verbs intended to provide detailed control over the backtracking process, by directly influencing the regexp engine and by providing monitoring techniques. As all the features in this group are experimental and subject to change or removal in a future version of Perl, the interested reader is referred to L for a detailed description. Below is just one example, illustrating the control verb C<(*FAIL)>, which may be abbreviated as C<(*F)>. If this is inserted in a regexp it will cause it to fail, just as it would at some mismatch between the pattern and the string. Processing of the regexp continues as it would after any "normal" failure, so that, for instance, the next position in the string or another alternative will be tried. As failing to match doesn't preserve capture groups or produce results, it may be necessary to use this in combination with embedded code. %count = (); "supercalifragilisticexpialidocious" =~ /([aeiou])(?{ $count{$1}++; })(*FAIL)/i; printf "%3d '%s'\n", $count{$_}, $_ for (sort keys %count); The pattern begins with a class matching a subset of letters. Whenever this matches, a statement like C<$count{'a'}++;> is executed, incrementing the letter's counter. Then C<(*FAIL)> does what it says, and the regexp engine proceeds according to the book: as long as the end of the string hasn't been reached, the position is advanced before looking for another vowel. Thus, match or no match makes no difference, and the regexp engine proceeds until the entire string has been inspected. (It's remarkable that an alternative solution using something like $count{lc($_)}++ for split('', "supercalifragilisticexpialidocious"); printf "%3d '%s'\n", $count2{$_}, $_ for ( qw{ a e i o u } ); is considerably slower.) =head2 Pragmas and debugging Speaking of debugging, there are several pragmas available to control and debug regexps in Perl. We have already encountered one pragma in the previous section, S>, that allows variable interpolation and code expressions to coexist in a regexp. The other pragmas are use re 'taint'; $tainted = <>; @parts = ($tainted =~ /(\w+)\s+(\w+)/; # @parts is now tainted The C pragma causes any substrings from a match with a tainted variable to be tainted as well. This is not normally the case, as regexps are often used to extract the safe bits from a tainted variable. Use C when you are not extracting safe bits, but are performing some other processing. Both C and C pragmas are lexically scoped, which means they are in effect only until the end of the block enclosing the pragmas. use re '/m'; # or any other flags $multiline_string =~ /^foo/; # /m is implied The C pragma (introduced in Perl 5.14) turns on the given regular expression flags until the end of the lexical scope. See Lflags' mode"> for more detail. use re 'debug'; /^(.*)$/s; # output debugging info use re 'debugcolor'; /^(.*)$/s; # output debugging info in living color The global C and C pragmas allow one to get detailed debugging info about regexp compilation and execution. C is the same as debug, except the debugging information is displayed in color on terminals that can display termcap color sequences. Here is example output: % perl -e 'use re "debug"; "abc" =~ /a*b+c/;' Compiling REx 'a*b+c' size 9 first at 1 1: STAR(4) 2: EXACT (0) 4: PLUS(7) 5: EXACT (0) 7: EXACT (9) 9: END(0) floating 'bc' at 0..2147483647 (checking floating) minlen 2 Guessing start of match, REx 'a*b+c' against 'abc'... Found floating substr 'bc' at offset 1... Guessed: match at offset 0 Matching REx 'a*b+c' against 'abc' Setting an EVAL scope, savestack=3 0 <> | 1: STAR EXACT can match 1 times out of 32767... Setting an EVAL scope, savestack=3 1 | 4: PLUS EXACT can match 1 times out of 32767... Setting an EVAL scope, savestack=3 2 | 7: EXACT 3 <> | 9: END Match successful! Freeing REx: 'a*b+c' If you have gotten this far into the tutorial, you can probably guess what the different parts of the debugging output tell you. The first part Compiling REx 'a*b+c' size 9 first at 1 1: STAR(4) 2: EXACT (0) 4: PLUS(7) 5: EXACT (0) 7: EXACT