## CL-PPCRE - portable Perl-compatible regular expressions for Common Lisp

> ### Abstract

> CL-PPCRE is a portable regular expression library for Common Lisp which has the following features:
> - It is compatible with Perl. (Well - as far as you can be compatible with a language defined by its ever-changing implementation. Currently, as of December 2002, CL-PPCRE is more compatible with the regex semantics of Perl 5.8.0 than, say, Perl 5.6.1 is...) It even correctly parses and applies Jeffrey Friedl's famous 6600-byte long RFC822 address pattern.
> - It is fast. If compiled with CMUCL it outperforms Perl's highly optimized regex engine (written in C) which to my knowledge is faster than most other regex engines around. If compiled with CLISP it is still comparable to CLISP's own regex implementation which is also written in C.
> - It is portable, i.e. the code aims to be strictly ANSI-compliant. If you encounter any deviations this is an error and should be reported to edi@agharta.de. CL-PPCRE has been successfully tested with the following Common Lisp implementations:
>   - CMUCL 18d+ on Gentoo Linux 1.1a
>   - SBCL 0.7.10 on Gentoo Linux 1.1a
>   - Scieneer Common Lisp 1.1.1 evaluation on Gentoo Linux 1.1a
>   - CLISP 2.30 on Gentoo Linux 1.1a
>   - Xanalys LispWorks 4.2.7 professional on Gentoo Linux 1.1a
>   - Allegro Common Lisp 6.2 trial on Gentoo Linux 1.1a
>   - CLISP 2.29 on Windows XP pro
>   - Xanalys LispWorks 4.2.7 professional on Windows XP pro
>   - OpenMCL 0.13.1 on MacOS X 10.2.2
>   - Macintosh Common Lisp 4.3 demo on MacOS 9.1
>   I'll gladly accept patches to make it work on Corman Lisp, ECL, or GCL. If you succeed in using CL-PPCRE on other platforms please let me know.
>   (Note that the tests mainly made sure that the package compiled without errors and that the test suite - which compiles about 1,500 regex strings into scanners and applies these to target strings with the `SCAN` function - yields the expected results. Other functions like `SPLIT`, `ALL-MATCHES`, `REGEX-REPLACE`, `REGEX-APROPOS`, or the `DO`-macros have only been tested on CMUCL and LispWorks which were my main development platforms.)
> - It is thread-safe. Although the code uses closures extensively, no state which dynamically changes during the scanning process is stored in the lexical environments of the closures, so it should be safe to use CL-PPCRE in a multi-threaded program. Tests with LispWorks and Scieneer Common Lisp seem to confirm this.
> - It comes with convenient features like a `SPLIT` function, a couple of `DO`-like loop constructs, and a `regex-based APROPOS feature` similar to the one found in Emacs.
> - In addition to specifying regular expressions as strings like in Perl you can also use S-expressions which obviously is more Lisp-y.
> - Is it is fully documented so I might have a chance to understand my own code in about six months... :)
> - It comes with a BSD-style license so you can basically do with it whatever you want.

### Contents

1. How to use CL-PPCRE
   1. `create-scanner` (for Perl regex strings)
   2. `create-scanner` (for parse trees)
   3. `scan`
   4. `scan-to-strings`
   5. `do-scans`
   6. `do-matches`
   7. `do-matches-as-strings`
   8. `all-matches`
   9. `all-matches-as-strings`
   10. `split`
   11. `regex-replace`
   12. `regex-replace-all`
   13. `regex-apropos`
   14. `regex-apropos-list`
2. Download and installation
3. Testing CL-PPCRE
4. Compatibility with Perl
   1. Empty strings instead of `undef` in `$1`, `$2`, etc.
   2. Strange scoping of embedded modifiers
   3. Inconsistent capturing of `$1`, `$2`, etc.
   4. Captured groups not available outside of look-aheads and look-behinds
   5. Alternations don't always work from left to right
   6. `"\r"` doesn't work with MCL
   7. What about `"\w"`?
5. Performance
   1. Benchmarking
   2. Other performance issues
6. Bugs and problems
   1. Stack overflow
   2. Strange-looking output of `regex-apropos` with CLISP
7. Remarks
8. Acknowledgements

### How to use CL-PPCRE

CL-PPCRE exports the following symbols:

[Function]
create-scanner string `&key` case-insensitive-mode multi-line-mode single-line-mode extended-mode => scanner

> Accepts a string which is a regular expression in Perl syntax and returns a closure which will scan strings for this regular expression. The mode keyboard arguments are equivalent to the embedded `"imsx"` modifiers in Perl.

> The function accepts most of the regex syntax of Perl 5 as described in `man perlre` including extended features like non-greedy repetitions, positive and negative look-ahead and look-behind assertions, "standalone" subexpressions, and conditional subpatterns. The following Perl features are (currently) not supported:

> - `(?{ code })` and `(??{ code })` because they obviously don't make sense in Lisp.
> - `\N{name}` (named characters), `\x{263a}` (wide hex characters), `\l`, `\u`, `\L`, `\U`, `\E`, and `\Q` because they're actually not part of Perl's regex syntax and (honestly) because I was too lazy. I might implement `\Q` in the future but don't hold your breath.
> - `\pP` and `\PP` (named properties), `\X` (extended Unicode), and `\C` (single character). But you can of course use all characters supported by your CL implementation.
> - Posix character classes like `[[:alpha]]`. I might add this in the future.
> - `\G` for Perl's `pos()` because we don't have it.
> Note, however, that `\t`, `\n`, `\r`, `\f`, `\a`, `\e`, `\033` (octal character codes), `\x1B` (hexadecimal character codes), `\c[` (control characters), `\w`, `\W`, `\s`, `\S`, `\d`, `\D`, `\b`, `\B`, `\A`, `\Z`, and `\z` are supported.

> The keyword arguments are just for your convenience. You can always use embedded modifiers like `"(?i-s)"` instead.

[Function]
create-scanner parse-tree `&key` case-insensitive-mode multi-line-mode single-line-mode extended-mode => scanner

> This is similar to `create-scanner` above but accepts a parse tree as its first argument. A parse tree is an S-expression conforming to the following syntax:
> - Every string and character is a parse tree and is treated literally as a part of the regular expression, i.e. parentheses, brackets, asterisks and such aren't special.
> - The symbol `:VOID` is equivalent to the empty string.
> - The symbol `:EVERYTHING` is equivalent to Perl's dot, i.e it matches everything (except maybe a newline character depending on the mode).
> - The symbols `:WORD-BOUNDARY` and `:NON-WORD-BOUNDARY` are equivalent to Perl's `"\b"` and `"\B"`.
> - The symbols `:DIGIT-CLASS`, `:NON-DIGIT-CLASS`, `:WORD-CHAR-CLASS`, `:NON-WORD-CHAR-CLASS`, `:WHITESPACE-CHAR-CLASS`, and `:NON-WHITESPACE-CHAR-CLASS` are equivalent to Perl's special character classes `"\d"`, `"\D"`, `"\w"`, `"\W"`, `"\s"`, and `"\S"` respectively.
> - The symbols `:START-ANCHOR`, `:END-ANCHOR`, `:MODELESS-START-ANCHOR`, `:MODELESS-END-ANCHOR`, and `:MODELESS-END-ANCHOR-NO-NEWLINE` are equivalent to Perl's `"^"`, `"$"`, `"\A"`, `"\Z"`, and `"\z"` respectively.
> - The symbols `:CASE-INSENSITIVE-P`, `:CASE-SENSITIVE-P`, `:MULTI-LINE-MODE-P`, `:NOT-MULTI-LINE-MODE-P`, `:SINGLE-LINE-MODE-P`, and `:NOT-SINGLE-LINE-MODE-P` are equivalent to Perl's embedded modifiers `"(?i)"`, `"(?-i)"`, `"(?m)"`, `"(?-m)"`, `"(?s)"`, and `"(?-s)"`. As usual, changes applied to modes are kept local to the innermost enclosing grouping or clustering construct.
> - `(:FLAGS {<modifier>}*)` where `<modifier>` is one of the modifier symbols from above is used to group modifier symbols. The modifiers are applied from left to right. (This construct is obviously redundant. It is only there because it's used by the parser.)
> - `(:SEQUENCE {<parse-tree>}*)` means a sequence of parse trees, i.e. the parse trees must match one after another. Example: `(:SEQUENCE #\f #\o #\o)` is equivalent to the parse tree `"foo"`.
> - `(:GROUP {<parse-tree>}*)` is like `:SEQUENCE` but changes applied to modifier flags (see above) are kept local to the parse trees enclosed by this construct. Think of it as the S-expression variant of Perl's `"(?:<pattern>)"` construct.
> - `(:ALTERNATION {<parse-tree>}*)` means an alternation of parse trees, i.e. one of the parse trees must match. Example: `(:ALTERNATION #\b #\a #\z)` is equivalent to the Perl regex string `"b|a|z"`.
> - `(:BRANCH <test> <parse-tree>)` is for conditional regular expressions. `<test>` is either a number which stands for a register or a parse tree which is a look-ahead or look-behind assertion. See the entry for `(?(<condition>)<yes-pattern>|<no-pattern>)` in `man perlre` for the semantics of this construct. If `<parse-tree>` is an alternation is must enclose exactly one or two parse trees where the second one (if present) will be treated as the "no-pattern" - in all other cases `<parse-tree>` will be treated as the "yes-pattern".
> - `(:POSITIVE-LOOKAHEAD|:NEGATIVE-LOOKAHEAD|:POSITIVE-LOOKBEHIND|:NEGATIVE-LOOKBEHIND <parse-tree>)` should be pretty obvious...
> - `(:GREEDY-REPETITION|:NON-GREEDY-REPETITION <min> <max> <parse-tree>)` where `<min>` is a non-negative integer and `<max>` is either a non-negative integer not smaller than `<min>` or `NIL` will result in a regular expression which tries to match `<parse-tree>` at least `<min>` times and at most `<max>` times (or as often as possible if `<max>` is `NIL`). So, e.g., `(:NON-GREEDY-REPETITION 0 1 "ab")` is equivalent to the Perl regex string `"(?:ab)??"`.
> - `(:STANDALONE <parse-tree>)` is an "independent" subexpression, i.e. `(:STANDALONE "bar")` is equivalent to the Perl regex string `"(?>bar)"`.
> - `(:REGISTER <parse-tree>)` is a capturing register group. As usual, registers are counted from left to right beginning with 1.
> - `(:BACK-REFERENCE <number>)` where `<number>` is a positive integer is a back-reference to a register group.
> - `(:CHAR-CLASS|:INVERTED-CHAR-CLASS {<item>}*)` where `<item>` is either a character, a character range, or a symbol for a special character class (see above) will be translated into a (one character wide) character class. A character range looks like `(:RANGE <char1> <char2>)` where `<char1>` and `<char2>` are characters such that `(CHAR<= <char1> <char2>)` is true. Example: `(:INVERTED-CHAR-CLASS #\a (:RANGE #\D #\G) :DIGIT-CLASS)` is equivalent to the Perl regex string `"[^aD-G\d]"`.
> Because `CREATE-SCANNER` is defined as a generic function which dispatches on its first argument there's a certain ambiguity: Although strings are valid parse trees they will be interpreted as Perl regex strings when given to `CREATE-SCANNER`. To circumvent this you can always use the equivalent parse tree `(:GROUP <string>)` instead.

> Note that currently `create-scanner` doesn't always check for the well-formedness of its first argument, i.e. you are expected to provide correct parse trees. This will most likely change in future releases.

> The usage of the keyword argument `extended-mode` obviously doesn't make sense if `CREATE-SCANNER` is applied to parse trees and will signal an error.

> If you want to find out how parse trees are related to Perl regex strings you should play around with `CL-PPCRE::PARSE-STRING` - a function which converts Perl regex strings to parse trees. Here are some examples:

>     * (cl-ppcre::parse-string "(ab)*")
>     (:GREEDY-REPETITION 0 NIL (:REGISTER (:SEQUENCE #\a #\b)))

>     * (cl-ppcre::parse-string "(a(b))")
>     (:REGISTER (:SEQUENCE #\a (:REGISTER #\b)))

>     * (cl-ppcre::parse-string "(?:abc){3,5}")
>     (:GREEDY-REPETITION 3 5 (:GROUP (:SEQUENCE #\a (:SEQUENCE #\b #\c))))
>     ;; (:GREEDY-REPETITION 3 5 (:SEQUENCE #\a #\b #\c)) would also be OK,
>     ;; or even (:GREEDY-REPETITION 3 5 "abc")

>     * (cl-ppcre::parse-string "a(?i)b(?-i)c")
>     (:SEQUENCE #\a
>      (:SEQUENCE (:FLAGS :CASE-INSENSITIVE-P)
>       (:SEQUENCE #\b (:SEQUENCE (:FLAGS :CASE-SENSITIVE-P) #\c))))
>     ;; same as (:SEQUENCE #\a :CASE-INSENSITIVE-P #\b :CASE-SENSITIVE-P #\c)

>     * (cl-ppcre::parse-string "(?=a)b")
>     (:SEQUENCE (:POSITIVE-LOOKAHEAD #\a) #\b)

For the rest of this section `regex` can always be a string (which is interpreted as a Perl regular expression), a parse tree, or a scanner created by `CREATE-SCANNER`. The `start` and `end` keyword parameters are always used as in `SCAN`.

[Function]
scan regex target-string `&key` start end => match-start, match-end, reg-starts, reg-ends

> Searches the string `target-string` from `start` (which defaults to 0) to `end` (which default to the length of `target-string`) and tries to match `regex`. On success returns four values - the start of the match, the end of the match, and two arrays denoting the beginnings and ends of register matches. On failure returns `NIL`. `target-string` will be coerced to a simple string if it isn't one already.

> `SCAN` acts as if the part of `target-string` between `start` and `end` were a standalone string, i.e. look-aheads and look-behinds can't look beyond these boundaries.

> Examples:

>     * (cl-ppcre:scan "(a)*b" "xaaabd")
>     1
>     5
>     #(3)
>     #(4)

>     * (cl-ppcre:scan "(a)*b" "xaaabd" :start 1)
>     1
>     5
>     #(3)
>     #(4)

>     * (cl-ppcre:scan "(a)*b" "xaaabd" :start 2)
>     2
>     5
>     #(3)
>     #(4)

>     * (cl-ppcre:scan "(a)*b" "xaaabd" :end 4)
>     NIL

>     * (cl-ppcre:scan '(:GREEDY-REPETITION 0 NIL #\b) "bbbc")
>     0
>     3
>     #()
>     #()

>     * (cl-ppcre:scan '(:GREEDY-REPETITION 4 6 #\b) "bbbc")
>     NIL

>     * (let ((s (cl-ppcre:create-scanner "(([a-c])+)x")))
>         (cl-ppcre:scan s "abcxy"))
>     0
>     4
>     #(0 2)
>     #(3 3)

[Function]
scan-to-strings regex target-string `&key` start end => match, regs

> Like `SCAN` but returns substrings of `target-string` instead of positions, i.e. this function returns two values on success: the whole match as a string plus an array of substrings (or `NIL`s) corresponding to the matched registers.

> Examples:

>     * (cl-ppcre:scan-to-strings "[^b]*b" "aaabd")
>     "aaab"
>     #()

>     * (cl-ppcre:scan-to-strings "([^b])*b" "aaabd")
>     "aaab"
>     #("a")

>     * (cl-ppcre:scan-to-strings "(([^b])*)b" "aaabd")
>     "aaab"
>     #("aaa" "a")

[Macro]
do-scans (match-start match-end reg-starts reg-ends regex target-string `&optional` result-form `&key` start end) declaration* statement* => result*

> A macro which iterates over `target-string` and tries to match `regex` as often as possible evaluating `statement*` with `match-start`, `match-end`, `reg-starts`, and `reg-ends` bound to the four return values of each match (see `SCAN`) in turn. After the last match, returns `result-form` if provided or `NIL` otherwise. An implicit block named `NIL` surrounds `DO-SCANS`; `RETURN` may be used to terminate the loop immediately. If `regex` matches an empty string the scan is continued one position behind this match.

> This is the most general macro to iterate over all matches in a target string. See the source code of `DO-MATCHES`, `ALL-MATCHES`, `SPLIT`, or `REGEX-REPLACE-ALL` for examples of its usage.

[Macro]
do-matches (match-start match-end regex target-string `&optional` result-form `&key` start end) declaration* statement* => result*

> Like `DO-SCANS` but doesn't bind variables to the register arrays.

> Example:

>     * (defun foo (regex target-string &key (start 0) (end (length target-string)))
>         (let ((sum 0))
>           (cl-ppcre:do-matches (s e regex target-string nil :start start :end end)
>             (incf sum (- e s)))
>           (format t "~,2F% of the string was inside of a match~%"
>                     ;; note: doesn't check for division by zero
>                     (float (* 100 (/ sum (- end start)))))))

>     FOO

>     * (foo "a" "abcabcabc")
>     33.33% of the string was inside of a match
>     NIL
>     * (foo "aa|b" "aacabcbbc")
>     55.56% of the string was inside of a match
>     NIL

[Macro]
do-matches-as-strings (match-var regex target-string `&optional` result-form `&key` start end) declaration* statement* => result*

> Like `DO-MATCHES` but binds `match-var` to the substring of `target-string` corresponding to each match in turn.

> Example:

>     * (defun crossfoot (target-string &key (start 0) (end (length target-string)))
>         (let ((sum 0))
>           (cl-ppcre:do-matches-as-strings (m :digit-class
>                                              target-string nil
>                                              :start start :end end)
>             (incf sum (parse-integer m)))
>           (if (< sum 10)
>             sum
>             (crossfoot (format nil "~A" sum)))))

>     CROSSFOOT

>     * (crossfoot "bar")
>     0

>     * (crossfoot "a3x")
>     3

>     * (crossfoot "12345")
>     6

[Function]
all-matches regex target-string `&key` start end => list

> Returns a list containing the start and end positions of all matches of `regex` against `target-string`, i.e. if there are `N` matches the list contains `(* 2 N)` elements. If `regex` matches an empty string the scan is continued one position behind this match.

> Examples:

>     * (cl-ppcre:all-matches "a" "foo bar baz")
>     (5 6 9 10)

>     * (cl-ppcre:all-matches "\\w*" "foo bar baz")
>     (0 3 3 3 4 7 7 7 8 11 11 11)

[Function]
all-matches-as-strings regex target-string `&key` start end => list

> Like `ALL-MATCHES` but returns a list of substrings instead.

> Examples:

>     * (cl-ppcre:all-matches-as-strings "a" "foo bar baz")
>     ("a" "a")

>     * (cl-ppcre:all-matches-as-strings "\\w*" "foo bar baz")
>     ("foo" "" "bar" "" "baz" "")

[Function]
split regex target-string `&key` start end with-registers-p => list

> Matches `regex` against `target-string` as often as possible and returns a list of the substrings between the matches. If `with-registers-p` is true, substrings corresponding to matched registers (if any) are inserted into the list as well. If `regex` matches an empty string the scan is continued one position behind this match. Empty matches at the start or the end of the target string are always left out.

> Examples:

>     * (cl-ppcre:split "\\s+" "foo   bar baz
>     frob")
>     ("foo" "bar" "baz" "frob")

>     * (cl-ppcre:split "\\s*" "foo bar   baz")
>     ("f" "o" "o" "b" "a" "r" "b" "a" "z")

>     * (cl-ppcre:split "(\\s+)" "foo bar   baz")
>     ("foo" "bar" "baz")

>     * (cl-ppcre:split "(\\s+)" "foo bar   baz" :with-registers-p t)
>     ("foo" " " "bar" "   " "baz")

>     * (cl-ppcre:split "(\\s)(\\s*)" "foo bar   baz" :with-registers-p t)
>     ("foo" " " "" "bar" " " "  " "baz")

[Function]
regex-replace regex target-string replacement `&key` start end preserve-case => list

> Try to match `target-string` between `start` and `end` against `regex` and replace the first match with the string `replacement`. `replacement` can contain the special substrings `"\&"` for the whole match, `"\`"` for the part of `target-string` before the match, `"\'"` for the part of `target-string` after the match, `"\N"` or `"\{N}"` for the `N`th register where `N` is a positive integer. If `preserve-case` is true (default is `NIL`), the replacement will try to preserve the case (all upper case, all lower case, or capitalized) of the match. The result will always be a fresh string, even if `regex` doesn't match.

> Examples:

>     * (cl-ppcre:regex-replace "fo+" "foo bar" "frob")
>     "frob bar"

>     * (cl-ppcre:regex-replace "fo+" "FOO bar" "frob")
>     "FOO bar"

>     * (cl-ppcre:regex-replace "(?i)fo+" "FOO bar" "frob")
>     "frob bar"

>     * (cl-ppcre:regex-replace "(?i)fo+" "FOO bar" "frob" :preserve-case t)
>     "FROB bar"

>     * (cl-ppcre:regex-replace "(?i)fo+" "Foo bar" "frob" :preserve-case t)
>     "Frob bar"

>     * (cl-ppcre:regex-replace "bar" "foo bar baz" "[frob (was '\\&' between '\\`' and '\\'')]")
>     "foo [frob (was 'bar' between 'foo ' and ' baz')] baz"

[Function]
regex-replace-all regex target-string replacement `&key` start end preserve-case => list

> Like `REGEX-REPLACE` but replaces all matches.

> Examples:

>     * (cl-ppcre:regex-replace-all "(?i)fo+" "foo Fooo FOOOO bar" "frob" :preserve-case t)
>     "frob Frob FROB bar"

>     * (cl-ppcre:regex-replace-all "(?i)f(o+)" "foo Fooo FOOOO bar" "fr\\1b" :preserve-case t)
>     "froob Frooob FROOOOB bar"

[Function]
regex-apropos regex `&optional` packages `&key` case-insensitive => list

> Like `APROPOS` but searches for interned symbols which match the regular expression `regex`. The output is implementation-dependent. If `case-insensitive` is true (which is the default) and `regex` isn't already a scanner, a case-insensitive scanner is used.

> Here are examples for CMUCL:

>     * *package*
>     #<The COMMON-LISP-USER package, 16/21 internal, 0/9 external>

>     * (defun foo (n &optional (k 0)) (+ 3 n k))
>     FOO

>     * (defparameter foo "bar")
>     FOO

>     * (defparameter |foobar| 42)
>     |foobar|

>     * (defparameter fooboo 43)
>     FOOBOO

>     * (defclass frobar () ())
>     #<STANDARD-CLASS FROBAR {4874E625}>

>     * (cl-ppcre:regex-apropos "foo(?:bar)?")
>     FOO [variable] value: "bar"
>         [compiled function] (N &OPTIONAL (K 0))
>     FOOBOO [variable] value: 43
>     |foobar| [variable] value: 42

>     * (cl-ppcre:regex-apropos "(?:foo|fro)bar")
>     PCL::|COMMON-LISP-USER::FROBAR class predicate| [compiled closure]
>     FROBAR [class] #<STANDARD-CLASS FROBAR {4874E625}>
>     |foobar| [variable] value: 42

>     * (cl-ppcre:regex-apropos "(?:foo|fro)bar" 'cl-user)
>     FROBAR [class] #<STANDARD-CLASS FROBAR {4874E625}>
>     |foobar| [variable] value: 42

>     * (cl-ppcre:regex-apropos "(?:foo|fro)bar" '(pcl ext))
>     PCL::|COMMON-LISP-USER::FROBAR class predicate| [compiled closure]

>     * (cl-ppcre:regex-apropos "foo")
>     FOO [variable] value: "bar"
>         [compiled function] (N &OPTIONAL (K 0))
>     FOOBOO [variable] value: 43
>     |foobar| [variable] value: 42

>     * (cl-ppcre:regex-apropos "foo" nil :case-insensitive nil)
>     |foobar| [variable] value: 42

[Function]
regex-apropos-list regex `&optional` packages `&key` upcase => list

> Like `APROPOS-LIST` but searches for interned symbols which match the regular expression `regex`. If `case-insensitive` is true (which is the default) and `regex` isn't already a scanner, a case-insensitive scanner is used.

> Example (continued from above):

>     * (cl-ppcre:regex-apropos-list "foo(?:bar)?")
>     (|foobar| FOOBOO FOO)

### Download and installation

CL-PPCRE together with this documentation can be downloaded from http://weitz.de/files/cl-ppcre.tgz.

CL-PPCRE comes with a simple system definition for MK:DEFSYSTEM so you can either adapt it to your needs or just unpack the archive and from within the CL-PPCRE directory start your Lisp image and evaluate the form `(mk:compile-system "cl-ppcre")` which should compile and load the whole system.

If for the some reason you don't want to use MK:DEFSYSTEM you can also get away with something like this:

    (loop for name in '("packages" "specials" "util" "lexer"
                        "parser" "regex-class" "convert" "optimize"
                        "closures" "repetition-closures" "scanner" "api")
          do (compile-file (make-pathname :name name
                                          :type "lisp"))
             (load name))

Note that on CL implementations which use the Python compiler (i.e. CMUCL, SBCL, SCL) you can concatenate the compiled object files to create one single object file which you can load afterwards:

    cat {packages,specials,util,lexer,parser,regex-class,convert,optimize,closures,repetition-closures,scanner,api}.x86f > cl-ppcre.x86f

(Replace ".`x86f`" with the correct suffix for your platform.)

### Testing CL-PPCRE

CL-PPCRE comes with a comprehensive test suite most of which is stolen from the PCRE library. You can use it like this:

    * (mk:compile-system "cl-ppcre-test")
    ; Loading #p"/home/edi/cl-ppcre/cl-ppcre.system".
    ; Loading #p"/home/edi/cl-ppcre/packages.x86f".
    ; Loading #p"/home/edi/cl-ppcre/specials.x86f".
    ; Loading #p"/home/edi/cl-ppcre/util.x86f".
    ; Loading #p"/home/edi/cl-ppcre/lexer.x86f".
    ; Loading #p"/home/edi/cl-ppcre/parser.x86f".
    ; Loading #p"/home/edi/cl-ppcre/regex-class.x86f".
    ; Loading #p"/home/edi/cl-ppcre/convert.x86f".
    ; Loading #p"/home/edi/cl-ppcre/optimize.x86f".
    ; Loading #p"/home/edi/cl-ppcre/closures.x86f".
    ; Loading #p"/home/edi/cl-ppcre/repetition-closures.x86f".
    ; Loading #p"/home/edi/cl-ppcre/scanner.x86f".
    ; Loading #p"/home/edi/cl-ppcre/api.x86f".
    ; Loading #p"/home/edi/cl-ppcre/ppcre-tests.x86f".
    NIL

    * (load "testdata")
    T

    * (cl-ppcre-test:test)

    ;; ....
    ;; (a list of incompatibilities with Perl)

With LispWorks and SCL you can also call `CL-PPCRE-TEST:TEST` with a keyword argument argument `THREADED` which - in addition to the usual tests - will also check whether the scanners created by CL-PPCRE are thread-safe.

Note that the file `testdata.lisp` provided with CL-PPCRE was created on a Linux system with Perl 5.8.0. You can (and you should if you're on Mac OS or Windows) create your own `testdata.lisp` with the Perl script `perltest.pl`:

    edi@bird:~/cl-ppcre > perl perltest.pl < testinput > testdata.lisp

Of course you can also create your own tests - the format accepted by `perltest.pl` should be rather clear from looking at the file `testinput`. Note that the target strings are wrapped in double quotes and then fed to Perl's `eval` so you can use ugly Perl constructs like, say, `a@{['b' x 10]}c` which will result in the target string `"abbbbbbbbbbc"`.

### Compatibility with Perl

Depending on your Perl version you might encounter a couple of small incompatibilities with Perl most of which aren't due to CL-PPCRE:

#### Empty strings instead of `undef` in `$1`, `$2`, etc.

(Cf. case #629 of `testdata.lisp`.) This is a bug in Perl 5.6.1 and earlier which has been fixed in 5.8.0.

#### Strange scoping of embedded modifiers

(Cf. case #430 of `testdata.lisp`.) This is a bug in Perl 5.6.1 and earlier which has been fixed in 5.8.0.

#### Inconsistent capturing of `$1`, `$2`, etc.

(Cf. case #662 of `testdata.lisp`.) This is a bug in Perl which hasn't been fixed yet.

#### Captured groups not available outside of look-aheads and look-behinds

(Cf. case #1439 of `testdata.lisp`.) Well, OK, this ain't a Perl bug. I just can't quite understand why captured groups should only be seen within the scope of a look-ahead or look-behind. For the moment, CL-PPCRE and Perl agree to disagree... :)

#### Alternations don't always work from left to right

(Cf. case #790 of `testdata.lisp`.) I also think this a Perl bug but I currently have lost the drive to report it.

#### `"\r"` doesn't work with MCL

(Cf. case #9 of `testdata.lisp`.) For some strange reason that I don't understand MCL translates `#\Return` to `(CODE-CHAR 10)` while MacPerl translates `"\r"` to `(CODE-CHAR 13)`. Hmmm...

#### What about `"\w"`?

CL-PPCRE uses `ALPHANUMERICP` to decide whether a character matches Perl's `"\w"`, so depending on your CL implementation you might encounter differences between Perl and CL-PPCRE when matching non-ASCII characters.

### Performance

#### Benchmarking

The CL-PPCRE test suite can also be used for benchmarking purposes: If you call `perltest.pl` with a command line argument it will be interpreted as the number of seconds each test should run. Perl will time its tests accordingly and create output which, when fed to `CL-PPCRE-TEST:TEST`, will result in a benchmark. Here's an example:

    edi@bird:~/cl-ppcre > echo "/((a{0,5}){0,5})*[c]/
    aaaaaaaaaaaac

    /((a{0,5})*)*[c]/
    aaaaaaaaaaaac" | perl perltest.pl .5 > timedata.lisp
    1
    2

    edi@bird:~/cl-ppcre > cmucl -quiet
    ; Loading #p"/home/edi/.cmucl-init".

    * (mk:compile-system "cl-ppcre-test")
    ; Loading #p"/home/edi/cl-ppcre/cl-ppcre.system".
    ; Loading #p"/home/edi/cl-ppcre/packages.x86f".
    ; Loading #p"/home/edi/cl-ppcre/specials.x86f".
    ; Loading #p"/home/edi/cl-ppcre/util.x86f".
    ; Loading #p"/home/edi/cl-ppcre/lexer.x86f".
    ; Loading #p"/home/edi/cl-ppcre/parser.x86f".
    ; Loading #p"/home/edi/cl-ppcre/regex-class.x86f".
    ; Loading #p"/home/edi/cl-ppcre/convert.x86f".
    ; Loading #p"/home/edi/cl-ppcre/optimize.x86f".
    ; Loading #p"/home/edi/cl-ppcre/closures.x86f".
    ; Loading #p"/home/edi/cl-ppcre/repetition-closures.x86f".
    ; Loading #p"/home/edi/cl-ppcre/scanner.x86f".
    ; Loading #p"/home/edi/cl-ppcre/api.x86f".
    ; Loading #p"/home/edi/cl-ppcre/ppcre-tests.x86f".
    NIL

    * (load "timedata")
    T

    * (cl-ppcre-test:test)
       1: 0.5559 (1000000 repetitions, Perl: 4.5330 seconds, CL-PPCRE: 2.5200 seconds)
       2: 0.4573 (1000000 repetitions, Perl: 4.5922 seconds, CL-PPCRE: 2.1000 seconds)
    NIL

We gave two test cases to `perltest.pl` and asked it to repeat those tests often enough so that it takes at least 0.5 seconds to run each of them. In both cases, CMUCL was about twice as fast as Perl.

Here are some more benchmarks (done with Perl 5.6.1 and CMUCL 18d+):

Test case | Repetitions | Perl (sec) | CL-PPCRE (sec) | Ratio CL-PPCRE/Perl
`"@{['x' x 100]}" =~ /(.)*/s` | 100000 | 0.1394 | 0.0700 | 0.5022
`"@{['x' x 1000]}" =~ /(.)*/s` | 100000 | 0.1628 | 0.0600 | 0.3685
`"@{['x' x 10000]}" =~ /(.)*/s` | 100000 | 0.5071 | 0.0600 | 0.1183
`"@{['x' x 100000]}" =~ /(.)*/s` | 10000 | 0.3902 | 0.0000 | 0.0000
`"@{['x' x 100]}" =~ /.*/` | 100000 | 0.1520 | 0.0800 | 0.5262
`"@{['x' x 1000]}" =~ /.*/` | 100000 | 0.3786 | 0.5400 | 1.4263
`"@{['x' x 10000]}" =~ /.*/` | 10000 | 0.2709 | 0.5100 | 1.8826
`"@{['x' x 100000]}" =~ /.*/` | 1000 | 0.2734 | 0.5100 | 1.8656
`"@{['x' x 100]}" =~ /.*/s` | 100000 | 0.1320 | 0.0300 | 0.2274
`"@{['x' x 1000]}" =~ /.*/s` | 100000 | 0.1634 | 0.0300 | 0.1836
`"@{['x' x 10000]}" =~ /.*/s` | 100000 | 0.5304 | 0.0300 | 0.0566
`"@{['x' x 100000]}" =~ /.*/s` | 10000 | 0.3966 | 0.0000 | 0.0000
`"@{['x' x 100]}" =~ /x*/` | 100000 | 0.1507 | 0.0900 | 0.5970
`"@{['x' x 1000]}" =~ /x*/` | 100000 | 0.3782 | 0.6300 | 1.6658
`"@{['x' x 10000]}" =~ /x*/` | 10000 | 0.2730 | 0.6000 | 2.1981
`"@{['x' x 100000]}" =~ /x*/` | 1000 | 0.2708 | 0.5900 | 2.1790
`"@{['x' x 100]}" =~ /[xy]*/` | 100000 | 0.2637 | 0.1500 | 0.5688
`"@{['x' x 1000]}" =~ /[xy]*/` | 10000 | 0.1449 | 0.1200 | 0.8282
`"@{['x' x 10000]}" =~ /[xy]*/` | 1000 | 0.1344 | 0.1100 | 0.8185
`"@{['x' x 100000]}" =~ /[xy]*/` | 100 | 0.1355 | 0.1200 | 0.8857
`"@{['x' x 100]}" =~ /(.)*/` | 100000 | 0.1523 | 0.1100 | 0.7220
`"@{['x' x 1000]}" =~ /(.)*/` | 100000 | 0.3735 | 0.5700 | 1.5262
`"@{['x' x 10000]}" =~ /(.)*/` | 10000 | 0.2735 | 0.5100 | 1.8647
`"@{['x' x 100000]}" =~ /(.)*/` | 1000 | 0.2598 | 0.5000 | 1.9242
`"@{['x' x 100]}" =~ /(x)*/` | 100000 | 0.1565 | 0.1300 | 0.8307
`"@{['x' x 1000]}" =~ /(x)*/` | 100000 | 0.3783 | 0.6600 | 1.7446
`"@{['x' x 10000]}" =~ /(x)*/` | 10000 | 0.2720 | 0.6000 | 2.2055
`"@{['x' x 100000]}" =~ /(x)*/` | 1000 | 0.2725 | 0.6000 | 2.2020
`"@{['x' x 100]}" =~ /(y|x)*/` | 10000 | 0.2411 | 0.1000 | 0.4147
`"@{['x' x 1000]}" =~ /(y|x)*/` | 1000 | 0.2313 | 0.0900 | 0.3891
`"@{['x' x 10000]}" =~ /(y|x)*/` | 100 | 0.2336 | 0.0900 | 0.3852
`"@{['x' x 100000]}" =~ /(y|x)*/` | 10 | 0.4165 | 0.0900 | 0.2161
`"@{['x' x 100]}" =~ /([xy])*/` | 100000 | 0.2678 | 0.1800 | 0.6721
`"@{['x' x 1000]}" =~ /([xy])*/` | 10000 | 0.1459 | 0.1200 | 0.8227
`"@{['x' x 10000]}" =~ /([xy])*/` | 1000 | 0.1372 | 0.1100 | 0.8017
`"@{['x' x 100000]}" =~ /([xy])*/` | 100 | 0.1358 | 0.1100 | 0.8098
`"@{['x' x 100]}" =~ /((x){2})*/` | 10000 | 0.1073 | 0.0400 | 0.3727
`"@{['x' x 1000]}" =~ /((x){2})*/` | 10000 | 0.9146 | 0.2400 | 0.2624
`"@{['x' x 10000]}" =~ /((x){2})*/` | 1000 | 0.9020 | 0.2300 | 0.2550
`"@{['x' x 100000]}" =~ /((x){2})*/` | 100 | 0.8983 | 0.2300 | 0.2560
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}FOOBARBAZ" =~ /[a-z]*FOOBARBAZ/` | 100000 | 0.2829 | 0.2300 | 0.8129
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}FOOBARBAZ" =~ /[a-z]*FOOBARBAZ/` | 10000 | 0.1859 | 0.1700 | 0.9143
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}FOOBARBAZ" =~ /[a-z]*FOOBARBAZ/` | 1000 | 0.1420 | 0.1700 | 1.1968
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /[a-z]*FOOBARBAZ/` | 1000000 | 0.9196 | 0.4600 | 0.5002
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /[a-z]*FOOBARBAZ/` | 100000 | 0.2166 | 0.2500 | 1.1542
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /[a-z]*FOOBARBAZ/` | 10000 | 0.1465 | 0.2300 | 1.5696
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}FOOBARBAZ" =~ /([a-z])*FOOBARBAZ/` | 100000 | 0.2917 | 0.2600 | 0.8915
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}FOOBARBAZ" =~ /([a-z])*FOOBARBAZ/` | 10000 | 0.1811 | 0.1800 | 0.9942
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}FOOBARBAZ" =~ /([a-z])*FOOBARBAZ/` | 1000 | 0.1424 | 0.1600 | 1.1233
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /([a-z])*FOOBARBAZ/` | 1000000 | 0.9154 | 0.7400 | 0.8083
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /([a-z])*FOOBARBAZ/` | 100000 | 0.2170 | 0.2800 | 1.2901
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /([a-z])*FOOBARBAZ/` | 10000 | 0.1497 | 0.2300 | 1.5360
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}FOOBARBAZ" =~ /([a-z]|ab)*FOOBARBAZ/` | 10000 | 0.4359 | 0.1500 | 0.3441
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}FOOBARBAZ" =~ /([a-z]|ab)*FOOBARBAZ/` | 1000 | 0.5456 | 0.1500 | 0.2749
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}FOOBARBAZ" =~ /([a-z]|ab)*FOOBARBAZ/` | 10 | 0.2039 | 0.0600 | 0.2943
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /([a-z]|ab)*FOOBARBAZ/` | 1000000 | 0.9311 | 0.7400 | 0.7947
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /([a-z]|ab)*FOOBARBAZ/` | 100000 | 0.2162 | 0.2700 | 1.2489
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /([a-z]|ab)*FOOBARBAZ/` | 10000 | 0.1488 | 0.2300 | 1.5455
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /[a-z]*FOOBARBAZ/i` | 1000 | 0.1555 | 0.0000 | 0.0000
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /[a-z]*FOOBARBAZ/i` | 10 | 0.1441 | 0.0000 | 0.0000
`"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /[a-z]*FOOBARBAZ/i` | 10 | 13.7150 | 0.0100 | 0.0007

As you might have noticed, Perl shines if it can reduce significant parts of the matching process to cases where it can advance through the target string one character at a time. This leads to C code where you can very efficiently test and increment a pointer into a string in a tight loop and can hardly be beaten with CL. In almost all other cases, the CMUCL/CL-PPCRE combination is usually faster than Perl - sometimes a lot faster.

Note that Perl as well as CL-PPCRE keep the rightmost matches in registers - keep that in mind if you benchmark against other regex implementations. Also note that `CL-PPCRE-TEST:TEST` automatically skips test cases where Perl and CL-PPCRE don't agree.

#### Other performance issues

While the scanners created by CL-PPCRE are pretty fast, the process which creates scanners from Perl regex strings and parse trees is currently rather inefficient and conses a lot. It is recommended that you store and re-use scanners if possible. The `DO-`macros will do this for you automatically.

Of course, the usual rules for creating efficient regular expressions apply to CL-PPCRE as well although it can optimize a couple of cases itself. The most important rule is probably that you shouldn't use capturing groups if you don't need the captured information, i.e. use `"(?:a|b)*"` instead of `"(a|b)*"` if you don't need to refer to the register. (In fact, in this particular case CL-PPCRE will be able to optimize away the register group, but it won't if you replace `"a|b"` with, say, `"a|bc"`.)

If you're really concerned with performance you can optimize the scanners for (special) character classes a little bit if you don't plan to use the whole character set of your CL implementation: Change the value of `+REGEX-CHAR-CODE-LIMIT+` in the file `util.lisp` before compiling CL-PPCRE. This might make sense with, e.g., LispWorks, AllegroCL, or CLISP where `CHAR-CODE-LIMIT` is as high as 65536.

Another thing to consider is that, for performance reasons, CL-PPCRE assumes that most of the target strings you're trying to match are simple strings and coerces non-simple strings to simple strings before scanning them. If you plan on working with non-simple strings mostly you might consider modifying the CL-PPCRE source code. This is easy: Change all occurences of `SCHAR` to `CHAR` and remove the parts in `api.lisp` where the coercion takes place - that's all.

### Bugs and problems

#### Stack overflow

CL-PPCRE can optimize away a lot of unnecessary backtracking but sometimes this simple isn't possible. With complicated regular expressions and long strings this might lead to stack overflows depending on your machine and your CL implementation.

Here's one example with CLISP:

    [1]> (defun target (n) (concatenate 'string (make-string n :initial-element #\a) "b"))
    TARGET

    [2]> (cl-ppcre:scan "a*" (target 1000))
    0 ;
    1000 ;
    #() ;
    #()

    [3]> (cl-ppcre:scan "(?:a|b)*" (target 1000))
    0 ;
    1001 ;
    #() ;
    #()

    [4]> (cl-ppcre:scan "(a|b)*" (target 1000))
    0 ;
    1001 ;
    #(1000) ;
    #(1001)

    [5]> (cl-ppcre:scan "(a|b)*" (target 10000))
    0 ;
    10001 ;
    #(10000) ;
    #(10001)

    [6]> (cl-ppcre:scan "(a|b)*" (target 100000))
    0 ;
    100001 ;
    #(100000) ;
    #(100001)

    [7]> (cl-ppcre:scan "(a|b)*" (target 1000000))
    0 ;
    1000001 ;
    #(1000000) ;
    #(1000001)

    ;; No problem until now - but...

    [8]> (cl-ppcre:scan "(a|)*" (target 100000))
    *** - Lisp stack overflow. RESET

    [9]> (cl-ppcre:scan "(a|)*" (target 3200))
    *** - Lisp stack overflow. RESET

With CMUCL the situation is better and worse at the same time. It will take a lot longer until CMUCL gives up but if it gives up the whole Lisp image will silently die (at least on my machine):

    * (defun target (n) (concatenate 'string (make-string n :initial-element #\a) "b"))
    TARGET

    * (cl-ppcre:scan "(a|)*" (target 3200))
    0
    3200
    #(3200)
    #(3200)

    * (cl-ppcre:scan "(a|)*" (target 10000))
    0
    10000
    #(10000)
    #(10000)

    * (cl-ppcre:scan "(a|)*" (target 100000))
    0
    100000
    #(100000)
    #(100000)

    * (cl-ppcre:scan "(a|)*" (target 1000000))
    0
    1000000
    #(1000000)
    #(1000000)

    ;; No problem until now - but...

    * (cl-ppcre:scan "(a|)*" (target 10000000))
    edi@bird:~ >

This behaviour can be changed with very conservative optimization settings but that'll make CL-PPCRE crawl compared to Perl.

You might want to compare this to the way Perl handles the same situation. It might lie to you:

    edi@bird:~ > perl -le '$_="a" x 32766 . "b"; /(a|)*/; print $1'

    edi@bird:~ > perl -le '$_="a" x 32767 . "b"; /(a|)*/; print $1'
    a

Or it might warn you before it's lying to you:

    edi@bird:~ > perl -lwe '$_="a" x 32767 . "b"; /(a|)*/; print $1'
    Complex regular subexpression recursion limit (32766) exceeded at -e line 1.
    a

Or it might simply die:

    edi@bird:~ > /opt/perl-5.8/bin/perl -lwe '$_="a" x 32767 . "b"; /(a|)*/; print $1'
    Segmentation fault

Your mileage may vary, of course...

#### Strange-looking output of `regex-apropos` with CLISP

This isn't really a problem of CL-PPCRE but of CLISP's (lack of) pretty printing support.

### Remarks

The sample output from CMUCL and CLISP has been slightly edited to increase readability.

All test cases and benchmarks in this document where performed on an IBM Thinkpad T23 laptop (Pentium III 1.2 GHz, 768 MB RAM) running Gentoo Linux 1.1a.

### Acknowledgements

Although I didn't use their code I was heavily inspired by looking at the Scheme/CL regex implementations of Dorai Sitaram and Michael Parker. Also, the nice folks from CMUCL's mailing list as well as the output of Perl's `use re "debug"` pragma have been very helpful in optimizing the scanners created by CL-PPCRE.

Thanks to the guys at "Café Olé" in Hamburg where I wrote most of the code and thanks to my wife for lending me her PowerBook to test CL-PPCRE with MCL and OpenMCL.

$Header: /usr/local/cvsrep/cl-ppcre/doc/index.html,v 1.1.1.1 2002/12/20 10:10:44 edi Exp $

BACK TO MY HOMEPAGE