How do regular expressions actually work?

The parts you actually need

Most characters match themselves. The language is the exceptions, and there are fewer than people expect.

Character classes say what kind of character. A dot matches almost anything, \d a digit, \w a word character, \s whitespace. Their capitalised forms invert them, so \D is anything that is not a digit. Square brackets build your own: [aeiou] matches one vowel, [a-z0-9] one lowercase letter or digit, and a caret just inside them negates, so [^,] means anything except a comma.

Quantifiers say how many. A star means zero or more, a plus one or more, a question mark zero or one. Braces are exact: {3} is precisely three, {2,4} is two to four, {2,} is two or more. Almost every pattern is character classes and quantifiers alternating.

Anchors say where. A caret at the start matches the beginning, a dollar the end, and \b a word boundary — which is what makes searching for a short word not match inside longer ones. Parentheses group, a pipe alternates, and a backslash escapes a character that would otherwise mean something.

Two more constructs earn their place once the basics are comfortable. Capture groups, written as ordinary parentheses, keep what they matched so it can be referenced afterwards — numbered from one in the order the opening brackets appear, which is worth knowing because nested groups are easy to miscount. A group that takes part in the pattern but never matches, because it sat behind an alternation that was not taken, reports as undefined rather than as an empty string, and that difference matters the moment the result reaches code.

Lookarounds assert without consuming. A positive lookahead says the next characters must match something, but does not include them in the result; a negative one says they must not. They are how you express "a number not followed by a percent sign" or "a word preceded by a dollar" without capturing the surrounding characters. They are also where patterns start becoming hard to read, so the practical advice is to use them where they genuinely simplify and to add a comment when you do — a pattern nobody can read is a pattern nobody will safely change.

That is close to the whole working language. Lookarounds, named groups and backreferences exist and are worth learning later; they appear in a small minority of real patterns.

Greedy by default

This causes more confusion than anything else in the syntax, and it is one character to fix.

Quantifiers are greedy: they take as much as they can while still allowing the rest of the pattern to match. Applied to a string containing two quoted values, the pattern for a quote, then any characters, then a quote does not match the first quoted section. It matches from the first quote to the *last* one, swallowing everything between including the quotes in the middle.

Adding a question mark after the quantifier makes it lazy, taking as little as possible. That single character is the difference between extracting one field and extracting the whole line, and it is the first thing to try when a pattern is matching far more than intended.

The better fix is often to be specific rather than to be lazy. Instead of "any characters" between the quotes, say "anything that is not a quote" — a negated character class. That cannot overshoot at all, is faster, and expresses what you actually meant. Reaching for a negated class rather than a lazy dot is the habit that separates patterns that work from patterns that work on the example.

Where regex stops being the right tool

Regular expressions describe regular languages, which is a precise mathematical category, and a great deal of what people want to parse falls outside it.

HTML is the standard example. Tags nest arbitrarily deep, and a pattern cannot count nesting — there is no way to express "matched pairs to any depth" in a regular language. A pattern can extract something from HTML you control, and it cannot parse HTML in general. The same applies to JSON, to source code and to anything with balanced brackets. Use a parser.

Email addresses are the other classic. The RFC grammar permits quoted local parts, comments and nested structures, so a fully compliant pattern runs to several kilobytes, matches forms no mail server accepts, and still cannot tell you whether the mailbox exists. For finding addresses in prose a permissive pattern is right; for validating one, send a confirmation email.

The useful rule is that regex is excellent at describing the *shape* of flat text and poor at anything requiring structure or memory. When a pattern starts growing conditionals and lookarounds to handle nesting, that is the signal to stop and use a parser instead.

The pattern that can take down a server

Catastrophic backtracking is worth understanding because it is a real vulnerability class rather than a performance curiosity.

When a pattern fails to match, the engine backtracks and tries other ways of dividing the input. Usually that is cheap. But when a quantifier is nested inside another — a group that can match many characters, itself repeated — the number of ways to split the input grows exponentially with length. A pattern like a repeated group of repeated letters, run against a string of forty letters that does not match, can take longer than the age of the universe to conclude it failed.

This has a name, ReDoS, and it has taken down real services. A user-supplied string hitting a vulnerable pattern in a request handler consumes a CPU core indefinitely. Cloudflare took a substantial outage from exactly this in 2019, from one pattern in a WAF rule.

The defences are straightforward. Avoid nesting quantifiers where the inner one can match the same characters as the outer. Prefer specific character classes over dots, which reduces the ways input can be divided. Never run a user-supplied pattern against a server. And test patterns against inputs that *fail*, since a pattern that matches quickly can still take exponential time to reject something similar — which is exactly the case an attacker supplies.

Questions