Wingtip Labs makes a Regular Expressions Tutorial that has given hundreds of programmers the chance to learn thousands of regular expressions.

Most people beat the first few levels with no trouble: match literal text, use | for alternatives, use [] for character ranges. But pretty soon you’re matching log lines by date, or validating IP addresses, and–because this is a learning tool–people start to make mistakes.

Teaching people to correct those mistakes is hands-down the best part of my job.

It turns out the mistakes people make while learning regular expressions follow a Pareto distribution: 80% of people make the same 20% of mistakes. So if we can anticipate a relatively small number of mistakes, we can help a large number of students.

I built the first generation of mistake-following-clues (we call them Oops Advice) based on watching friends and family play the game. And, because mistakes follow the 80:20 rule, that body of advice was pretty widely useful. For example, ~half of students will put spaces around a pipe the first time they use it, just like my Dad did.

But this week, I sat down and doubled the number of Oops Advice patterns in the app, by harvesting our error logs. Here’s how.

Find the Hot Spots

First, I needed to get the mass of data into a form where my pattern-matching unconscious could be helpful; to see student progress, and to see a lot of it all at once.

Each row is a student who has created an account to get 10 free levels. In the progress-at-a-glance diagram, the 10 big boxes for each user are levels (green if they’ve ever beaten it), the smaller boxes inside are one attempt (the levels are generated dynamically so most students will play some levels a few times), and the pixels inside are solutions they tried. A yellow is an abandon (they saw the problem but never tried a solution), red is a failing regex, and green is a winning regex.

So from the glance data, I can see which levels are giving people trouble, and even how much trouble it’s giving them (reds and yellows). Then I can dig in either by student or by level to see if I can diagnose what advice to give.

Find the Patterns

Let’s look at one specific error. We’re seeing one user spend ~4 minutes solving level 32:

What can we tell from this attempt?

1. He’s not building the regular expression interactively (type a little, see what matches); he worked for two minutes before submitting an answer.
2. The second answer catches one syntactical problem, an extra close paren.
3. The “ah ha” moment came right after he submitted the second try: he forgot the space between Jun and the second numeric pattern!

In isolation, that’s an unfortunate typo. In aggregate, 1 in 7 students who try that level make that exact mistake. 14%!

Getting help when you need it

“Questions are places in your mind where answers fit.

If you haven’t asked the question, the answer has nowhere to go.”

— Clayton Christensen paraphrased by Jason Fried

A tutorial has a leg up on other applications. I know what my students’ intent is: they want to write a regex that matches all the green rows, doesn’t match any of the red rows. I can interpret every regex they submit as reaching toward that one goal, and provide the right advice just after a student makes that mistake.

We use a pretty simple JavaScript data structure to encode Oops Clues. Here’s the one for that mistake on that level:

1
2
3
4

{
  "pattern" : /[A-Z][a-z][a-z][^ ]/,
  "advice" : "You're missing the space between the month name and the date number."
}

The pattern attribute is a regex that acts on the regex the student submitted. Here I’m looking for three letters not followed by a space. And the advice is extremely contextual, to the problem the student is trying to solve in this level, and to their specific error.

Takeaways

1. In-person usability testing is fantastic, but it’s an expensive way to get deep access to a few people’s thoughts. It’s especially helpful for figuring out the intent behind really perplexing mistakes.
2. Scanning your logs is an inexpensive way to cull hundreds of mistakes, and look for wider patterns. But you’ll be forced to imagine how they got there.
3. Intent is key. I’ve got it easy: people are playing a puzzle game. In other apps you might spend more time to figure out what they hoped would happen.
4. Display your errors in a way your intuitive mind can suck out insights. I didn’t do this by paging through /var/log/httpd/error_log, that would have been agony.

If you liked this post, you should definitely see the regular expressions tutorial in action.