What should you look for when rating the difficulty of a puzzle?

After you're finished making a puzzle, what types of details should you look for when rating it's difficulty? What makes a puzzle easier or harder? Keep in mind a puzzle includes both the question and the answer.

Answer

The best way to run difficulty tests on your puzzle is the tried-and-true method of a test audience. As others have pointed out, do not include yourself in this audience - you are biased because you already know the solution.

You can try to be rigorous in your testing b getting a large sample audience to test your puzzle against, which is the usual manner in which these things are tested (analogous to Game Testing), but for smaller puzzles, or if you're strapped for participants, you can choose a small handful.

If this is a personally-made puzzle for fun, you're more likely to use a small handful of participants, so choose your testers wisely. Know how skilled they are at puzzles of your type ahead of time, and base their test results on that information - if they're not typically good at puzzles, them solving it quickly might be a sign that you need to make it harder, though if they're particularly good at it, taking a very long time when they shouldn't have a problem could be a sign that you need to make it easier or include more hints.

Testing puzzles is an imprecise science. Sometimes what seems like a minor adjustment can make the puzzle unbeatable, and other times your focus on a preconceived solution will blind you to an obvious 'easy' solution you didn't think about before.

And of course this depends on what type of puzzle you've made - if it's wholly new, you'll need a lot more testers to really get a feel for how hard/easy you've made it.