https://blog.lvmbdv.dev/posts/the-puzzle-safety-net/

The dailyprog puzzle safety net

Ata Kuyumcu

dailyprog needs a new puzzle every day. I wrote the first few drafts by hand, then built a generator that writes them with an LLM and validates the output against real sandbox execution. The generator works, but it still needs someone to run it. Pick a pattern, kick off the script, review the result, commit. Fifteen minutes on a good day. More if the model gets stuck in a repair loop and I need to nudge it.

Photo by Basile Morin, cropped (CC BY-SA 4.0)

dailyprog needs a new puzzle every day. I wrote the first few drafts by hand, then built a generator that writes them with an LLM and validates the output against real sandbox execution. The generator works, but it still needs someone to run it. Pick a pattern, kick off the script, review the result, commit. Fifteen minutes on a good day. More if the model gets stuck in a repair loop and I need to nudge it.

Some days I don’t have fifteen minutes. Some days I forget. Before last week, either of those meant the site went dark at midnight UTC and stayed dark until I woke up and fixed it. Now it doesn’t.

At 22:00 UTC every night, two hours before the puzzle day rolls over, a CI job runs on the Forgejo runner. It checks whether tomorrow’s puzzle exists in the catalogue. If it does, nothing happens. If it doesn’t, the job generates one, validates it, commits it, and deploys it. The site never goes dark.

The job is a single workflow file. The check step is four lines of bash:

TOMORROW=$(date -u -d '+1 day' +%Y-%m-%d)
if [ -f "workspaces/app/content/puzzles/$TOMORROW.json" ]; then
  echo "puzzle for $TOMORROW is already authored — nothing to do"
fi

If the file is missing, the rest of the job runs: install toolchain, install dependencies, call the same gen:puzzle script I run manually. The model defaults to DeepSeek v4 Flash. Medium difficulty, no pattern hint. The diversity constraint (scan the last 7 puzzles’ technique tags, block repeats) keeps the model from producing three binary search puzzles in a row. It usually closes in 2 attempts.

The generation is the easy part. The interesting thing is what comes after.

The generator script only writes files to disk when it passes every gate: schema validation, reference solution against all test cases, brute-force collapse proof, technique diversity. That’s the first safety net. But the CI job runs a second one anyway. After the generator finishes, it fires the full catalogue test suite:

pnpm run check:ci
pnpm --filter @dailyprog/app exec vitest run lib/puzzles.test.ts lib/puzzle-quality.test.ts

puzzles.test.ts loads every .solution.ts file in the catalogue and runs the reference solution against every visible and hidden test case. If the generator produced a solution that doesn’t pass its own tests, this catches it. puzzle-quality.test.ts runs the guardrails: no duplicate technique in the recent window, no back-to-back hard days, every language produces output within the runtime budget. These are the same tests that run on every push to main. The generated puzzle goes through them twice.

If the tests pass, the job commits as puzzle-bot and pushes straight to main:

feat: add puzzle 47 - staggered delivery
Auto-generated by the nightly safety-net workflow.
Co-Authored-By: DeepSeek V4 Flash <noreply@deepseek.com>

Forgejo skips CI on pushes made with the run’s own token (standard loop prevention), so the commit doesn’t trigger another build. The puzzle is in git but not on the site. One more step: the job POSTs to Forgejo’s API to dispatch the deploy workflow. The deploy workflow SSHes into the VPS, pulls the new commit, rebuilds the container image with --no-cache, and brings it up. The new puzzle goes live about 10 minutes after generation started.

The two-hour lead time matters. I check the site most nights. If the model produced something weird (a puzzle where the English prompt is technically correct but the story doesn’t make sense, or one that passes every gate but just feels lazy), I have two hours to write a replacement. The safety net isn’t a substitute for human review. It’s a guarantee that the worst case is “a puzzle I didn’t author” rather than “no puzzle at all.”

The app doesn’t know any of this exists. It reads JSON files from content/puzzles/. The CI runner lives on the same VPS as the site, as an unprivileged user with no special access. The Deepseek API key is a repo secret scoped to the workflow. If the runner goes down, the site still serves puzzles from whatever is already in the catalogue.

The manual path and the automatic path converge on the same checks. I author a puzzle, run the tests, commit, deploy. CI generates one, runs the same tests, commits, deploys. The tests don’t care who wrote the JSON. If CI can’t pass them, the puzzle doesn’t ship. If it can, the puzzle is safe to serve.

There’s no handoff. No separate “auto-generated” code path with weaker validation. No flag that marks a puzzle as machine-authored and lets it skip a gate. The safety net is just the manual workflow running unattended.

Most days I still write the puzzle myself. But the days I can’t, nobody notices :^)