Common validation failures and what they mean

A reference catalogue of the most common error messages you'll see during validation, deploy, and post-push verification — grouped by tier, with the actual cause of each and the fastest fix.

Written By Alan Gandy

Last updated About 1 month ago

When something goes wrong, CodeTeach shows you a one-line summary. This article is the catalogue: error messages grouped by where they happen (Tier 1 sandbox, Tier 2 replica, generation, deploy, post-push), what each one actually means, and the fastest fix.

For the broader workflow of "how do I respond when validation fails?" see When validation fails. This article is the reference you grep when you have a specific error message in front of you.

Solution-side failures (Tier 1 — sandbox)

Test #N failed: expected X, got Y

The most common failure. Solution ran but produced different output than the test expected.

Sub-cases:

  • Output is "almost right" (extra newline, extra space, different capitalisation) → the test's comparison is too strict. Switch from Exact match to Output contains in the Tests tab, or trim whitespace in the solution.
  • Output is structurally wrong → solution has a bug. Open the Solution tab, find the issue, fix, re-validate.
  • You'd rather change what's expected → open the Tests tab, edit the test's Expected Output to match what the solution actually produces.

Solution timed out (30 seconds)

Solution ran longer than the sandbox timeout. Usually:

  • Infinite loop
  • Reading from stdin without an EOF marker (program waits forever for input that never comes)
  • Recursive call without a base case
  • O(n²) or worse on a test input larger than expected

Open the Solution tab, identify the offending function, fix, re-validate.

Solution crashed: ImportError: No module named X

Solution depends on a Python package that isn't in the sandbox. The sandbox has the standard library plus common scientific packages (numpy, pandas, matplotlib, scikit-learn) for jupyter-python assignments — anything else needs to be added explicitly.

Easiest fix: rewrite the solution to avoid the dependency.Hard fix: add a pip install step to the workflow YAML (Tier 2 territory; not for the faint of heart).

Solution crashed: NullPointerException / TypeError / RuntimeError

Solution has a bug. The error message includes the line number — open the Solution tab, navigate to that line, fix.

Permission denied / FileNotFoundError

Solution tried to read or write a file the sandbox doesn't have or doesn't allow. Common causes:

  • Hardcoded absolute path (/home/user/data.txt) → change to relative path
  • Missing input file the test setup didn't create → include the file in the test's input or restructure the assignment to read from stdin

Workflow-side failures (Tier 2 — replica)

Test command failed: process substitution / not supported

The workflow tried to use bash syntax (<(...)) but the runner uses POSIX /bin/sh. CodeTeach's mechanical-fix layer normally catches this — if you're seeing it, the AI Wizard regenerated the workflow and introduced a bash-ism.

Click Fix with AI Wizard — usually resolves automatically.

YAML parse error

The workflow file has invalid YAML. Common causes:

  • Missing quotes around 'on': (YAML 1.1 parses bare on as boolean, breaking the GHA trigger)
  • Mis-indented step

Click Fix with AI Wizard, or open the workflow file via Download ZIP → fix manually → manual deploy.

Step ID 'test-1' contains invalid characters

GitHub Actions step IDs must be alphanumeric only (no hyphens, dots, etc.). The grader uppercases them to match env vars.

Mechanical-fix should catch this. If it persists, open the Tests tab and rename the failing test to something purely alphanumeric (test1 instead of test-1). The AI builder uses test names to generate step IDs.

Action requires Node 20 but runner provides Node 24

The classroom-resources actions were built for Node 20 and refuse to run on default Node 24. The workflow needs FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true on the job's env: block.

Mechanical-fix adds this automatically. If you're seeing it after a manual edit, restore the env block.

Workflow is missing required permissions: block

Workflow needs permissions: { checks: write, actions: read, contents: read }. The deterministic builder always emits this.

Click Fix with AI Wizard, or restore the block manually.

Generation-side failures

AI rate limit exceeded

You've hit the per-minute token limit on your selected AI provider. Wait 60 seconds and try again, or switch to a different model in the wizard form.

Out of credits

Your account is out of generation credits. Visit the Credits page in the sidebar to top up, or switch to BYOK (Bring Your Own Key) in Settings to use your own API key without spending CodeTeach credits.

Generation timed out

The AI took longer than 5 minutes to respond. Often a sign of a model overload on the provider side. Try again, or switch models.

AI output didn't parse / was malformed

The AI returned content CodeTeach couldn't parse (e.g., JSON with a syntax error). Rare, but happens. Just try again — usually resolves on retry.

Deploy-side failures

GitHub App not installed on selected org

You picked an org from the dropdown that doesn't actually have the App. Click Install on another org / account, install on the right one, return.

Repository already exists

The repo name is taken in that org. Pick a different name.

Object does not exist (eventual consistency)

GitHub git database hasn't caught up. CodeTeach retries automatically. Just wait — if it ultimately fails, click Try Deploy Again.

Cannot deploy: org is the same as personal username

You installed the App on your personal GitHub account (not an org). Personal accounts can't host Classroom templates. Install on an actual organization instead.

Post-push failures (real GitHub Actions)

These show in the post-push banner after deploy.

Real-GHA test #N failed but Tier 2 passed

Drift between CodeTeach's replica and real GitHub Actions. Usually one of:

  • POSIX-shell incompatibility that escaped Tier 2
  • Timing-sensitive output (clock-based, locale-based)
  • Environment difference (Python version, line endings)

Open the failing run on GitHub, look at the test command, identify the difference. File a bug report via the chat widget so we can fix the replica drift — every report makes the next deploy more reliable.

Workflow not found in repo

The deploy didn't push the workflow file successfully. Open the repo on GitHub, check .github/workflows/. If empty, redeploy.

Autograder action couldn't access repo

The CodeTeach App's permissions don't include what the autograder needs. Re-install the App with the latest permissions list.

When nothing in this list matches your error

  • Read the full error message, not just the summary. The bounce-back banner has a "Show details" link that expands the full failure output.
  • Try the AI Wizard. It often catches things this catalogue hasn't covered.
  • Use the chat widget (bottom-right of every page) and paste the full error. We respond within a day, usually faster.

Where to go next