Debugging Failed Actions Runs

Debugging Failed Actions Runs

Overview

When a Forgejo Actions workflow fails, the standard Forgejo web UI gives you some information, but the native REST API is severely limited: there are no endpoints for run logs, per-job status, or structured failure data. You are forced to click through pages in the browser.

The proxy changes this. By scraping the Forgejo web UI under the hood and re-exposing the results as proper REST endpoints, it gives you a full machine-readable view of every run, every job, and every log line — all accessible from the terminal or from automation.

The key advantage in a debugging session is specificity: instead of downloading an entire run’s worth of logs and grepping through them yourself, the proxy lets you target exactly the failed job, ask for structured diagnostics, and get a summarised root-cause in a single request.

---

Quick Start: One Command

The fastest path from “the run failed” to “here is why” is the diagnose command:

thfg run diagnose <owner/repo> --latest

For a specific run number:

thfg run diagnose templates/forgejo-github-script-compat-template --run 2

This command hits the /diagnostics endpoint on your behalf, inspects every job, identifies the first failed step in each failing job, and prints a structured summary. In most cases it will tell you whether the failure is an action fetch timeout, a token scope problem, or a network error before you have read a single raw log line.

If thfg itself is not responding, jump straight to When thfg itself fails before continuing.

---

Step-by-Step Manual Path

Sometimes you need finer control — for instance, when you want to diff two specific jobs, pipe logs through your own tooling, or call the API from a CI script. The manual path follows four steps.

Step 1 — Confirm the run actually failed

Before diving into logs it is worth confirming the run status, especially in long-running multi-stage pipelines where a partially-failed run can still show a green badge on the branch summary.

Using the CLI:

thfg run list <owner/repo>

Using curl directly against the proxy:

curl -s https://forgejo-proxy.hochguertel.work/api/v1/repos/<owner>/<repo>/actions/runs/latest \
  -H "Authorization: token $FORGEJO_TOKEN" | jq '{run_number, status, conclusion}'

A run that failed will have "conclusion": "failure". A run that is still executing will have "status": "in_progress" — wait for it to finish before reading logs or diagnostics.

Step 2 — List jobs and find the failing ones

A multi-job workflow has one status per job. The overall run conclusion is the worst conclusion across all jobs, so a single failing job marks the whole run as failed even if every other job succeeded.

List all jobs for the latest run:

curl -s https://forgejo-proxy.hochguertel.work/api/v1/repos/<owner>/<repo>/actions/runs/latest/jobs \
  -H "Authorization: token $FORGEJO_TOKEN" |
  jq '.[] | {job_index, attempt, name, status, step_count: (.steps | length)}'

Or with the CLI:

thfg run jobs <owner/repo>
thfg run performance <owner/repo> --latest

For a specific run number:

curl -s https://forgejo-proxy.hochguertel.work/api/v1/repos/<owner>/<repo>/actions/runs/2/jobs \
  -H "Authorization: token $FORGEJO_TOKEN" |
  jq '.[] | {job_index, attempt, name, status, step_count: (.steps | length)}'

Real-world example. For run #2 of templates/forgejo-github-script-compat-template, the job list looked like this:

[
  { "job_index": 0, "attempt": 1, "name": "Preflight", "status": "success" },
  { "job_index": 1, "attempt": 1, "name": "github-script v7 basic", "status": "failure" },
  { "job_index": 2, "attempt": 1, "name": "github-script v7 context", "status": "failure" },
  { "job_index": 3, "attempt": 1, "name": "github-script v9 basic", "status": "failure" },
  { "job_index": 4, "attempt": 1, "name": "github-script v9 graphql", "status": "failure" }
]

Job 0 succeeded; jobs 1–4 all failed. Because they all failed at the same point (before any step code ran), a single root cause was likely — which the logs confirmed.

thfg run performance reads the same enriched payload and is often the fastest way to spot a slow or failing step before you open the raw log.

Step 3 — Fetch logs for the right job

Pass the job_index from the jobs payload or the job name to the logs endpoint. The proxy accepts both:

# by index
curl -s "https://forgejo-proxy.hochguertel.work/api/v1/repos/<owner>/<repo>/actions/runs/latest/logs?job=1" \
  -H "Authorization: token $FORGEJO_TOKEN"

# by name
curl -s "https://forgejo-proxy.hochguertel.work/api/v1/repos/<owner>/<repo>/actions/runs/latest/logs?job=github-script+v7+basic" \
  -H "Authorization: token $FORGEJO_TOKEN"

# convenience selector: first failed job
curl -s "https://forgejo-proxy.hochguertel.work/api/v1/repos/<owner>/<repo>/actions/runs/latest/logs?job=failed" \
  -H "Authorization: token $FORGEJO_TOKEN"

With the CLI, the --job flag accepts the same index, name, or the failed keyword:

thfg run logs-latest <owner/repo> --job failed
thfg run logs-latest <owner/repo> --job "github-script v7 basic"
thfg run logs-latest <owner/repo> --job 1

Note on default behaviour. When you omit ?job= entirely and the run is a failure, the proxy now automatically selects the first failed job. This is a deliberate change from the original behaviour (which defaulted to job 0). If you always want job 0 regardless of outcome, pass ?job=0 explicitly.

Step 4 — Use structured diagnostics

Raw logs are useful but noisy. The /diagnostics endpoint returns a machine-readable JSON object that identifies failed jobs, failed steps, their exit codes, and a short human-readable summary of each failure — all in one call:

curl -s https://forgejo-proxy.hochguertel.work/api/v1/repos/<owner>/<repo>/actions/runs/latest/diagnostics \
  -H "Authorization: token $FORGEJO_TOKEN" | jq .

For a specific run:

curl -s https://forgejo-proxy.hochguertel.work/api/v1/repos/<owner>/<repo>/actions/runs/2/diagnostics \
  -H "Authorization: token $FORGEJO_TOKEN" | jq .

See Interpreting the diagnostics response for a detailed walkthrough of the JSON structure.

---

Common Failure Modes

HTTP 408 — Action fetch timeout

What you see in the logs:

git fetch 'https://actions.hochguertel.work/actions/github-script' HTTP 408

What it means. The Forgejo runner tries to download the action repository over HTTPS before executing any step code. If the action registry (the server hosting the action source) is unreachable or takes too long to respond, the fetch times out with HTTP 408. The job fails immediately; no step code ever runs.

What it is not. This is not a GITHUB_*_URL or base-url configuration issue. Those settings control where API calls inside your step scripts go. The git fetch that fails here is the runner downloading the action definition itself, which is completely separate.

How to confirm. Fetch the logs for any one of the failing jobs. If the very first line of meaningful output is a git fetch … HTTP 408, all other failing jobs will have the same error. You do not need to read every job’s logs individually.

Fix. Check that the action registry host (actions.hochguertel.work in this example) is reachable from the runner. This is a network/firewall problem between the runner and the registry, not a proxy configuration problem.

---

Authentication failures (HTTP 401 / 403)

What you see in the logs or in an API response:

HTTP 401 Unauthorized

{ "message": "Unauthorized" }

What it means. Either the token you passed to the proxy does not have sufficient scope, or the proxy’s own session with Forgejo has expired.

Checklist:

• Confirm FORGEJO_TOKEN is set and non-empty in your shell.
• Confirm the token has read:repository and read:issue scopes at minimum; Actions endpoints may require read:actions if Forgejo has that scope.
• If the proxy itself is returning 403 from an upstream scrape call, the proxy’s admin session may need to be refreshed. Check the proxy logs.

---

git RPC failures

What you see in the logs:

remote: Repository not found
fatal: repository 'https://…' not found

or

error: RPC failed; curl 56 OpenSSL SSL_read: Connection reset by peer, errno 104

What it means. These are network errors between the runner and the proxy (or between the proxy and Forgejo). They are usually transient and go away on retry.

Fix. Rerun the workflow: thfg run rerun <owner/repo> --latest. If it keeps failing, check proxy → Forgejo connectivity.

---

Interpreting the Diagnostics Response

The /diagnostics endpoint returns a structure like this:

{
  "run_number": 2,
  "status": "completed",
  "conclusion": "failure",
  "failed_job_count": 4,
  "total_job_count": 5,
  "jobs": [
    {
      "index": 0,
      "name": "Preflight",
      "conclusion": "success",
      "failed_steps": []
    },
    {
      "index": 1,
      "name": "github-script v7 basic",
      "conclusion": "failure",
      "failed_steps": [
        {
          "name": "Run actions/github-script@v7",
          "exit_code": 1,
          "summary": "git fetch 'https://actions.hochguertel.work/actions/github-script' HTTP 408"
        }
      ]
    }
  ],
  "root_cause_hint": "Action fetch timeout — runner cannot reach the action registry at actions.hochguertel.work"
}

Key fields to read:

Field	Meaning
conclusion	Overall run conclusion: success, failure, cancelled
failed_job_count	How many jobs failed — gives you the blast radius at a glance
jobs[].failed_steps	The specific step(s) that failed within each job
failed_steps[].summary	A short extract from the log around the failure point
root_cause_hint	Best-effort plain-English diagnosis generated by the proxy

The root_cause_hint is heuristic. It is helpful for the common cases (408 timeout, 401 auth, RPC error) but may be empty or generic for unusual failures. Always verify against the raw log if the hint does not feel right.

---

Multi-Job Workflow Tip

When a workflow has multiple jobs, the proxy’s default log-fetch behaviour matters:

• Before the change: GET .../runs/latest/logs (no ?job=) returned job 0’s logs.
• After the change: the same request returns the first failed job’s logs when the run conclusion is failure. Job 0’s logs are still available as ?job=0.

This default is intentional. In most debugging sessions you want to see why it failed, not the passing setup job that ran first. But if you are writing automation that always needs job 0 — for example, a pre-flight check that logs environment info — always pass ?job=0 explicitly to opt out of the smart default.

The jobs payload already includes job_index, attempt, log_url, and per-step duration data, so it is worth fetching that endpoint before diving into a long log stream.

---

When thfg Itself Fails

If thfg commands error out or return unexpected results, run the doctor command first:

thfg doctor

This prints:

• The resolved proxy URL and Forgejo host
• Whether the proxy is reachable (HTTP ping)
• Whether authentication is valid (token check)
• The config file that was loaded and which env vars overrode it

The most common issues caught by thfg doctor are:

1. FORGEJO_PROXY_URL not set — the CLI is hitting the wrong host or localhost.
2. FORGEJO_TOKEN missing or expired — all authenticated calls fail silently.
3. Wrong config file — a local thfg.config.json in the working directory is overriding the global config.

You can also check the resolved version to make sure you are running the expected build:

thfg --version

---

Decision-Tree Diagram

The flowchart below summarises the debugging process from first observation to resolution.

flowchart TD
    A([Run failed or suspected failed]) --> B{thfg itself working?}
    B -- No --> C[thfg doctor\nfix config / connectivity]
    C --> B
    B -- Yes --> D[thfg run diagnose owner/repo --latest]
    D --> E{root_cause_hint clear?}
    E -- Yes --> F{Failure type?}
    E -- No --> G[thfg run logs-latest\n--job failed]
    G --> H[Read raw log\nnear first error line]
    H --> F
    F -- HTTP 408 timeout --> I[Action registry unreachable\nCheck network / firewall\nbetween runner and registry]
    F -- HTTP 401/403 --> J[Auth issue\nCheck FORGEJO_TOKEN scope\nor proxy admin session]
    F -- git RPC error --> K[Network blip\nRetry: thfg run rerun\nowner/repo --latest]
    F -- Other --> L[thfg run jobs owner/repo\nIdentify all failing jobs\nFetch logs per job]
    I --> M([Resolved or escalate])
    J --> M
    K --> M
    L --> M

---

Further Reading

• Actions Extensions reference — full endpoint list with parameters
• Authentication — token scopes and proxy auth setup
• github-script Compatibility Matrix — known actions/github-script behaviour on this stack