Documentation previewThese docs are actively being built. Some pages may change as the framework and examples are finalized.
GoForjUnreleased

Runtime Observability ​

Verified Scenario

This page is generated from an executable spec. An automated suite renders a fresh App from the current GoForj templates, applies every step below in order, and runs every verification command. If any step fails, the page does not ship.

Scenario 7 of 7 in the verified path. Plan on about 15 minutes.

This scenario follows the same application behavior through the surfaces operators use to trust a running App.

The goal is not to add new business behavior. The goal is to prove that HTTP requests, events, jobs, schedules, metrics, inspects, Lighthouse, and logs all describe the same runtime story.

What You Will Observe ​

  • route:list shows the HTTP surface.
  • API logs show request and lifecycle behavior.
  • Metrics expose bounded labels for routes, jobs, queues, and schedules.
  • Inspects capture request, job, scheduler, and CLI execution records.
  • Lighthouse provides a first-party operator view over recent runtime state.
flowchart TD
  request["POST /api/v1/users"] --> route["route:list registered route"]
  request --> inspectHTTP["HTTP inspect"]
  request --> event["users.created"]
  event --> job["reports:generate job"]
  job --> inspectJob["job inspect"]
  schedule["reports:daily schedule"] --> inspectSchedule["scheduler inspect"]
  job --> metrics["metrics with queue and job labels"]
  schedule --> metrics
  inspectHTTP --> lighthouse["Lighthouse"]
  inspectJob --> lighthouse
  inspectSchedule --> lighthouse
  metrics --> lighthouse

Prerequisites ​

Complete these scenarios first:

  1. JSON API Route
  2. Cached User Profile
  3. File Upload to Storage
  4. Users Created Event
  5. Reports Generate Job
  6. Reports Daily Schedule

Enable metrics and Lighthouse when creating or configuring the App.

Golden Path State ​

Before this scenario, the App has routes, cache, storage, events, jobs, workers, and a schedule.

After this scenario, you should know where to prove each runtime boundary: route list for HTTP, logs for process behavior, metrics for bounded counters and timings, inspects for execution records, and Lighthouse for operator-facing runtime state.

Build and Verify ​

bash
forj build
bash
go test ./...
bash
grep -Fx LIGHTHOUSE_INSPECT_ENABLED=true .env.local

Expected output includes:

  • LIGHTHOUSE_INSPECT_ENABLED=true
bash
forj route:list

Expected output includes:

  • /api/v1/users
  • /metrics

Trigger the Workflow ​

List the registered routes:

bash
forj route:list

With the default process-local workerpool driver, start the combined App so API, Jobs, and scheduler runtimes share one process:

bash
forj app

The built binary uses the same combined topology:

bash
./bin/app

To run API, workers, and scheduler as separate processes, first select a shared queue driver as described in Reports Generate Job. Each command is a long-running process, so run one in each of three terminals:

bash
forj api
bash
forj worker
bash
forj scheduler

Create a user:

bash
curl -X POST http://localhost:3000/api/v1/users \
  -H 'Content-Type: application/json' \
  -d '{"name":"Grace Hopper","email":"grace@example.test"}'

Expected behavior:

  • the API handles POST /api/v1/users
  • the service publishes users.created
  • the subscriber dispatches reports:generate
  • the worker processes reports:generate
  • storage receives a report artifact

Then check route output, process logs, metrics endpoints, inspect records, and Lighthouse for the same bounded names: /api/v1/users, users.created, reports:generate, and reports:daily.

Check Metrics ​

The combined local App exposes one shared metrics endpoint:

bash
curl http://localhost:3000/metrics

In a split topology using a shared queue backend, each direct runtime owns a dedicated process metrics endpoint:

bash
curl http://localhost:10000/metrics
curl http://localhost:10001/metrics
curl http://localhost:10002/metrics

These are the API, scheduler, and worker metrics listeners, respectively. The API still serves GET /metrics on port 3000, while the dedicated port keeps scrape traffic separate from application requests.

Look for bounded labels such as registered route, event topic, queue name, job name, scheduler job name, source, and status. Do not expect user IDs, emails, raw URLs, or storage filenames to appear as labels.

Useful evidence across the different metric families includes:

text
route="/api/v1/users"
topic="users.created"
job_name="reports:generate"
job_name="reports:daily"
queue="default"
source="jobs"
source="scheduler"

Scheduler metrics use the bounded job_name label for the registered schedule name. Metric names can evolve with the metrics package and generated App version, but the label discipline must remain stable: bounded operational names, not user-controlled data.

Check Inspects ​

Open Lighthouse and inspect recent executions.

Look for records from:

  • HTTP request handling
  • queued job processing
  • scheduler runs
  • CLI commands such as route:list

Each inspect should tell a bounded execution story: source runtime, duration, status, timeline events, and safe payload details where enabled.

Use inspect for the product surface. trace_id may still appear as a correlation field in logs or payloads.

Check Logs ​

Use logs to confirm lifecycle and failure behavior:

bash
forj app

When using a shared queue backend and separate processes, inspect the output from forj api, forj worker, and forj scheduler individually.

Good logs should answer:

  • which runtime started
  • which runtime is shutting down
  • whether optional resources degraded
  • whether a job or schedule failed
  • which bounded runtime identity emitted the line

Logs should not be the only way to discover registered routes, queue depth, or scheduler state. Use route lists, metrics, inspects, and Lighthouse for those surfaces.

Follow the Schedule ​

Run the combined App with the temporary short interval from Reports Daily Schedule when testing locally. Separate scheduler and worker processes require a shared queue backend.

Expected evidence:

  • scheduler logs show reports:daily
  • scheduler metrics include the schedule name
  • a scheduler inspect is retained by Lighthouse
  • workers process one or more reports:generate jobs
  • job metrics and job inspects use reports:generate

This proves the schedule dispatches queued work instead of performing report generation inside scheduler bootstrap.

Operations ​

Operational notes:

  • Use route:list as the HTTP source of truth.
  • Use forj app for the normal local workerpool topology; split runtimes only after selecting a shared queue backend.
  • Use metrics for bounded counters and timings; do not put user IDs, raw URLs, emails, request IDs, or filenames in labels.
  • Use inspect records and Lighthouse to follow request, job, scheduler, and CLI execution stories.
  • Use logs to confirm lifecycle and failure behavior, not as the only route or queue inventory.

Troubleshooting ​

If no route appears, run forj build and then forj route:list.

If no job is processed, confirm the API and worker processes use a shared queue backend. workerpool is process-local; use Redis, SQL-backed queues, or another shared backend when API and worker run separately.

If metrics are empty, confirm metrics were enabled for the surface you are checking:

text
METRICS_HTTP_ENABLED=true
METRICS_QUEUE_ENABLED=true
METRICS_EVENTS_ENABLED=true
METRICS_SCHEDULER_ENABLED=true

Local Apps enable inspect capture with LIGHTHOUSE_INSPECT_ENABLED=true in .env.local. If Lighthouse has no inspect records, confirm that setting is still active, Lighthouse is enabled, and the inspect buffer is not saturated.

Common Mistakes ​

Common mistakes

  • Do not use metrics labels for user IDs, emails, raw paths, request IDs, or filenames.
  • Do not treat Lighthouse as the only observability surface.
  • Do not call inspects traces in user-facing docs.
  • Do not rely only on logs to discover registered routes or queue state.
  • Do not hide worker or scheduler startup inside constructors.
  • Do not expect workerpool queues to cross process boundaries.
  • Do not treat missing Lighthouse records as proof that work did not happen; confirm inspect configuration and buffer limits.

Next Steps ​