A real dashboard, frozen in time

Six-Month Backend AI Agent Engineering Plan

Build one production-style TypeScript, Node.js, and PostgreSQL agent service over four milestones at roughly eight focused hours per week. Each milestone extends the same repository and produces inspectable proof: durable orchestration, safe tool execution, behavioral evaluation, observability, and a tested production release. Keep the agent’s domain narrow enough that reliability can be demonstrated rather than hidden behind a large feature set.

Nothing staged. This is a saved copy of a real learner’s plan. No database, no AI, nothing running behind it. Just the receipts.

Plan stateACTIVE

Milestones4

Tasks18

Active now1

Target role

AI Agent Engineer (backend)

Learner goal

Move from full-stack web work into backend AI-agent engineering and ship a production-quality agent service within six months.

Milestone 1

1. Build a Durable Stateful Workflow Core

Weeks 1–6: turn existing backend skills into an agent workflow service that persists every meaningful transition and can resume interrupted work.

active

Deliverable

A versioned repository containing a TypeScript service, PostgreSQL migrations, architecture decision record, workflow-state diagram, REST API, deterministic fake model and tools, and a runnable demonstration of a multi-step task surviving process interruption. The data model must represent runs, steps, attempts, tool calls, outputs, timestamps, and terminal states.

active

Choose One Narrow Agent Job

Define a small workflow with a clear outcome, two or three tools, and at least one multi-step decision. Write its success and failure boundaries before implementing it.

A one-page scope document with example input, expected output, workflow steps, permitted tools, exclusions, and completion criteria.

scheduled

Model Workflow History in PostgreSQL

Design tables and constraints for runs, steps, attempts, tool calls, outputs, and state transitions.

Versioned SQL migrations plus a diagram or concise data-model document containing keys, relationships, status values, and invariants.

scheduled

Implement the Workflow State Machine

Build orchestration behind a REST API using deterministic fake model responses and tools first.

Endpoints to create, inspect, cancel, and resume runs, with state-transition logic separated from HTTP handlers.

scheduled

Add Recovery and Failure Tests

Exercise crashes, duplicate requests, tool timeouts, retry exhaustion, cancellation, and restart recovery.

Automated integration tests against PostgreSQL, including a test that interrupts and resumes an in-progress workflow.

Review rubric · 5 criteria

State is durable

A reviewer can stop the service during a workflow, restart it, and resume without losing completed steps or duplicating committed effects.

Transitions are explicit

Every workflow state and allowed transition is documented and enforced in code; invalid transitions produce a defined error.

Agent records are inspectable

Database queries can reconstruct a run’s steps, attempts, tool calls, outputs, and final status in chronological order.

Failure semantics are defined

Timeout, retry, cancellation, and terminal-failure behavior have explicit limits and automated tests.

Setup is reproducible

A new reviewer can start the database, apply migrations, run the service, and execute the demo using documented commands.

Milestone 2

2. Add Safe, Idempotent Tool Execution

Weeks 7–12: introduce real model-mediated tool selection while keeping authority and side effects under backend control.

locked

Deliverable

An extension of the workflow service with typed tool contracts, schema validation, caller permissions, per-run authorization, idempotency controls, audit records, timeout and retry policies, and at least one simulated side-effecting tool. Include an adversarial test suite proving denied actions do not execute.

scheduled

Define Typed Tool Contracts

Create a registry in which every tool declares its input schema, output schema, timeout, side-effect classification, and required permission.

Two or three registered tools with TypeScript types, runtime schemas, and contract tests.

scheduled

Build the Authorization Boundary

Pass authenticated caller context and run-scoped permissions into the executor, then check them before dispatch.

A policy module with allow and deny tests for each tool and role used by the demonstration.

scheduled

Protect Side Effects

Add idempotency keys, transactional claim logic, bounded retries, and timeouts to the simulated side-effecting tool.

An executor implementation and concurrency tests showing duplicate or simultaneous requests yield at most one committed effect.

scheduled

Run Adversarial Tool Tests

Test prompt-driven attempts to invoke unavailable tools, alter arguments, bypass permissions, exceed limits, or replay calls.

A documented test matrix and automated results showing the expected denial or safe behavior for every case.

Review rubric · 5 criteria

Inputs are validated

Malformed, oversized, or unexpected tool arguments are rejected before tool implementation code runs.

Permissions are enforced server-side

Tests show that caller identity and run policy determine tool access independently of model instructions.

Side effects are idempotent

Repeating the same authorized tool request does not create duplicate external effects.

Audit history is complete

Each attempted call records the requesting run, tool, validated arguments or safe summary, authorization result, attempt count, timing, and outcome.

Unsafe requests fail closed

Unknown tools, missing permissions, policy ambiguity, and exhausted limits result in denial without execution.

Milestone 3

3. Prove Behavior with Evals and Traces

Weeks 13–18: create repeatable evidence that the agent succeeds at its task and make failures diagnosable across model calls, tools, and state transitions.

locked

Deliverable

A versioned evaluation dataset with normal, edge, and adversarial cases; an automated eval runner; explicit outcome metrics; regression thresholds; structured logs, metrics, and correlated traces; and a short failure-analysis report based on actual eval results.

scheduled

Write the Evaluation Contract

Define what successful completion means, which failures matter, and which metrics represent quality, safety, reliability, latency, and cost.

A metric specification with calculation rules, thresholds, and examples of passing and failing outcomes.

scheduled

Create a Versioned Eval Dataset

Build representative normal, edge, recovery, and adversarial cases for the chosen agent job.

At least 20 reviewable cases with inputs, expected properties, permitted variations, tags, and rationale; avoid unverifiable exact-output matching where several answers are valid.

scheduled

Implement the Eval Runner

Run cases against deterministic components and a configured model, store results, calculate metrics, and compare them with thresholds.

A command-line runner that emits machine-readable results and a concise human-readable report, returning failure when required thresholds are missed.

scheduled

Instrument End-to-End Tracing

Add correlation identifiers, structured events, latency measurements, retry counts, token usage where available, and redaction rules.

A trace or local observability view for each eval run plus documentation mapping telemetry fields to diagnostic questions.

scheduled

Diagnose Two Real Failures

Select two failing evals, use traces to identify their causes, make bounded changes, and rerun the suite.

A short report containing evidence, root-cause hypotheses, changes made, before-and-after metrics, and remaining uncertainty.

Review rubric · 5 criteria

Success is outcome-based

Each eval checks task-specific correctness or completion evidence, not merely absence of exceptions.

Cases are reproducible

The dataset, model configuration, prompts, tool versions, evaluator version, and run identifiers are recorded sufficiently to repeat a run.

Regressions are detectable

A deliberately weakened prompt, policy, or workflow causes the suite to fail a documented threshold.

Traces explain behavior

A reviewer can follow one correlation identifier through the request, workflow states, model calls, tool attempts, retries, and final outcome.

Telemetry is safe and useful

Logs avoid secrets and unnecessary sensitive content while retaining the fields needed to diagnose tested failures.

Milestone 4

4. Ship and Defend a Production-Quality Service

Weeks 19–26: harden, deploy, operate, and explain the complete service as a backend engineering portfolio artifact.

locked

Deliverable

A deployed service or reproducible production-like environment with authentication, migrations, health checks, graceful shutdown, concurrency controls, secret handling, CI, automated tests and eval gates, dashboards or queryable telemetry, an incident runbook, a threat model, and a recorded or written technical walkthrough. Include a tagged release and a portfolio-ready case study grounded in repository evidence.

scheduled

Establish the Production Checklist

Translate the service’s risks into concrete controls for authentication, secrets, migrations, concurrency, retention, health, shutdown, rate limits, and dependency failures.

A checked-in readiness document linking every claimed control to code, configuration, a test, or an explicitly accepted limitation.

scheduled

Build the CI Quality Gate

Create fast checks for every commit and a slower stable eval and integration suite for protected changes or scheduled runs.

A CI workflow with retained test and eval reports plus a documented intentional-regression demonstration that fails the gate.

scheduled

Deploy and Exercise Failure Modes

Run the service in a production-like environment and inject provider timeouts, tool failure, process interruption, and duplicate delivery.

A scripted resilience exercise with traces, database evidence, observed recovery, and follow-up issues for weaknesses discovered.

scheduled

Write the Threat Model and Runbook

Document assets, trust boundaries, likely misuse, mitigations, alert signals, and operator recovery procedures.

A concise threat model and incident runbook reviewed against the implemented tool and workflow boundaries.

scheduled

Prepare the Technical Defense

Explain the system to another engineer and answer questions using repository evidence; do not rely on polished claims that cannot be demonstrated.

A 10–15 minute walkthrough or written case study covering architecture, one successful trace, one failure, eval results, safety controls, tradeoffs, and the next three improvements.

Review rubric · 5 criteria

Quality gates are automated

CI runs formatting or linting, type checks, unit tests, database integration tests, safety tests, and the stable eval subset; a seeded regression demonstrably blocks the pipeline.

Operations are documented

Runbooks explain deployment, migration, rollback, stuck-run recovery, retry storms, provider failure, and investigation using actual telemetry.

Production boundaries are explicit

Authentication, authorization, secret storage, retention, concurrency, rate limits, and external-provider assumptions are implemented or clearly documented as limitations.

Reliability is demonstrated

A scripted exercise injects at least three failures and records recovery behavior, data integrity, alerts or signals, and any unmet service objective.

Engineering decisions are defensible

The walkthrough explains state modeling, orchestration, tool safety, eval design, observability, tradeoffs, and known weaknesses using links to code, tests, traces, and results.

Latest evidence

The learner’s work stays attached.

2 attempts and 2 earned hints ship inside this saved copy. The plan never floats free of the work.

Attempt · Choose One Narrow Agent Job
I started on the evidence checker. Here is the function so far: async function listEvidence(repoUrl: string) { const { owner, repo } = parseRepoUrl(repoUrl); const commits = await octokit.rest.repos.listCommits({ owner, repo, per_page: 25 }); return commits.data.map((c) => ({ sha: c.sha, message: c.commit.message })); } It works on public repos, but on a private test repo it throws RequestError [HttpError]: Not Found (404). I expected 401 or 403 for auth problems, so I'm confused about the 404. I think I need to pass an auth token but I'm not sure where it belongs or why GitHub answers 404 instead of 403.

Level 2/5 hint · Choose One Narrow Agent Job
Look up GitHub REST API “authentication” and “404 Not Found for private resources.” Focus on how Octokit receives authentication when the client is constructed—not in `listCommits`—and why GitHub may conceal the existence of a private repository from an unauthenticated or unauthorized caller.

Read all messages →