← Keystone
Testing Command Centre

ERP Testing Command Centre

Canonical 8-level test pipeline · sign-off chains · entry/exit gates · go-live readiness
v1.0 · Test Manager reference
SI-led Client-led Joint Client + SI Click any level for detail
LEVEL 8 · PARALLEL NFT — Non-Functional Test Performance · Load · Security · DR · Batch · Resilience

Click a level to see detail

Each card shows who executes, who signs off, entry / exit criteria, deliverables, and the most common pitfall.

Level decoder

Level Full name The question it answers Executed by
UnitUnit TestDoes each piece of code work on its own?SI developers
FATFunctional Acceptance TestDoes each configured function work as designed?SI Tester, per sprint
Mini-BATMini Business Acceptance TestAre the benefits still on track, checked early?Benefit Owners
SATSystem Acceptance TestDoes the whole system hang together after build?SI functional consultants / test team (SI Test Lead owns and signs)
SITSystem Integration TestDoes it still work with the other systems connected?SI + client testers, co-run by Client Test Manager + SI Test Lead (exit signed by Client Test Manager + Solution Architect)
Pre-UATPre-UAT readiness checkIs the UAT environment ready for the business to test in?Client Test Manager
UATUser Acceptance TestCan real users run their real business processes end to end?End users
BATBusiness Acceptance TestDoes it deliver the benefits the business signed up for?Benefit Owners + Executive Sponsor
NFTNon-Functional Test (parallel)Is it fast, secure and resilient enough for production?IT Ops Lead

The fourteen testing roles — reports-to, active stages, sign-off authority

Sign-off chains depend on knowing who does what — and who does NOT do what. The most common testing failure on ERP programmes is muddled roles. Each tile carries reports-to, active stages, responsibilities, sign-off authority, and the common mistake.

SI side

SI Programme Director
Reports to SI account / delivery exec · Active S10–S17
  • Programme-level accountability for delivery quality
  • Final SI escalation point on commercial and resource issues
  • Signs off the SI side of joint deliverables
Signs off: SI half of test exit reports
Common mistake: Getting drawn into individual defect triage rather than steering quality at programme level.
SI Solution Architect
Reports to SI Programme Director · Active S9–S17
  • Owns the Solution Design Document end-to-end
  • Jointly signs off architecture decisions with the Client SA
  • Maintains design integrity through build and test
Signs off: SIT (architecture-level integrations) and NFT (non-functional requirements), jointly with Client SA
Common mistake: Drifting away from the SDD without raising change requests when implementation reality bites.
SI Functional Lead(s)
Reports to SI Solution Architect · Active S11–S15
  • Owns the functional design within their workstream
  • Supports FAT/SAT/SIT/UAT — clarifies functional behaviour
  • Triages functional defects; reproduces issues with the test team
Signs off: Contributes to FAT acceptance; no formal sign-off authority
Common mistake: Fixing defects in isolation without checking impact on adjacent processes.
SI Test Lead
Reports to SI Programme Director · Active S11–S17
  • Owns the SI side of the test strategy with the Client Test Manager
  • Runs SAT execution; co-runs SIT with Client Test Manager
  • Owns the defect triage process from the SI side
Signs off: SAT
Common mistake: Not engaging early enough — Test Lead must be active in Discovery (S11) so Pre-UAT and SAT prep aren't rushed at end of build.
SI Tester / SI Test Team
Reports to SI Test Lead · Active S13–S15
  • Executes FAT (per sprint)
  • Executes SAT (post-build)
  • Supports SIT execution; logs defects in Azure DevOps
Signs off: Contributes to FAT and SAT; no formal sign-off authority
Common mistake: Under-staffing the test team — testing is the role most often cut when programmes are squeezed.
SI Technical Lead
Reports to SI Solution Architect · Active S11–S17
  • Resolves technical and integration defects
  • Owns the dev environment, CI/CD pipeline, code quality
Signs off: Unit testing
Common mistake: Treating Unit testing as a tick-box exercise rather than enforcing meaningful coverage.
SI Data Migration Lead
Reports to SI Solution Architect · Active S11–S17
  • Owns data migration design, build, mock loads, cutover loads
  • Runs data validation across mock cycles
Signs off: SI data migration sign-off (feeds into SIT and Cutover Readiness)
Common mistake: Optimistic mock-load estimates that collapse under real production volumes at cutover.

Client side

Client Test Manager
Reports to Programme Manager · Active S11–S17
  • Owns the test strategy from the Client side
  • Co-authors Pre-UAT scripts with Process Owners
  • Runs SIT alongside SI Test Lead; coordinates UAT and BAT
  • Manages the defect process from the Client side
Signs off: SIT (with Solution Architect); UAT (with Process Owners)
Common mistake: Starting too late — Client Test Manager must be in role by end of S10 at the latest.
Process Owners
Reports to Programme Manager (functionally) and business leadership · Active S2–S19 (most active S11–S17 for testing)
  • Accept FAT at Sprint Review
  • Coordinate Pre-UAT and UAT (write scripts with Client Test Manager, manage coverage)
  • Review BAT results
  • Do NOT execute UAT scripts
Signs off: UAT (per workstream, with Client Test Manager)
Common mistake: Getting drawn into executing UAT scripts because users are unavailable. Resist this — Process Owners executing UAT produces paper acceptance with no operational signal.
Users / End Users
Reports to Business line management · Active S14 (UAT) and S17 (Hypercare)
  • Execute UAT scripts hands-on
  • Provide usability feedback
  • Report defects through the defect process
Signs off: UAT script-by-script confirmation; no formal sign-off authority
Common mistake: Not being freed from BAU during UAT. UAT needs dedicated time, not squeezed-in time.
Benefit Owners
Reports to Executive Sponsor (named at S2) · Active S2–S19
  • Accountable for delivery of named KPI benefits
  • Review Mini-BAT outputs
  • Review BAT against the benefits map
Signs off: Mini-BAT and BAT (against benefits map)
Common mistake: Treating Benefit Owner as a ceremonial title — it's an operational accountability that runs from S2 to S19.
Solution Architect (Client)
Reports to CIO or equivalent · Active S9–S17
  • Co-owns the Solution Design Document with the SI Solution Architect
  • Ensures architectural fit with the wider IT estate
  • Signs off SIT alongside SI counterpart
Signs off: SIT (with Client Test Manager)
Common mistake: Deferring architecture decisions to the SI without independent challenge.
IT Ops Lead
Reports to CIO or equivalent · Active S10–S19
  • Owns NFT execution (or commissions it)
  • Manages production environment readiness
  • Runs DR rehearsals
Signs off: NFT (with Solution Architect); operational cutover readiness
Common mistake: NFT prep starting too late in S14 because IT Ops weren't engaged from S11.
Executive Sponsor
Reports to CEO / Board · Active S0–S19
  • Programme accountability to the Board
  • Final escalation point
  • Binary go/no-go authority at the Go-Live Gate
Signs off: GLG (Executive Go-Live Gate); BAT alongside Benefit Owners
Common mistake: Signing off GLG before NFT certification — they are separate gates and both must be evidenced.
Defect Manager (Client side)
Reports to Client Test Manager (escalation line to Programme Manager) · Active S14–S17 (light prep from S11)
  • Owns the end-to-end defect process across all test levels (SAT, SIT, UAT, BAT, NFT)
  • Daily triage with SI Test Lead and Client Functional Leads; calibrates severity per agreed P1–P4 definitions
  • Forecasts defect burn-down; reports daily during UAT, weekly to Steering, with clear view of zero-P1 and P2-with-workaround status at exit
  • Chases fixes across SI workstreams; escalates ageing P1/P2 defects
  • Carries the audit trail at exit — defect log reconciled to test scripts, sign-off evidence, residual defects with documented workarounds
Sign-off: Contributes evidence to UAT, BAT, NFT exit. No formal sign-off authority but the gatekeeper for the evidence those sign-offs rest on.
Common mistake: leaving this role implicit and assuming the Client Test Manager will absorb it. By UAT the Client Test Manager is running the testing programme; managing the defect process at the operational level needs a separate named person with no other accountability during the test phases. Scales: small programmes — Client Test Manager absorbs; medium — 3 days/week from S14 ramping to 5; large — full-time from S14.

Architecture roles with testing involvement

Two named Client-side architecture roles sit alongside the Solution Architect. Both report to the Solution Architect; both have testing involvement (integration test scope, master data integrity, cutover data state) but are programme-wide roles, not testing-specific. They do not count in the fourteen testing tiles above — they sit in the broader role catalogue.

Architecture roles (Client side)

Integration Lead (architecture role · involved in test design)
Reports to Solution Architect (Client side) · Active S11–S17
  • Owns programme-level integration requirements (S11) and architecture decisions (S12 ADR)
  • Owns integration build and test from S13 through hypercare
  • Co-chairs Integration Working Group with SI Technical Lead during S11–S14
  • On dual-pillar: owns the synchronisation framework (Dual Write or equivalent)
Sign-off: Accountable on Integration build and test exit; Responsible on Dual Write ADR (S12); contributes evidence to SIT and NFT exit
Common mistake: Staffing the role from S13 when integration build starts. Architecture wasn't designed for the integrations actually needed.
Sizing: Single-pillar small — SI Technical Lead absorbs (named risk); single-pillar medium+ — named; all dual-pillar — named, mandatory.
Data Architect (architecture role · involved in test design)
Reports to Solution Architect (Client side) · Active S11–S17
  • Owns the data model, integration data flows, master data ownership decisions, reference data design
  • Accountable on the Master Data Ownership Decision Record at Solution Design (S12)
  • Partners with Data Lead (workstream coordination) and Data Owners (per-domain hygiene) — distinct from both
  • Influences test data strategy (Section 11 of Test Strategy) and cutover data state
Sign-off: Accountable on the Master Data Ownership Decision Record (S12); contributes evidence to SIT data integrity and cutover data state
Common mistake: Assuming the Solution Architect carries it. SDD signed off at S12 with a hole where master data ownership should be.
Sizing: Small — Solution Architect absorbs with named risk; medium+ — named separately; large — Lead Data Architect on largest programmes.

Architecture roles sit Client-side only. SI counterparts are the SI Solution Architect, SI Technical Lead and SI Data Migration Lead (already in the testing role catalogue above).

FAT vs UAT — the most commonly confused boundary

The line is field-level vs process-level. Get this wrong and your acceptance evidence doesn't reflect real user experience.

FAT — Functional Acceptance Test

Field-level / configuration-level. Tests that individual user stories or configurations work as designed.

"The system allows creation of a case record with fields X, Y, Z. Confirm field types and validation."
Executed by: SI Tester · Accepted by: Process Owner at sprint review

UAT — User Acceptance Test

Process-level / business-flow. Tests end-to-end business journeys.

"A client advisor opens a case, captures contact details, progresses through the case journey to closure."
Executed by: Users (end users) · Signed off by: Client Test Manager + Process Owners

Quick test

If a tester is checking field X accepts data type A → that's FAT.
If a tester is walking through "I'm a stock controller, show me my Tuesday morning workflow" → that's UAT.

P1 — Critical

System unusable, no workaround

Blocks all testing or renders the area untestable. Ship-stopper.

ResponseImmediate
Resolution target24 hrs
Go-live toleranceZero open
P2 — High

Major function impaired, workaround exists

Important function broken but business can continue with manual or alternative steps.

Response4 hrs
Resolution target3 working days
Go-live toleranceZero or formally accepted
P3 — Medium

Minor function impaired, acceptable workaround

Inconvenience but not blocking. Logged with post-go-live fix plan.

Response1 working day
Resolution targetNext sprint
Go-live tolerance≤ 10 (FAT exit)
P4 — Low

Cosmetic or minor

Typos, alignment, label inconsistencies. Backlog candidates.

ResponseNext planning
Resolution targetBacklog
Go-live tolerance≤ 15 (FAT exit)

Defect lifecycle

Raised
Classified
Assigned
Resolved
Retested
Closed
/
Reopened

Go-live rule: zero P1; zero P2 unless a workaround is agreed and documented; P3 within published ceiling, with post-go-live fix plan; P4 in backlog. P2 workarounds must be agreed by the Design Authority and Process Owner and recorded in the Test Exit Report — no grey area.

A change made to fix or add one thing can quietly break something that already worked. Regression testing re-runs tests that previously passed to catch exactly that.

What’s the regression pack?
  • A curated set of previously-passed tests covering the critical end-to-end business processes.
  • Built during SAT by the SI Test Lead — incrementally, not as a final task.
  • Carried forward through every later phase, updated with every fix, growing as the system grows.

Regression strategy across the pipeline

Regression is not a phase — it runs continuously across the pipeline, with different intensity at different stages. Risk-driven, not ceremonial. The regression pack is a permanent operational capability that runs through every release, patch and config change for the life of the platform — not a Build-phase activity that ends at go-live.

Per-phase scope and ownership

Same discipline, different intensity. Every phase answers the same three questions: what triggers regression, what gets re-run, and what shape the re-run takes. The shape varies by design — sprint loops in Build, three cycles in SAT and SIT, daily targeted runs in UAT, single proven passes at Pre-UAT and BAT, and risk-based runs for the life of the platform.

Phase Regression scope Owner
Build (S13) — per sprintRegression here is the sprint loop itself: each sprint re-runs the passed tests from previous sprints alongside FAT on the new stories (dev/test environment). Smoke-test for tightly-scoped UI tweaks; full re-run for changes to shared logic. No separate cycle structure — the sprint cadence is the cycle.SI Tester / SI Test Team
SAT (S13/14 boundary)SAT establishes the regression pack (SI test environment). Three cycles plus a sanity test — defects found in cycle one are fixed and re-proven in cycles two and three; the sanity run confirms the final state. Pack is the deliverable carried forward.SI Test Lead
SIT (S14)SAT regression pack re-run with third parties connected (integrated test environment) — same three-cycle shape as SAT, usually compressed: full pass, prove the fixes, prove convergence. Catches regression introduced specifically by integration changes.Client Test Manager (owns) + SI Test Lead
Pre-UAT (S14)Single spot-check pass on the UAT environment to confirm parity with SIT. Not a phase in its own right — a readiness confirmation before users arrive.Client Test Manager
UAT (S14)Daily targeted regression on the areas touched by that day’s defect fixes (UAT environment) — a short, impact-based cycle, not the full pack. Where the pack is automated it runs overnight in addition; full manual pack re-runs happen at cycle boundaries and exit, not nightly.Client Test Manager + Defect Manager
BAT (S14)One proven pass of the pack at BAT entry (production-shaped data environment) — confirming the system the Benefit Owners and Executive Sponsor are assessing still passes everything it ever passed.Client Test Manager
Post-go-live (S17 onward)Risk-based for the life of the platform: the automated pack runs against every patch and config change (production); a full run — automated core plus manual critical processes — before each platform update (e.g. D365 release waves). Permanent capability with a named owner — not a Build-phase activity that ends at go-live.IT Operations + Platform Owner

Sign-off is non-transferable. Regression-pack sign-offs at each phase are signed by the named owner only. If the named role is unavailable on the day, the sign-off waits — it does not delegate down. Substituted sign-off on regression cycles is a common cause of post-go-live disputes about whether a fix was actually validated against the wider pack.

How a test phase actually runs

Three cycles, two weeks (SAT and SIT). NFT runs in parallel alongside them on its own track — same three-cycle shape (test → prove fixes → prove convergence) applied to performance, security and resilience, converging on the same exit.

Cycle 1 — the testing itself. First execution of every script in scope. This is not regression — nothing has been run before. Defects raised, fixes flowing. (Days 1–5)
Cycle 2 — prove the fixes. Re-run the failures and the areas the fixes touched. Regression starts here: confirming fixes work and broke nothing that passed in cycle 1. (Days 6–8)
Cycle 3 — prove convergence. Remaining failures re-run; defect arrivals falling toward zero. Closes with one full proven pass — the evidence the exit signature rests on. (Days 9–10)
Underneath all three: the automated pack runs nightly throughout — the machine works the night shift, so the days stay focused on new ground and fixes.
Two-week phase — SAT / SIT
1
2
3
4
5
6
7
8
9
10
Cycle 1 · 5d
Cycle 2 · 3d
Cycle 3 · 2d
automated pack — nightly, throughout
NFT ∥ — same cycle shape, own track
Day 10 · Exit run → sign-off

UAT runs differently: users execute daily, targeted regression follows the day’s fixes, one proven run before exit — no formal cycle structure.

Why three cycles — and why it doesn’t cost a month.

Cycle 1 finds the defects. Cycle 2 proves the fixes — and that they broke nothing else. Cycle 3 proves the trend: defect arrivals falling toward zero. Exit on evidence of convergence, not a snapshot.

The cycles interleave with the phase’s normal execution — plan roughly two weeks inclusive for the phase, scaling with team and pack size.

What it buys: a defect contained at SAT is fixed SI-side at SI cost. The same defect in UAT burns business-tester time and confidence; in live, it’s an incident. Each escape is roughly ten times dearer. Common mistake: cutting to one cycle to save a week — the unproven fixes surface in the next phase and cost the week back tenfold.

Manual vs automated regression

Manual regression

  • A person executes previously-passed scripts step by step in the test management tool (Azure Test Plans on a D365 programme), recording pass/fail as they go.
  • The right tool where human judgement and sign-off are needed — which is why UAT and BAT regression stay manual.
  • Executed by the test team — on most programmes the SI functional consultants double as the SI test team; business testers, coordinated by the Client Test Manager, run UAT-phase cycles.
  • The SI Test Lead and Client Test Manager own the pack and the sign-off, per the RACI — they don’t execute it personally.
  • Slow and costly to repeat — a manual “nightly full regression” by consultants who’ve spent the day fixing defects doesn’t survive week one.

Automated regression

  • The same scripts coded once, executed by a tool — scheduled overnight or triggered when a fix lands; results waiting in the morning, unattended.
  • The practical case for automating the SAT pack: the only regression free of contracted-hours constraints.
  • D365 examples: RSAT for Finance & Operations, Playwright or similar for CE, orchestrated through Azure DevOps pipelines alongside the defect log. Platform-agnostic: Selenium, Tosca, Leapwork.
  • The tool matters less than the discipline: a named owner, a maintained pack, automation aimed at the scripts re-run constantly.

Regression rhythm in practice

Three layers, three clocks — they run together, not instead of each other:

Automated pack — every night, throughout. Runs unattended via the test tooling (RSAT for D365 F&O, Playwright for CE, or equivalents), results by morning.
Manual, targeted — daily, during the working day. Short cycle on the areas yesterday’s fixes touched. Rostered: the same consultants are fixing defects, so the roster covers both.
Full manual pack — at the close of each test cycle, and at exit. Every test in the pack, run end to end by people, so the next cycle starts on a proven system and the exit signature rests on evidence. Too costly to run nightly — which is why the other two layers exist.

Common trap: consultants self-verifying their own fixes with no independent check — the Defect Manager’s evidence trail and client-side sign-off are the control. And regression cover beyond the contracted day is bought, not assumed — set it in the SOW, not in hope.

From test exit to cutover

Common mistakes

Regression effort estimation (D365 — practitioner rules of thumb)

A starter-for-ten on sizing regression effort against build effort. Regression test effort isn’t a fixed ratio — it varies sharply by build artefact type. The figures below are honest practitioner estimates based on long experience with D365 implementations, not vendor-published numbers. Use as a starting point for sanity-checking your own programme’s test plan, not as benchmarks.

Build artefact type Regression test effort (per build day)
Pure configuration (workflows, roles, dimensions)0.5 – 1.0 day
Extensions (per-tenant or per-app)2.0 – 2.5 days
Integrations (Dual Write, Logic Apps, custom)2.5 – 3.0 days
Data migration logic1.5 – 2.0 days
Reports & Power BI0.5 – 1.0 day
Blended (typical D365 programme)1.5 – 2.0 days regression per build day

Blended figure assumes a typical D365 mix: ~50% configuration, ~25% extensions, ~15% integrations, ~10% other. Heavier customisation pushes the blended ratio toward 2.5; near-out-of-the-box implementations pull it down toward 1.0.

Caveats worth being explicit about

Use as a sanity-check on your own test plan sizing — not as a benchmark.

Common pitfalls — call these out the moment you see them

Go-live gate checklist

Tick items as evidence is in. The sponsor signs the final authorisation only when all are green.

Readiness

0%
Not ready

Start with the test-completion sign-offs (FAT, SAT, SIT, UAT, BAT). Each is non-negotiable.

Level Finder

Describe what's being tested and who's doing it. The finder maps it to the right level.

Type a description above. Try phrases like: "checking field validation", "load test 5000 concurrent users", "end-to-end order to cash with 3rd party", "process owner running through case creation in sprint review".