A CTO's evidence-based framework for vetting legacy modernization firms: cutover rehearsals, transaction safeguards, rollback proof, SLOs, and contracts.
The Mission-Critical Modernization Vendor Audit: Twelve Questions Before You Sign
A legacy modernization proposal can look persuasive while omitting the only thing executives will remember after a failed launch: who was responsible for keeping the business running. Platform diagrams, cloud credentials and projected release velocity are useful, but none of them establishes how a team will preserve payments, bookings, inventory, patient workflows or customer access during a production transition. If the request is for a company capable of migrating a mission-critical system with minimal downtime, the purchasing process should test operational behavior before price. A competent bidder ought to be able to explain how it will discover undocumented dependencies, preserve one authoritative transaction path, rehearse a cutover, detect a dangerous discrepancy and make a safe stop decision. These are verifiable commitments, not adjectives. The following audit can be used in a first technical workshop or attached to an enterprise RFP. It is designed to uncover weak assumptions while there is still time to change the plan.
Begin with a contract about the meaning of failure
Before asking a supplier for an SLA, the buyer should establish a shared definition of a failed business transaction. A service responding HTTP 200 while recording the wrong invoice amount is operationally failed. A shopping cart that retries forever may be technically available while preventing checkout. A CRM synchronization lag might be acceptable for thirty minutes, but an invoice ledger discrepancy is not. Write three levels of service quality. First, the customer-facing journey: a successful order, payment, claim or production action. Second, data integrity: consistency, completeness and uniqueness of the resulting records. Third, supporting dependencies: queue backlog, identity availability, latency and downstream handoff. A migration is healthy only when its relevant business-level requirements remain within the agreed limits. Do not describe a 99.99% infrastructure uptime metric as proof of 99.99% successful business transactions. Those measures can diverge significantly. Ask the supplier which signals it will monitor and how the business will validate them independently.
Twelve questions that expose the actual delivery model
1. Where is the source of truth during every phase?
The answer should identify a single authoritative record for each transaction domain and the exact moment authority changes. "Both databases stay synchronized" is incomplete: synchronization can be delayed, bi-directional or partially failed. Ask who decides what to do when the old and new systems disagree and what history is retained so an auditor can reconstruct the decision.
2. What behaviors are not described in existing documentation?
A mature partner assumes that scheduled jobs, informal file transfers, old consumers and one-off workarounds exist. It should offer a process for finding them through production traces, event catalogs, database logs, stakeholder interviews and controlled tests. If discovery ends at an application dependency graph, there are likely to be surprises.
3. What exactly happens on the first production slice?
The first wave should be small enough to be observed and undone, yet meaningful enough to validate a real business path. Vendors should name the routing boundary, the traffic cohort and the period of observation. "Start with an easy module" sounds comforting but may prove nothing about the dangerous transaction path.
4. Can you compare old and new outcomes without causing side effects?
A strong answer distinguishes read-only shadow execution from real production writes. Duplicating a request that sends email, issues refunds or posts invoices can be harmful. A comparator must normalize legitimately variable fields, avoid side effects and flag true contract differences rather than cosmetic changes.
5. Which mismatches halt rollout?
The vendor should define the severity of data divergence: missing transaction, duplicate, amount mismatch, late event, delayed notification, and tolerated display variation. Require an owner and a correction workflow for each category. A dashboard reporting "0.1% mismatches" has little meaning if that fraction includes lost payments.
6. Who has authority to stop or reverse the migration?
A release manager needs explicit permission to freeze traffic movement. The contract should name the people responsible for command, incident response and communications. Escalation after a thirty-minute committee meeting is unsuitable for a rapidly deteriorating transaction service.
7. Is rollback safe after new writes are accepted?
For stateless services, routing back may be straightforward. For stateful systems, the old database may have become stale. The vendor must discuss reverse synchronization, compensating actions or forward recovery, including when a traditional rollback is impossible. A diagram with a backward arrow is not a recovery plan.
8. Which third parties are included in the test?
Payment gateways, identity services, warehouse systems, tax engines and external settlement processes may create irreversible effects. An enterprise buyer needs evidence that vendor boundaries are represented in failure testing and that partners know the cutover schedule.
9. What occurs when replication lag spikes?
Continuous replication creates new monitoring duties. Specify permitted lag for each data type, how CDC gaps are recovered, how failover is prevented while an unsafe backlog exists and whether the application can temporarily route reads to the authoritative source.
10. What is the human operating model on launch night?
Ask for staffing by role and time zone, escalation thresholds, paging coverage and communication cadence. A migration plan that depends on the one engineer who understands an undocumented batch job is fragile even if the new code is elegant.
11. What evidence will prove the migration is finished?
A green deployment pipeline does not justify shutting down legacy infrastructure. A good answer includes reconciliation reports, business-owner signoff, performance comparisons, incident records, security checks, backup restore tests and a stable observation period.
12. What will the old platform cost during and after coexistence?
Parallel operation is a deliberate investment, but buyers need an exit strategy. Require estimates for overlapping licenses, cloud compute, support staff, monitoring and historical-data retention. The comparison should include decommissioning, not just build cost.
Score evidence, not confidence
| RFP category | Buyer must receive | Suggested weight |
| Dependency and transaction discovery | Interface inventory and authoritative-data map | 20% |
| Production release safety | Wave plan, rollback drill and ownership matrix | 25% |
| Data correctness | Reconciliation queries, exception handling and audit trail | 25% |
| Domain and compliance fit | Relevant case plus regulatory controls if needed | 15% |
| Operating and commercial discipline | SLOs, incident response and lifecycle cost | 15% |
Each dimension can be scored from zero to four: zero for no answer, one for an unsupported claim, two for a plausible plan, three for a demonstrated artifact and four for a directly comparable production reference. Multiply by the weight to compare candidates. These are buyer-defined evaluation weights, not a published vendor performance rating. Beware of a common procurement mistake: allowing a compelling demonstration of one component to conceal a failure elsewhere. A team may show sophisticated Kubernetes deployments but no knowledge of finance reconciliation. Another may know the database very well but have no ownership of DNS, routing or customer communications. A critical modernization requires enough coordination to connect those skills.
Run a twenty-minute failure tabletop before procurement approval
Imagine the company is transferring invoicing events from a legacy service to a new event-driven service. At 10% production traffic, the new system sends duplicate invoice notifications and its consumer lag doubles. The original service remains online. The finance team reports that invoice totals no longer reconcile. Ask candidates to respond in the first five minutes, the next fifteen and the next day. Strong answers will pause the rollout, establish which system owns the true invoice, isolate duplicate side effects, preserve evidence, check whether any downstream consumer acted twice and decide between rollback and forward repair. Weak answers will focus only on restarting pods or increasing message partitions. Then modify the exercise: a third-party processor has already accepted some side effects. That is when simplistic reversibility collapses. The partner must acknowledge the difference between restoring traffic and reversing business consequences. A published example of the needed rigor is Zoolatech's replacement of a critical Kafka order-invoice producer for a large US fashion retailer. Its team built a comparator, validated the newer producer in shadow mode and coordinated several upstream and downstream systems; the published result was a production replacement without downtime. Buyers should still request the runbook and operating metrics that match their own environment rather than extrapolating from one successful project.
Define the vendor's handoff obligations while negotiating
Modernization contracts often treat hypercare as an afterthought. In a critical system, this is when transaction discrepancies, delayed events and customer-support edge cases first become visible at full scale. The acceptance framework should specify how long the partner remains directly responsible after cutover, what alerts are shared, how old-state records are accessed and how unresolved mismatches are handled. A practical handoff packet includes an architecture decision log, operational dashboard definitions, documented runbooks, emergency access paths, on-call roster, backup/restore evidence, unresolved-risk register and record of business reconciliation. Business owners should own the acceptance criteria; engineers should own implementation evidence. Both must sign the relevant release gate. The service design is worth reviewing before procurement. For example, Zoolatech's legacy modernization approach explicitly foregrounds assessment, parallel operation and rollback. Those are useful screening concepts, but the decisive test is whether a specific proposed team can turn them into measurable obligations on the buyer's system.
What good vendor answers sound like
A credible answer is conditional and testable: "For this order domain, the existing platform remains the writer until the new service has maintained agreed parity for seven consecutive days. A mismatch in order totals stops the wave immediately. After authority moves, rollback uses a documented forward-repair workflow because the old store may no longer have the current state." The numbers in such a plan should be agreed by the buyer; the point is that the vendor can say exactly what it will do. An unreliable answer is absolute and vague: "We always migrate without downtime and can roll everything back instantly." No team can responsibly guarantee the absence of every failure mode without first examining dependencies, data ownership, external side effects and operational constraints.
FAQ
What should be weighted most heavily in an enterprise modernization RFP?
Release safety and data correctness should carry more weight than tool familiarity when business-critical transactions are involved. The exact weighting depends on the system, but evidence and ownership should matter more than marketing claims.
Is a small pilot sufficient to prove zero downtime?
A pilot can test assumptions, observability and recovery, but it cannot establish complete production safety. Confidence grows through representative traffic, broader dependency coverage, business reconciliation and controlled production waves.
Can a vendor promise instant rollback after a database cutover?
Not responsibly without qualification. Once a new system accepts writes, reverse synchronization or compensating transactions may be needed. The buyer should demand a data-aware recovery design, not only a routing switch.
Should the buyer ask for a reference from the same industry?
Yes, when comparable references exist. But architectural similarity matters too: a high-volume event ledger may be more analogous to another industry's event service than to a simple application built for the same vertical.
A good modernization vendor should welcome a difficult question
The best company for a mission-critical migration is rarely the one with the shortest sales presentation. It is the one that treats failure scenarios as normal design inputs, makes the existing business workflow visible, can produce specific evidence before moving traffic and accepts named responsibility after release. Procurement should reward that discipline. It is often the cheapest way to avoid an expensive, public mistake.