Good technology metrics are rare. Most organizations track operational metrics (is the system up? are tickets being closed?) without connecting them to business outcomes. The people building and maintaining technology don't know whether their work is delivering value, and the business stakeholders evaluating technology investments don't have meaningful data to work with.
Setting up the right metrics before a technology system goes live is a prerequisite for measuring whether the investment was worth it. This guide covers how to identify which metrics matter, how to establish a pre-deployment baseline, and how to structure the measurement system so the data remains useful over time.
Step 1: Start With Business Questions, Not Technical Metrics
The common mistake is to start with what's easy to measure: system uptime, API response time, error rate. These are real metrics worth tracking for operational purposes. They become a problem when they're treated as proxies for business value.
Start instead with the business questions the technology investment is supposed to answer:
- Will this reduce the time our team spends on manual data entry?
- Will this reduce the number of customer complaints related to order errors?
- Will this help us identify pricing opportunities we're currently missing?
- Will this let us onboard new customers faster?
Each business question maps to one or more measurable outcomes. Working backward from the question to the metric ensures the metrics you track are the ones that actually matter. A metric without a business question attached is an answer to a question nobody asked.
Step 2: Identify What Needs to Be True to Measure the Outcome
For each business question, define what data you need to answer it:
What is the unit of measurement? Hours, dollars, count, percentage, days. Be specific. "Reduce order processing time" is a direction; "reduce average order processing time measured in hours from order received to order shipped" is a metric.
What is the data source? Where does this data live today? Is it in a system, a spreadsheet, a report someone generates manually, or does it not exist and need to be created? If the data doesn't exist yet, who is responsible for starting to capture it before the new system goes live?
What is the measurement cadence? Daily, weekly, monthly, or by transaction? More frequent measurement catches problems faster but creates more data management work. Choose a cadence that matches how quickly the metric is expected to change.
Who is responsible for reporting it? Metrics without an owner get stale. Assign a specific person or role responsible for pulling, verifying, and reporting each metric on the defined cadence.
Photo by www.kaboompics.com on Pexels
Step 3: Establish a Pre-Deployment Baseline
You can't measure improvement if you don't know where you started. The baseline is the current state of each metric before the new technology is deployed.
Collecting baseline data requires starting measurement before the project is finished, not after. A common failure pattern: the team is focused on building, nobody thinks about baseline measurement until close to go-live, and then baseline data is reconstructed from imperfect sources or estimated. Reconstructed baselines are weaker than measured ones because they rely on memory and potentially incomplete records.
For time-based metrics: Track a representative sample of the current process for at least 2-4 weeks before the new system goes live. The sample should cover normal variations in volume and complexity, not just typical cases.
For count-based metrics (error rates, ticket volumes): Pull historical data from existing systems. Most support tools and operational databases have enough history to establish a 90-day or 6-month baseline. Note any anomalies or seasonal effects that might distort the baseline.
For cost-based metrics: Gather actual cost data from finance or accounting. Staff time costs should use a fully-loaded rate (salary plus benefits plus overhead), not just base salary.
For business outcome metrics (revenue, margin, retention): These come from business systems and typically have longer historical records. The challenge is establishing what portion of the outcome metric is influenced by the technology investment vs. other factors. Document the attribution logic explicitly at baseline, so you're applying the same logic after deployment.
Step 4: Define Leading and Lagging Indicators
Key performance indicators can be either leading indicators (early signals of whether the system is working) or lagging indicators (confirmation of the final outcome after enough time has passed).
Leading indicators are closer to the technology behavior and change quickly:
- User adoption rate (what percentage of intended users are actively using the system?)
- Feature utilization (are users using the features that deliver the expected value?)
- Data quality scores (is the system producing clean outputs or are there validation errors?)
- Error rate in the new system vs. the old process
Lagging indicators are the actual business outcomes and take longer to materialize:
- Cost reduction measured in dollars over a quarter
- Time savings measured over a sustained period (not just the first week, when novelty effects inflate usage)
- Customer outcome improvements (satisfaction, retention, order accuracy)
Track both. Leading indicators tell you whether the system is being adopted correctly. Lagging indicators tell you whether the adoption is producing the business outcomes that justified the investment. If leading indicators look good but lagging indicators don't materialize, the causal model between system usage and business outcome needs to be revisited.
Step 5: Build the Measurement Infrastructure Before Go-Live
Data that isn't automatically captured requires someone to manually collect it, which means it will be inconsistent at best and missing at worst. Build the measurement infrastructure as part of the deployment, not as an afterthought.
Automated reporting from source systems. Most operational platforms (support tools, CRMs, ERPs, cloud infrastructure) have built-in reporting or API access. Set up automated exports or scheduled reports that feed your tracking system before the new system goes live.
Event logging in custom-built systems. If your organization is building custom software, build structured logging into the application from the start. Log the events that matter for your business metrics: transaction completed, error occurred, user action taken. A structured log is much easier to analyze than unstructured output.
Integration with business intelligence tools. If your organization uses a BI tool, connect the new system's data to it before go-live. Waiting until after deployment means data accumulates somewhere inaccessible while the measurement conversation gets deferred.
Python is widely used for the data extraction and transformation work that feeds measurement systems. Node.js is common when the source systems are web APIs. The specific language matters less than having a consistent, automated pipeline that doesn't depend on someone remembering to run a manual export.
Step 6: Define the Review Cadence Before Launch
The measurement system exists to enable decisions. That only happens if someone reviews the data on a regular schedule and is empowered to act on what they see.
Before the system goes live, schedule the first three post-deployment reviews:
30-day review: Is the system being adopted? Are leading indicators trending in the right direction? Are there early signs of problems that need to be addressed?
90-day review: Are the metrics stabilizing? Is the trajectory consistent with the projected benefit realization timeline? Are the costs tracking against projections?
6-month review: Are the lagging indicators materializing? How does actual performance compare to the projections from the ROI model? What does this tell us about how to model future investments?
These scheduled reviews create accountability for the estimates made during planning and build the organizational habit of treating technology investment as something that requires measurement, not just implementation.
The full framework for measuring technology return on investment, including how to define success before the project starts and how to communicate results to business stakeholders, is covered in the article on How to Measure the ROI of Business Technology Investments.
137Foundry works with organizations on technology strategy and implementation, including helping teams build the measurement discipline that makes technology investments accountable.
