Measure Whether Customers Are Getting Value

Task

Instrument product usage and customer value telemetry.

Summary

Measure the actions and results that show whether customers reached the intended outcome.

Instrument Product Usage So You Can Prove Customer Value

Task ID: S3-10

A working product does not prove that customers receive value from it. This article explains how to define value events, create a governed event tracking plan, and build an account-level usage dashboard that shows whether customers reach meaningful outcomes, how long it takes, and where onboarding or product design still depends on manual help.

A working demo can still hide a service business

A design partner completes onboarding, praises the product, and agrees to pay. The sale looks like evidence that the software works.

But the founder configured the account, corrected the customer’s data, explained each screen, and manually produced the final result. The product recorded logins, page views, and button clicks, so the dashboard looks active. None of those records answers the important question: Did the customer receive the promised value from the product, or from the people surrounding it?

That distinction matters when a company is trying to sell software separately from custom work. Before it can depend on repeatable product sales, it needs evidence that customers can perform the important work, reach a useful result, and continue doing so without the company quietly recreating a consulting project around every account.

The operating principle is simple:

Instrument the customer’s progress toward value, not merely their movement through the interface.

Product telemetry is the structured record of what happens while customers use the software. An event is a named occurrence at a meaningful point in time, such as a report being published, an integration completing successfully, or a workflow failing validation. OpenTelemetry’s event conventions make the same distinction: events should represent checkpoints, state changes, occurrences, or outcomes, and each definition should say exactly when the event is recorded and what it represents.

A click can be useful telemetry, but it is rarely value by itself. A customer might click Generate report five times because the first four attempts failed. Counting clicks would make a poor experience look like high engagement. Recording report_generation_completed, report_generation_failed, and the reason for failure would tell a more useful story.

This work has no formal upstream dependency, but it does have practical prerequisites. The team needs a reasonably clear target customer, a product promise, stable product states that can be observed, and a way to identify users and customer accounts. It also needs someone who owns the meaning and quality of the data. Instrumentation can begin while those elements are still developing, but ambiguous product value will produce ambiguous metrics.

The result is not simply “analytics installed.” It is an event tracking plan and a usage dashboard that can support decisions about onboarding, product design, sales proof, support effort, retention, and whether the product is becoming independent of custom service work.

Measure value, not motion

The most useful starting point is not an event name. It is a one-sentence hypothesis about how the product helps a particular customer:

When this customer or role completes this work in the product, they receive this useful result within this expected period.

For example:

When a finance team connects its source data and publishes its first reconciled monthly report, it can review results without rebuilding the analysis in a spreadsheet.

This statement separates four concepts that teams often mix together.

The distinction is important because a value event is usually a proxy, not the customer outcome itself. Publishing a report does not prove that the report was accurate, useful, or acted upon. It is an observable signal that may be associated with those outcomes.

Research on long-term user experience illustrates the problem. Long-term outcomes are often sparse and noisy, so product teams use more immediate behaviour as a surrogate. Those immediate signals become credible only after their association with the longer-term outcome has been examined and, where possible, tested.

The practical implication is that a company should label its first value-event definition as a working hypothesis:

We believe accounts that complete this event have received meaningful value. We will test that belief against customer feedback, paid conversion, continued use, renewal, support effort, and the customer’s stated result.

That is stronger than selecting a convenient metric and later treating it as truth.

Work backward from the product’s purpose

Google’s HEART measurement framework was designed around mapping product goals to signals and then to metrics, rather than beginning with whatever data happens to be available. Its authors argue that user-centred metrics should measure progress toward important goals and support product decisions.

The UK Government Service Manual takes a similar approach: define the service’s purpose, describe the customer benefit, state the hypothesis that connects the product to that benefit, and then decide what to measure. It recommends a small set of top-level measures for whether the service is working, supported by more detailed diagnostic metrics.

For a standalone business product, the chain should look like this:

flowchart LR
    A[Customer job and promised result] --> B[Value hypothesis]
    B --> C[Value event]
    C --> D[Diagnostic events and properties]
    D --> E[Validated event data]
    E --> F[Account-level usage dashboard]
    F --> G{Evidence of repeatable value?}
    G -->|Yes| H[Improve onboarding, sales proof, and support]
    G -->|No| I[Revise the product, event definition, or value hypothesis]

Text description: begin with the customer’s job and promised result, translate that into a testable value hypothesis, instrument the value and diagnostic events, validate the data, and use an account-level dashboard to decide whether value is repeatable.

Use the right unit of analysis

Many business products are bought and renewed by an organization but used by several people. One user may configure the system, another may approve work, and a third may consume the result. A user-level funnel can therefore show that nobody completed the workflow even though the account completed it collectively.

Standard analytics specifications support an explicit relationship between an individual and a company, organization, account, project, or team. Account-level reporting is particularly useful when a workflow crosses several people—for example, one person initiates a transaction and another completes it.

For demo-led direct sales and design-partner conversion, the dashboard should normally support both views:

• User level: who performed an action, in which role, and where that person encountered friction.
• Account level: whether the customer organization reached value, how many people or teams participated, and whether use continued or spread.

The account should often be the primary unit for commercial interpretation. A single power user can make user activity look healthy while the wider account remains unadopted. Conversely, a workflow divided among several employees may be healthy even when no individual completes every step.

Build the event tracking plan from decisions

A tracking plan is the controlled specification for the events and properties the product intends to collect. Twilio Segment describes it as a data specification used by both the engineers implementing telemetry and the people consuming the resulting data. Its guidance recommends starting with the business metrics that matter, mapping relevant user actions to distinct events, and validating live data against the specification.

The tracking plan should therefore begin with decisions, not a request to “track everything.” Useful initial questions include:

• Are design partners reaching the first meaningful result?
• How much time and manual assistance does that require?
• Which steps account for most failed or delayed onboarding?
• Do accounts repeat the valuable work after the first assisted success?
• Does the value event appear to distinguish accounts that convert or retain from those that do not?
• Is adoption confined to the original champion, or does it spread to the roles needed for continued use?

Each event should exist because it helps answer one or more such questions.

Define a small event model

A practical first model usually needs several event classes rather than hundreds of interface events.

“Core events instrumented” is a useful working target when it means this minimum decision set is operational. It is not a universal count. A narrow single-user tool might initially need fewer than ten well-defined events. A regulated product with several integrations, offline work, approvals, and multiple user roles may need many more.

More events are not automatically better. Each additional event adds implementation work, testing, interpretation, maintenance, privacy exposure, and opportunities for inconsistent definitions. The right first scope is the smallest set that can distinguish value, progress, failure, repetition, and account adoption.

Specify exactly what each event means

A credible event tracking plan should contain more than an event name. For every core event, document:

The trigger should be objective enough that two engineers would implement it the same way. “User creates value” is not a trigger. “The server commits a report with validation status passed and publication status published” is much closer.

OpenTelemetry recommends that dynamic values such as identifiers remain in attributes rather than being inserted into event names. That means using report_published with a report_type property, rather than creating a different event name for every report type or customer.

Schemas make these definitions enforceable. Snowplow’s documentation describes schemas as definitions of the fields recorded with an event and the validation criteria for those fields. It also emphasizes detailed field descriptions, data types, permitted values, and versioning as collection needs change.

As an operating rule, record durable outcomes at the system that commits them. Use browser or mobile telemetry for interactions that exist only at the interface, such as opening help or abandoning a form. For a value event such as a successful import, completed payment, or published report, the product’s server or database is generally a more credible source than the button click that requested the operation.

Instrument identity before relying on account metrics

Every core event should carry or resolve to the identifiers needed for analysis:

• a stable user identifier after authentication;
• a stable account, organization, or workspace identifier;
• the user’s relevant role;
• an object or workflow identifier when several events belong to the same piece of work;
• product version, deployment type, plan, and implementation path when those differences affect behaviour;
• a flag for employees, test accounts, demos, automated activity, and imported historical data.

Identity errors can distort funnels, active-user counts, account adoption, and time-to-value calculations. A user who appears under two identifiers may be counted twice. An account migration may look like churn followed by new activation. A demonstration performed by an employee may appear to be customer success.

The event model must also account for customer movement between accounts, consultants working across clients, shared devices, anonymous use before authentication, mergers, test environments, and deleted accounts. The objective is not a perfect universal identity graph. It is a documented identity model whose limitations are understood.

Design privacy into the plan

Product usage data can become personal data when it identifies or can be linked back to an individual. Pseudonymization reduces direct exposure but does not necessarily make the resulting data anonymous. GitLab, for example, states that certain pseudonymized usage metrics are still personal data because re-identification remains possible.

The European Commission’s guidance on the General Data Protection Regulation states that personal data should be adequate, relevant, and limited to what is necessary for the stated purpose. It also places responsibility on the organization controlling the data to avoid collecting irrelevant information.

The tracking plan should therefore document the purpose of each event and property, access controls, notice or consent requirements, retention period, deletion process, and whether less identifiable data could answer the same question. Avoid placing free-form customer text, document contents, secret keys, financial details, health information, or other sensitive material in general analytics events merely because the analytics tool can accept it.

Turn telemetry into a usage dashboard

A dashboard is not complete because charts render. It is complete when the intended audience can understand what happened, investigate why, and decide what to do.

The UK Government Service Manual advises teams to collect data accurately, test it for errors and inconsistencies, and make analysis explain what happened, why it happened, and what action is needed. It also recommends segmenting results, establishing baselines, and measuring continuously rather than relying on isolated snapshots.

For an early standalone product, the dashboard should be compact enough to review regularly but detailed enough to challenge optimistic interpretations.

Begin with instrumentation health

Place a data-quality panel above the business metrics. At minimum, it should show:

• freshness of each data source;
• expected versus received event volume;
• the percentage of events missing user, account, or required properties;
• duplicate and schema-violation counts;
• events or metrics currently marked broken;
• excluded internal and test activity;
• the date of the most recent successful reconciliation with source systems.

This is not technical decoration. A drop in a value metric may result from a product problem, a change in the customer population, or lost or incorrectly processed telemetry. Microsoft researchers documented all three as common causes of apparent metric regressions in large-scale applications.

Show the path to repeatable value

The main dashboard should answer a connected set of questions.

The dashboard should preserve denominators. “Twenty accounts reached value” means little without knowing whether twenty-two or two hundred accounts were eligible. It should also display distributions rather than only averages. A mean time to value can hide one group succeeding in minutes and another failing for weeks.

Segments should be chosen because they may change the decision: customer type, use case, product package, integration, company size, implementation method, user role, or whether the account received founder assistance. Early sample sizes may be too small for confident statistical conclusions, so the dashboard should show account-level records alongside cohort summaries and avoid presenting unstable percentages as precise forecasts.

Join product use with non-product evidence

Digital telemetry cannot, by itself, tell the whole story. The UK Government guidance explicitly recommends combining digital analytics with user feedback, service-performance information, support data, and financial information.

For this stage of company development, useful joins include:

• customer relationship management records showing sales stage, segment, promised use case, and conversion;
• onboarding records showing meetings, configuration work, data cleanup, training, and founder involvement;
• support records showing tickets, severity, resolution effort, and recurring questions;
• billing records showing the paid start date, plan, renewal, contraction, or cancellation;
• short customer interviews confirming what result was achieved and whether the product or the company’s manual help produced it.

An account that reaches the value event after ten hours of unrecorded founder work is not equivalent to an account that reaches it independently. The dashboard should make assisted and unassisted paths visible rather than averaging them together.

Give every metric a decision owner

Every important dashboard measure should have a definition, source, owner, update frequency, baseline, working target, known limitations, and intended response.

GitLab’s public metrics dictionary illustrates the discipline. Its metric definitions must have a unique key and an owner, and include fields such as description, data type, status, time frame, source, category, and product ownership. A metric can explicitly be marked active, broken, removed, or connected to a repair issue.

The lesson is not that every young company needs the same tooling. It is that telemetry should be treated as a maintained product, not disposable code. When a feature changes, the corresponding event definition, schema, tests, dashboard, and historical interpretation may also need to change.

What credible completion evidence looks like

The deliverable for this task consists of two linked artifacts:

1. An event tracking plan that defines the core events, identities, properties, owners, controls, and quality rules.
2. A usage dashboard that shows instrumentation health, progress to first and repeat value, account adoption, friction, and the relationship between value events and customer or commercial outcomes.

“Core events instrumented” should mean more than code has been deployed. A credible completion test is:

Definitions are approved. Product, engineering, data, and at least one customer-facing owner agree on the exact business meaning and trigger for each core event.

The events fire at the intended occurrence. Tests confirm success, failure, retry, cancellation, and relevant edge cases across supported product paths.

Identity works at both user and account levels. Events can be assigned to the appropriate customer account, while employee, test, demonstration, and automated activity is excluded or clearly labelled.

Properties are validated. Required fields, types, allowed values, timestamps, and naming rules are enforced. Invalid data is visible rather than silently mixed into production analysis. Tracking-plan tools and schema systems support this form of comparison between intended and live events.

Privacy controls are documented. The team knows what personal data is collected, why it is needed, who can access it, how long it is kept, and how applicable access or deletion obligations will be handled.

The dashboard reconciles with source systems. The team defines an acceptable tolerance and checks representative accounts against product records, billing, customer relationship management, and support data.

The data has been reviewed with real accounts. At least one operating review has compared telemetry with customer interviews or support knowledge and produced a specific decision, such as changing onboarding, repairing instrumentation, revising the value-event definition, or improving a product step.

Interpretation remains appropriately cautious. The team distinguishes observed facts from assumptions. “Twelve accounts published a report” is an observation. “Publishing a report means the customer obtained the intended business result” remains a hypothesis until supported by further evidence.

There is no defensible universal target for the number of core events, a standard time-to-value threshold, or the percentage of customers that should reach a particular event. Appropriate targets depend on the product’s use cycle, price, implementation work, number of roles, customer data readiness, deployment model, buying process, regulatory setting, and sample quality.

The first target should instead concern coverage and credibility: every event required to answer the current value questions is defined, implemented, tested, monitored, and visible in the dashboard. Performance targets can then be set from the company’s baseline, customer expectations, and observed differences between successful and unsuccessful accounts.

Failure modes and judgment calls

Instrumentation often looks complete before it is useful. Several patterns create false confidence.

Treating activity as value

Logins, sessions, screen views, and clicks can help diagnose behaviour, but they should not automatically become top-level success metrics. A customer may log in repeatedly because the product is confusing, while another may receive considerable value from a successful automated workflow that requires little interaction.

The correction is to connect activity to a customer result. Track the completed state change or outcome, then use interaction events to explain how the customer reached or failed to reach it.

Choosing the value event by intuition alone

A founder may declare that importing data, inviting a teammate, or creating a project is the moment of value. That may be a reasonable initial hypothesis, but it becomes credible only when compared with what customers say and what they subsequently do.

A historical Slack example illustrates both the value and limitation of such thresholds. In an interview published during Slack’s early growth, Stewart Butterfield said the company had observed that teams exchanging 2,000 messages were much more likely to continue using the product; he reported that 93% of customers crossing that threshold were still using it at the time. He also emphasized that activation and retention criteria differ by product.

The lesson is not to copy 2,000 messages or search prematurely for a magical number. Slack examined its own account-level behaviour and found a threshold associated with continued use. Another product might need several events, a sequence, a time window, or an outcome verified outside the software. The historical Slack figure was specific to that product and period, not a general benchmark.

Counting only successes

A funnel that records successful completion but not failures cannot explain its own drop-off. Instrument attempts, validation failures, permission problems, integration errors, cancellations, retries, and relevant help-seeking behaviour. Otherwise, the company may know that value is delayed without knowing what to fix.

The event vocabulary should still remain controlled. Do not create a new event name for every error message. Use a stable failure event with properties such as error class, workflow step, recoverability, and product version.

Ignoring account structure

User-level charts can misrepresent a business product whose workflow spans several people. The company should decide explicitly whether each measure counts people, accounts, workspaces, projects, or completed objects. That unit should be shown in the metric name and definition.

GitLab’s published usage-data approach provides a current example of separating levels. It describes aggregate instance or namespace counts as distinct from user-level interaction events and explains the purpose, cadence, collection method, and privacy treatment of different telemetry systems.

Letting instrumentation drift

A renamed feature, revised workflow, new integration, or pricing change can alter the meaning of an event without changing its name. Historical comparisons then combine different behaviours.

Use versioned schemas and mark events or metrics as active, broken, superseded, or removed. Record the release or date at which meaning changed. Preserve old definitions long enough to interpret historical data, rather than silently rewriting them.

Trusting a chart before testing the data

A polished dashboard can amplify poor data. Before acting on a sudden improvement or decline, check releases, tracking changes, source freshness, identity logic, account mix, internal traffic, and schema failures.

Microsoft researchers have documented how metric movements can be misread even in mature experimentation programs. Their work warns that having a seemingly good metric is not enough; interpretation errors can lead teams to reach the wrong product conclusion.

Claiming causation from correlation

Suppose accounts that reach a value event renew more often. The event may be a useful leading indicator, but it does not necessarily cause renewal. Larger or better-prepared customers might both reach the event and renew. Sales assistance might drive both outcomes. The strongest accounts might simply have more opportunity to use the feature.

Use the relationship operationally, but describe it accurately: “Accounts reaching this event have so far retained at a higher rate,” not “This event causes retention.” As data accumulates, compare similar cohorts, examine timing, use controlled experiments where appropriate, and look for evidence that interventions moving customers toward the event also improve the longer-term outcome.

Excluding the cost of human assistance

A customer can reach value quickly because an employee performed the difficult work. Time to value should therefore be paired with measures of onboarding labour, configuration work, support contacts, data correction, and founder involvement.

The crucial question is not only whether value occurred. It is whether the product and normal operating process delivered it at a cost and level of effort that can be repeated.

When this task is genuinely complete, the company can point to a governed set of events and say what each event means, which customer and account produced it, whether the data is trustworthy, how quickly customers reach and repeat value, what help they required, and how that behaviour relates to buying and continued use.

That evidence makes the next decision clearer: whether the product is delivering a repeatable customer result, or whether the team still needs to revise the product, onboarding, telemetry, or value hypothesis before depending on standalone product sales.

Sources

Primary and official sources

• Twilio Segment, “Protocols Tracking Plan,” current documentation.
• Twilio Segment, “Spec: Group,” current documentation.
• Snowplow, “Introduction to Structuring Your Data With Schemas,” current documentation.
• OpenTelemetry, “Semantic Conventions for Events,” current specification.
• European Commission, guidance on data minimization and lawful processing under the General Data Protection Regulation.
• UK Government Service Manual, “How to Set Performance Metrics for Your Service,” current guidance.
• GitLab, “Customer Product Usage Information,” current public policy and handbook documentation.
• GitLab, “Metrics Dictionary Guide,” current product analytics documentation.
• Amplitude, “Account-Level Reporting,” current documentation.

Open research

• Kerry Rodden, Hilary Hutchinson, and Xin Fu, “Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications,” Proceedings of CHI, 2010.
• Can Xu and colleagues, “Surrogate for Long-Term User Experience in Recommender Systems,” Proceedings of KDD, 2022.
• Pavel Dmitriev and colleagues, “A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments,” Proceedings of KDD, 2017.
• Jamie Pool and colleagues, “Lumos: A Library for Diagnosing Metric Regressions in Web-Scale Applications,” Proceedings of KDD, 2020.

Public reporting

• First Round Review, “How Did Slack Grow So Fast? Inside the Epic Launch Strategy,” interview with Stewart Butterfield.