User Adoption Metrics that Matter

If a system is deployed and no one uses it, does it exist? How can you recognize the full business value of a system if no one is actually using it correctly? User adoption is, at least indirectly, a critical project success metric. I say "indirectly" because while project success (e.g., increase sales by 10%) can be obtained without strong user adoption, user adoption of the system is almost always a necessary condition for project success. But how can you tell if your user group is actually using the system? More importantly, how can you tell whether they are using the system as intended in order to meet the project's business objectives? User adoption metrics should not simply measure who is logging into the new system and how many hours a day they are using it. Such statistics are ultimately meaningless if the system's use does not fulfill some business need that justified the project's existence. After all, the user group may be incentivized in some way to "use" the system, when in fact, they are logging in and blindly clicking through the system without really fulfilling a business need. Thus, you must look not only at the system, but at factors outside the system to get a true feel for user adoption:

1. Find out what the legacy process is. Most IT projects involve designing systems which replace or augment some legacy process. Hence, it is important to show that not only is the new system being used, but that the old process is phased out. In general, if the legacy process that the system solution was designed to replace is still in use, then the project is a failure. What business owner would want to pay to maintain a legacy process, and also incur extra costs to support a solution that was supposed to replace or enhance the legacy process?

2. Look at KPIs. Most business teams report some Key Performance Indicators back to their management. There should be KPIs for both the legacy process and new system process, as well as general KPIs which measure performance of the overall business process (e.g., time to process an order, number of order errors, etc.). KPIs related to the legacy process should show evidence of that process being phased out. For example, if a KPI measures the number of orders processed using the legacy system, this number should be trending downward as number of orders processed using the new system trends upward. If the measure is in time to process an order, then the number of orders used to calculate this metric using the old system should decrease, while the number used to calculate this metric in the new system should increase. The key is that there should be some quantifiable, measurable difference between use of a legacy versus new process. Measuring the use of the new system alone is not sufficient for determining user adoption.

3. Measure project success. A fundamental assumption of every project is (or should be) that the project will result in the satisfaction of some business objective. If the project does not meet the business objective, then it doesn't really matter whether anyone is using the new system. As an example, if the implementation of a project was supposed to result in 10% cost reduction to process an order, and the cost actually increases, then something is wrong somewhere. Either the system is not being used properly or at all, the old system is not being phased out, or some requirement critical to project success was not implemented or captured. Measuring how many people are logging into the system or the number of licenses purchased will not determine whether the system is being used to its full potential.

Finally, when measuring user adoption, it is important to be patient. Some legacy systems and business processes (as well as the people executing or managing them) have been in place for years, sometimes decades. Recognize that at product launch, user adoption may be low, and the project will not be successful right away. In the early stages of product launch, the project may actually look like a total failure. Don't give up! Showing the business value of a project takes time. With the right amount of training, communication, and change management, you should begin to see the project trend towards success.