In part I, we discussed how inadequate attention to problem statement identification could be one of the major causes for the failure of many innovation projects in healthcare. We also discussed the advantages of using design thinking approaches to problem identification, the "D" of the D.A.R.T. model. In part II, we explored the "A" of the D.A.R.T. model, which stands for Assess. We discussed how important it was to converge to the right POV, identify an optimal partner, and most importantly, define both the baseline as well as the success measure, as quantitatively as possible. In this post, we will discuss the "R', which stands for Race.
The Race stage is the "execution" part of the project, the part where typically the solution is tried out for the first time, and a Proof of Concept (POC) is established, or the solution is tried on a pilot study to see how successful the solution is. It must be acknowledged that this is what most people would consider the "meat" of the project; the part where you determine if the solution actually works. And consequently, a lot of importance is given to this stage of the project. However, there are a few important steps in this stage of the project, and most healthcare innovation projects miss a couple of crucial ones.
One could think of 4 key steps in the Race stage of the project: Project planning and management, rapid prototyping, end user testing, and POC outcome evaluation against the baseline. Let's look at each one of these individually. While these 4 steps could apply to any kind of a project, I will be considering one where an external vendor is delivering a solution.
Project planning and management
Almost everybody starts this activity with the utmost zeal; after all, this is the high visibility stage. A project manager is announced (and granted a generous 20% of their time to devote to this activity), typically with a lot of fanfare, and the project manager very efficiently draws up a Gantt chart or some such tool with the appropriate stage-gates, dependencies, etc. All of this is done with due diligence, and the project is kicked off.
Fast-forward a few months, and the situation is quite different. The project hits some timelines, and misses some, but the progress is still slow. The project manager finds that it takes much more than the 20% to keep tabs on this project; in fact, most of their time is spent in mentoring the vendor on domain specificities. Their regular "day-job", as others start to call it is piling up, and they soon realize that they need to prioritize that over this innovation project if they want to have a good rating at the end of the year. By then, someone else has started on another innovation project, very similar to the one they are leading, and suddenly, all the attention is on the new project, making it even less rewarding for the project manager to devote time to this project. And this turns into a vicious cycle, with them giving less time to the project, the project getting slower, and needing more of their attention.
One of the contributing factors to the above situation is that the project manager really does not have a lot to show to the sponsors, for a long period of time. Rapid prototyping is the practice of creating "rough and dirty" models, just to see if what is being worked on aligns with what was thought of in the first place.
It seems almost commonsense that we would want to take a quick look at feasibility through a prototype before we launches into spending the time and money on the real deal. However, most healthcare innovation projects completely miss this step. There are 2 things that drive this fear of rapid prototyping: the first is the belief that most healthcare innovation projects are so complex that they really cannot be simplified into a rapid prototype, and that we would need to see a readout of the full POC or pilot to be able to determine success or failure. The second is the fear of failure. One of the greatest benefits of rapid prototyping is that it allows us to fail fast, but failing fast is still not looked upon favourably. This is changing, there are more companies that are willing to accept fast failure as almost as good an outcome as a success, however these numbers are still quite small. There are other constraints (like data privacy concerns, etc.) which could also hamper rapid prototyping, however, many of these could be addressed by some very common solutions like using dummy data etc.
Prototyping is essential for incremental iterative refinement. Showing is always better than telling, and of course, if you were to be able to experience it, that would be the best. Prototyping also increases collaboration and communication, which are key ingredients to project success.
End user testing
This goes hand in hand with rapid prototyping. Once a prototype is developed, it makes sense to test it with the end users and see if the solution meets the intended objectives. This again serves multiple purposes, it allows the team to pivot or modify if it does not exactly meet end user requirements, it also allows communication and collaboration between the team members, and most importantly, it gives everybody a sense of progress being made, and thereby keeps the project front and centre in people's minds. These are all desirable things to have. However, we also need to ensure that we do not go overboard with the user testing; beyond a certain point, there's very little gain for every additional user that is involved in testing. Some studies have shown that 5 may be an optimal number of users involved in testing.
Another useful thing to test with end users is the user interface. In projects which require extended user interaction with the solution, user interface becomes another important aspect to consider. This is overlooked often, resulting in solutions that may be doing all the right things, but are too complicated for end users.
Outcome evaluation against the baseline
As we had seen earlier, having a quantitative assessment of baseline itself is often missed, which would make this step impossible. However, if we have done things right, and defined baseline as well as the assessment criteria appropriately, then this should be an easy step which will allow us to have an objective view of whether the POC or pilot has been successful.
Routinely, though, this step is either completely missed, or twisted to proclaim successful POC. We have all seen projects where the original POV definition, and objectives were completely missed, and the POV was adjusted or changed along the way so as to make the final outcome look successful. If the POV adjustment was based on formal rapid prototyping and end user testing processes, then this would not be an issue. However, often, this is not the case. More often than not, it is to be able to declare a successful POC/pilot. The result is that while the POC is successful, the project is never scaled and fully implemented, and therefore, the benefits of the innovation project are never fully realized.
Let's consider the fairly famous collaboration between IBM Watson and M.D. Anderson. IBM Watson was to develop AI based cognitive computing which would help M.D. Anderson achieve its goal of eradicating cancer. IBM Watson was to develop a clinical decision support technology called the Oncology Expert Advisor (OEA). While many acknowledged the pilot to be a success, M.D. Anderson still decided to put this project on hold. Among the many reasons that were proposed for this decision, a few seem to illustrate what we have been discussing so far.
1.The focus changed multiple times during the project; the initial project focused on one type of leukemia, then another, finally settled on lung cancer.
2.There were planned pilots in other hospitals, this never happened.
3.The EHR system changed midway through the project, and the OEA could not work with the new system.
It seems likely that some of the concepts discussed earlier, especially around rapid prototyping as well as end user testing could have helped to either build a solution that could have been scaled up and implemented, or helped to stop the project earlier, saving millions of dollars.
The Race part of the model is the implementation stage. This stage tends to be one of the longest stages, and the most exciting from the viewpoint of seeing an idea blossom into a solution. However, it is important to make sure that the different steps of project management, rapid prototyping, user testing, and evaluation against the baseline are done methodically. Failure to do this would result either in the POC failing after much effort and time or being declared a success based on improperly changed objectives. Both these outcomes are undesirable. In the concluding part of this series, we will discuss one of the most important and hard to navigate part of the innovation journey - how to integrate and scale the innovation project to actually reap the benefits.