Test and Validate

Phase 4: Test and validate — The Run to the End: Part 2

Pat Klingberg Idea-to-Launch, Machine Design

No matter which Idea-to-Launch process you use, or even if you employ the continuous refinement cycle discussed in Setting the Criteria for Success and The Run to the End, you must test the product of the idea with the user and validate it against the product definition.

For new product development (NPD) this phase also includes the commercialization aspects of the product, but for internal use it really is about the usage tests.

“There comes a time in every project to shoot the engineers and begin production.”Anonymous
As discussed previously, the refinement cycle has a form of testing within it; however, the above quote is very appropriate for this phase of the process. You need to produce the machine and test it against the product definition.

In NPD, the testing is done on a prototype, or a series of prototypes, as well as the first production run.

A small “focus factory” is setup ─ outside the end location to allow production to continue uninterrupted ─ to test and validate the machine.

The prototype is shown to a group of end users ─ and possibly even be operated for them ─ to get their feedback. In the case of an in-house project, the prototype may be the only physical product that is manufactured and tested.


As in NPD projects, there can be two different types of tests performed ─ preference and performance.

  • Preference tests gauge the likeability of the new machine.

    The group of testers are exposed to the machine. Shown features corresponding to the identified needs and wants from the product definition, although not all of the features may be present on the machine.

    These tests provide the design team the chance to hear about changes to the design that can make the product even better.

A picture is worth a thousand words

Capture the users’ impressions of the machine using audio and/or video recordings. Video recordings are preferable, since they obtain voice inflections and facial expressions.

These reactions can provide valuable insight into the feelings that can’t be captured when taking written notes or through a questionnaire. But, don’t eliminate the questionnaire.

  • Performance tests show how the machine works, inputs required and outcome(s) produced.

    These tests can and should be held at the same time as the preference tests. Developing the performance tests is as much art as science.

    While the machine’s requirements are clearly stated in the product definition, there may be features missing in the prototype that could affect the outcome.

    Be aware of this and try to factor in the probable success or failure of these added features to the overall requirements. Proof of concept should be the goal at this point. If the concept is proven, then it is time to move on to the phase review.

You have collected valuable data during the preference and performance tests. Now what do you do with all of the data that you collected?

Analyze the data!

Caution must be taken when analyzing the responses to the tests.

Any, and all, suggested changes should be weighed against the product definition and the needs and wants of all of the users.

If too many changes are allowed to happen design creep can set in. Once it does, it’s very difficult to get the project back on track.

Have the project team come to a consensus as to the top changes and their associated reasons to pursue. Then prioritize them.

If there are any major changes, then the team needs to go back over the product definition and determine if conditions changed or something was missed in defining the machine.

Phase Review

In this phase review, data ─ both financial and production ─ should be clearer than it was at the beginning of the development phase.

  • Is the project still within budget?
  • Is the “waste” reduced or eliminated?
  • Does the machine still satisfy the product definition?
  • Have any major obstacles appeared from the preference and/or performance tests?
  • Is the project ready to move on to the next phase?

In addition to being presented with the answers to questions such as these, the timeline is further refined. Showing the added activities. Further milestones are added through to the installation of the final machine.

Is the project ready to move to the installation (launch) phase? If so, we will discuss the launch phase and the steps leading up to that point in the next installment.


Share this Post

Pat Klingberg
Follow me