This article walks through the fundamentals of FAT, how it differs from other testing phases, and how to prepare your system and your customer for a smooth buy-off.

Background

Before we dive into the topic of Factory Acceptance Testing, I want to outline an example of the Project Process DMC follows. After thousands of projects, DMC has found that a phased approach is the best way to make a system production-ready.

• Requirements & Specifications – DMC works collaboratively with the client to determine system requirements. This phase often culminates in a “System Functional Review.”
• Preliminary Design – DMC defines a Preliminary Design as a ~20% complete design, where the high-level architecture has been mapped out and identified risks have been evaluated to the point where a plan is in place. This phase often culminates in a “Preliminary Design Review.”
• Comprehensive Design – DMC defines a Comprehensive Design as an ~80% complete design, with the remaining tasks to be finalized during the Development & Fabrication phases. (i.e., what nuts and bolts will be used to mount the hardware). This phase often culminates in a “Critical Design Review.”
• Development – The software development portion of the implementation phase, where any custom software is created and tested.
• Fabrication – The hardware build portion of the implementation phase, where any hardware is assembled into a complete system.
• Factory Acceptance Testing – The point in the project process where DMC validates the system prior to shipping it to the client site.
• Site Acceptance Testing – The final validation testing performed after the system arrives at the client site.
• Training & Support – After delivery and acceptance, DMC can provide training on using or extending the system, as well as support for troubleshooting and future improvements.

Learn more about DMC’s Project Management that makes this flow happen.

Project Specific Customizations

At DMC, we understand that not all projects are the same, so the Project Phases are customized on an individual project level.

• If a customer already has a well-defined Requirement & Specification document, we can jump straight into the Design phases.
• For small projects, we might combine “Requirement & Specification, Preliminary Design, and Comprehensive Design” into a single “Design” phase.
• For software-only projects, we remove the Fabrication phase and may combine “Factory Acceptance Testing and Site Acceptance Testing” into a single “Testing” phase.

Now that we have a better understanding of DMC’s typical project process, let’s talk about how we can prepare for a successful Factory Acceptance Testing phase.

FAT vs. SAT vs. ATP: What’s the Difference?

Before we dive into any details, let’s define terms that will be used in this blog.

Factory Acceptance Testing (FAT)

FAT is testing at the fabrication location to verify that the system meets the agreed-upon requirements before shipment. FAT focuses on testing that can be performed without the systems or hardware only available at the customer’s facility.

Site Acceptance Testing (SAT)

SAT is a test conducted after the system has been installed at the customer’s facility and is the first opportunity to test with systems or hardware only available at the site. SAT will often repeat tests performed during FAT to ensure the system wasn’t damaged during transport.

Acceptance Test Plan (ATP)

The ATP is the test plan used to execute FAT or SAT. It defines what is tested, how it is tested, and what constitutes a pass or fail. A well-written ATP is essential for efficient execution and clear customer sign-off.

What Does a Successful FAT Look Like?

FAT is successful when all parties are in agreement that the system meets its requirements and is ready for shipment to its final location.

How Do We Ensure FAT Is Successful?

Preparation is critical to ensuring that FAT is an efficient and productive use of time. We will dive a little deeper into how we can maximize testing prior to FAT, define participation and roles during FAT, and how to develop the ATP.

What Testing Should Occur Before FAT?

FAT should not be the first time testing occurs during the project process. Systems and code should be tested as they are developed. Testing strategy is a huge topic and will not be discussed further in this blog.

Who Participates During FAT?

At DMC, customer involvement during the FAT process is highly encouraged. At the end of the day, the customer will own and operate the system, so DMC strives to include the customer in as many aspects of our project process as possible. Some benefits of customer involvement during FAT are:

• A chance to make tweaks to the system without major cost or schedule impacts to the project.
• An opportunity to receive hands-on training with the team that developed the system.
• Provides an opportunity to identify gaps or misunderstandings in requirements.

Regardless of whether or not the customer attends FAT, they should be involved in developing the ATP. Ideally, either DMC or the client would write the acceptance test plan, while the other reviews and approves.

Typically, the customer should be responsible for writing the acceptance test plan to ensure complete test coverage and feel comfortable accepting the system at the end of the day. However, DMC engineers are experts and will often develop the ATP.

How Do You Develop an ATP?

Each item in an ATP should be clear, repeatable, and objective. In order to achieve this goal, each test should have the following core elements at a minimum:

• Test Procedure: Step-by-step instructions
• Acceptance Criteria: Objective pass/fail conditions
• Observed Results: What actually occurred
• Result Files: Logs, screenshots, or data files as evidence

Depending on your specific requirements, you can also consider tracking the Test Type, Test Operator, Test Date, and the approval status of Individual Results. If you’re looking for more information on why you might need an ATP, check out this blog.

Does the Order of ATP Execution Matter?

The order of individual ATP items is crucial for reducing the amount of time spent troubleshooting and debugging the system. The highest system-level impact should come first, followed by more narrowly scoped items. This narrows down potential causes of issues, reducing the overall time spent troubleshooting and debugging the system.

As you test systems and subsystems, start with the simplest components and add more complex capabilities sequentially.

A high-level concept for ATP Execution Order is as follows:

Conclusion

Factory Acceptance Testing is more than a checkbox. By involving the customer early, planning FAT thoughtfully, and executing against a clear, well-structured ATP, teams can streamline buy-off, minimize rework, and set the project up for long-term success.

The post Factory Acceptance Testing 101: Preparing for Customer Buy-Off appeared first on DMC, Inc..

This article describes why developing a comprehensive Acceptance Test Plan (and then rigorously applying it) is important to overall project success.

Benefits of an Acceptance Test Plan

An Acceptance Test Plan provides many significant benefits to the project process.

Improves System Quality

Perhaps the most obvious benefit of an Acceptance Test Plan is that it provides a systematic, trackable, structured, and unified approach to performing system verification and validation.

Saves Calendar Time

Save time by parallelizing the work of fully defining test cases up front (as the Acceptance Test Plan is formulated), instead of doing this in-line (thus on critical project path) during actual commissioning activities. This minimizes total calendar time required for testing activities.

Reduces Engineering Time Spent on Testing

Writing a good Acceptance Test Plan takes time, but performing those tests takes even MORE time. Developing a complete Acceptance Test Plan in advance allows the full team to review and refine the plan prior to starting commissioning. This facilitates consensus agreement on what to test how it should be tested. This makes commissioning time more focused and productive.

The Plan Gets Utilized Twice for a Dual Benefit

The Acceptance Test Plan can be applied both during the upfront development level or “simulated” testing phases, and during final commissioning of the completed system. Thus, many of the benefits of an effective test plan get magnified. Although some aspects of the Acceptance Test plan won’t apply to the preliminary testing stage, there is still a significant ROI across the subsequent stages.

Guides and Supports System Development

Good test plans inherently flush out any ambiguities, gaps, or misunderstandings relating to system requirements. Applying these corrections sooner saves time and money.

Establishes Clearly Defined Milestones for Completion

Define successful completion of the project, or at least the current phase. A good Acceptance Test Plan avoids ambiguity or discrepancies on what it means for the system to be “working the way it needs to work.” Under the guidance of a good Acceptance Test Plan, all members of the team (both Client personnel and DMC) are striving towards a mutually understood goal.

Characteristics of a Good Acceptance Test Plan 

As a point of reference to everyone involved in the project, DMC suggests that a good Acceptance Test Plan should exhibit the following characteristics:

• Once it’s successfully executed, the Client is confident that the new system is properly doing its job, and that it will meet business needs.
• The Acceptance Test Plan should clearly define all test cases needed to accomplish the above goal. Each such case should:
  • Define the relevant input parameters and setup (both hardware and software) that need to be in place to start the test.
  • Define the process to execute that test case in clear, concrete, and actionable terms. This makes testing activities repeatable and self-documenting. Later, someone can look back and see a pass/fail result and exactly how it was created.
  • Define the expected results and/or pass-fail criteria for that test in clear, unambiguous, quantifiable terms. Tests should be broken down to a level where the answer to “Did it do what it was supposed to do” is a very clear YES or NO answer. 
    • Bad: The Configuration GUI screen operates as expected.
    • Good: When the “Save As” button is pressed, the operator is prompted with a “Browse” dialog, allowing them to select a file path and file name.  
    • Good: When “OK” is pressed on this Browse dialog, the program saves the parameter file in an INI format in the specified file path location with the specified file name. The parameter file shall contain all previously entered parameter values.

Return on Upfront Time Investment 

The following questions commonly arise when considering Acceptance Test Plan development:

Is it worth it to spend this time to define how the system needs to be tested, and exactly how each test will be conducted?  

Aren’t we smart enough to figure this out as we go along?

When developing complex, customized, and mission-critical systems, omitting rigorous system testing from the project process is NOT an option. Doing so may detract from the quality of the final system and risk unacceptable failures and issues.

It is true that defining the full process for each test case is no small task and will take some time. It’s critical to keep in mind that, to perform any commissioning tests, the work to define tests must still be completed at some point. 

We have two options for when to figure out these details:

• During acceptance testing – In this case, all work happens in-line with actual commissioning/testing activities. It is on the project critical path and adds time to overall project schedule.
• Before acceptance testing/commissioning – Include this information in a pre-defined Acceptance Test Plan. In this case, all time happens in PARALLEL to other critical path development activities, and IN ADVANCE OF actual commissioning activities. This effort is parallelized, off the critical path, and has limited impact on overall project schedule.

Another time-saving benefit is that, once documented in an Acceptance Test Plan, tests can be reliably repeated. If something fails once (which is going to happen many times in a complex system), having a good Acceptance Test Plan means that, after making corrective modifications to the system, any person will be able to conduct the same test in the same way it was originally done.  

Moreover, this can be done without needing to re-formulate the steps and without requiring that the exact same people conduct the test. An Acceptance Test Plan adds structure, documentation, and repeatability to the test process.

Who Should Write the Acceptance Test Plan? 

The generation of the Acceptance Test Plan should be a collaborative effort among all team members. Certain team members know specific aspects of the system better than others, and all members of the team are invested and share responsibility in making sure the system works properly and meets Client needs.

DMC suggests that the Client team is best suited to have primary responsibility for creating the Acceptance Test Plan.

Why is this the case?

The Acceptance Test Plan will be the primary way for the Client to exercise “quality assurance” over the new system and make sure it does the job it needs to do. When push comes to shove, “the job it needs to do” is a matter of the Client’s business needs. Thus, the definition of how to check if the system is doing its job (i.e., the Acceptance Test Plan) is most meaningful if it comes from the relevant stakeholders and end users on the Client’s team.

Additionally, having client stakeholders develop test cases creates a system of “checks and balances.” A supporting purpose of an Acceptance Test Plan is to verify that the developers correctly interpreted each requirement. For example, if DMC writes a test case about a particular requirement, it’s inherently based our interpretation of the requirement, which will naturally align with how we implemented that requirement. The system is likely to pass that test case, whether our requirement interpretation is accurate.

Therefore, it’s generally not best practice for developers to create the final test cases for their code. Test cases are often driven by those who are the “end users” of the given system. Having Client stakeholders develop test cases creates a system of “checks and balances” and enables the Client to act in an oversight capacity for system quality assurance.

Requirements Vs. Test Cases

A common view on test strategies is that testing should consist of going through the original requirements document and checking that each requirement is satisfied. This makes sense because a good Acceptance Test Plan should essentially be a set of test cases that provide 100% coverage of the system requirements. However, the requirements document is not an Acceptance Test Plan. There is a fundamental difference between a requirement (a statement of what a system shall be capable of) and a test case (a statement of conditions to establish, steps to follow, and criteria to evaluate to determine if a system is meeting a requirement). Yet there is a close correlation between requirements and test cases.

DMC recommends that one should reference the system requirements documentation while drafting the Acceptance Test Plan. We should be translating and expanding each requirement into a test case (or multiple test cases) that effectively checks whether the system is meeting that requirement. If we do this for each system requirement, we end up with a good Acceptance Test Plan that has 100% coverage of the requirements. 

The post Deciding If Your Project Needs an Acceptance Test Plan appeared first on DMC, Inc..