Test methods and results

Testing was conducted to validate that the design would perform as expected and support the workflows, users, and intended load. System tests provide the opportunity to discover and correct problems during system deployment in lower environments, ideally before they appear in production. For this test study, the focus of the testing approach was system performance and end-user experience. Each component was monitored as the workflows were conducted against different load scenarios.

Upon test completion, results were assembled and analyzed to identify both bottlenecks and over-resourced components in the system. This information was used to identify system components that needed to be scaled up, down, or out before further testing was repeated.

Manual user experience testing was conducted by capturing screen recordings of the workflow testers to ensure users of the system could complete their workflows productively.

For more information, see how to design an effective test strategy.

Workflow pacing

This test study applied a pacing model to the tested workflows. The pacing model shows how the test intends to simulate the pace of work at a utility, where workflows are performed as some number of operations per hour across a team of staff resources. This approach was based on Esri customer input and aimed to match the small to medium gas utility customer scenario that the data was based on.

The various workflows were spread out through a one-hour test period and staggered so as to not start at the same time, while overlapping with each other as real-world workflows also would. This overall breakdown of workflow pacing is considered the “design load” that the system is subjected to. The load was then increased by multiplying the workflows to a point where the system was no longer able to provide acceptable responses or support successful workflows. Note that the workflow pacing model applied in this test study might not match typical daily use at your organization.

Performance testing tools

Because ArcGIS is a multi-tier system, performance tests were conducted across client, service, and data storage tiers, as well as the underlying infrastructure itself. In this test study, JMeter was used to simulate the user workflows and measure system performance under different loads. ArcGIS Pro requests were recorded and then replayed to simulate load in addition to manual workflows that were performed to assess end-user experience. Windows Performance Monitor and ArcGIS Monitor were also used to monitor resource utilization across different components.

For more information, see tools for performance testing.

Test results

This architecture was validated with automated load tests and manual users in three scenarios, and you can see the results from each below. At a high level, the test results show that as implemented, the system is adequately resourced to support loads from the design load through 4x the design load. Tests also reinforced the importance of proper application and system configuration for performance. Across each scenario system utilization increases proportionally with load.

Test scenario: Design load

• The system supported this load
• The hosting servers generally ran below 20% CPU utilization
• The GIS Servers generally ran below 20% CPU utilization
• The SQL Server generally ran below 20% CPU utilization

Test scenario: 4x design load

• The system supported this load
• The hosting servers generally ran below 40% CPU utilization
• The GIS Servers generally ran below 40% CPU, peaking at 50-60% utilization
• The SQL Server generally ran below 50% CPU utilization

Test scenario: 8x design load

• The system did not support the load
• The hosting servers peaked at over 80% CPU utilization
• The GIS servers peaked at over 80% CPU utilization
• The SQL Server peaked at over 90% CPU utilization
• The successful concurrent users ramped up as requests were unable to be completed due to the system becoming overloaded, and the system began to return errors in the logs and telemetry

User experience - conducted workflow times

While the system was under load, conducted workflow times were captured as experienced by the users. This represents the time it took to complete all the steps listed in the workflows. Conducted workflow times were consistent until the system was overloaded at 8x design load.

User experience - conducted workflow step times

While the system was under load, conducted workflow times for key individual steps across all eight workloads were captured. This represents the average time it took to complete a given step. Conducted times are consistent until the system becomes overloaded at 8x design load.