ArcGIS system testing should validate that the implemented design will perform as expected and support the workflows, users, and load it is intended for. System tests provide the opportunity to discover and correct problems before they appear in production. While it may not be realistic to test every possible component, event, or situation, some pragmatic approach should be defined to test key aspects of the system. For example, the ability of the system to support defined workflows, recover from a component failure, or handle a new workload.
Intentional and structured testing is critical to the success of any application, workflow, or system. There are many different types of testing that have relevance through the development cycle, including functional testing, accessibility testing, acceptance testing, and other types. Testing is a key part of the architecture process as it helps to both validate an architectural recommendation, but also provide inputs that might lead to a suggested architectural change – such as a certain database or software release performing significantly better or worse than another offering.
For the purposes of this section, this topic will focus on performance testing – the methods and processes that help with identifying whether system baseline performance characteristics are being met. Testing must occur in parallel to telemetry monitoring and should permit the testing of each monitored component. As changes are introduced into the system it is imperative that testing is re-performed to identify any degradations. Testing should also be used to inform the potential architecture bottlenecks based upon anticipated load changes.
This ideal level of iterative, standardized, and repeatable testing requires automation of the process to be successful. With SaaS deployment patterns, the implementation of automated testing and especially performance testing also requires coordination with other providers, systems, and stakeholders.
Above all, a solid testing approach should determine the goals of testing, what you will and will not test (the scope), what will be measured (telemetry), and the success criteria.
During performance testing, many factors influence the performance of a workflow, including the end client’s hardware, the network connectivity to the backend system, and the backend system configuration. Understanding the details of these components is critical to understanding whether performance measurements are primarily influenced by the ArcGIS software being tested, or by other components such as a proxy, database, or network switch that introduces latency or interruptions.
Different types of testing can be used to achieve different goals. For example, you may want to:
When building a test strategy, by clearly identifying the goals, additional steps will proceed more efficiently.
Many systems maintain a “lower” environment (test, or staging or pre-production, for example) that is intended for functional or load testing. For more information on successful use of testing or staging environments, see the environment isolation page in this site.
An enterprise system often has multiple separate applications, where workflows are conducted, including desktop, mobile and web clients, and may include both read/write and reporting workflows, as well as support for many different types of users. Simply trying to “test the system” without first defining this boundary will lead to many test cases, and a reduced likelihood of the information being useful for further action.
A good system test will mimic the way the system will be used, so you know that it will function and perform as expected. Therefore, the test scope should include workflows the system is intended specifically to support and designed to provide the metrics needed to evaluate the success criteria. For example, if you are upgrading software on an existing system, the scope should include all key workflows exercised under a load, representing the expected maximum operations per hour.
It’s possible that some workloads do not occur at the same time, such as overnight processing of scripts that import or export data, versus normal daytime operations. In those cases, the system might be tested twice; once to ensure it will handle daytime loads of people accessing the system and once to ensure overnight processing is completed in the required time. The system should also be re-tested when any changes are introduced to establish whether they caused any negative impacts.
Telemetry is used during system testing to collect information about the system’s performance under different scenarios to assess whether the system meets requirements and for identifying issues or anomalies. While many people may initially think of load testing the ArcGIS Server deployment, it is important to capture telemetry across the entire system. For example, you may gather telemetry across the server infrastructure that shows the system was highly performant, only to learn later that end users struggled to complete workflows because their desktop machines were under resourced, so the acceptable performance of the backend system looked to them like an unacceptably slow system.
One key goal should be to model the test script and requests that are sent in testing based on requests collected via telemetry monitoring – both average requests and outliers. Testing should also look to test individual components along with end-to-end processes. With the complexity of most enterprise systems, it may not be possible to use a single test tool to perform all testing functions, but automated test runners, such as JMeter, should be configured to permit the adjustment of test routines with standardized test inputs and outputs.
Good success criteria will enable you to determine if the system passed or failed a test, and where any problems might lie. Your success criteria will inform the scope of your tests and the type and granularity of telemetry you capture. Criteria could include metrics such as the number of operations per hour, concurrent users, hardware performance, or workflow completion time.
Additional resources related to testing include: