About test studies

A test study is a structured examination of a system’s functionality, performance, and end-user experience to determine whether it meets specified requirements and operates as expected. Currently, there are two general types of test studies available:

1. System test studies - developed using a particular reference architecture as a starting point. The study builds on the reference architecture by:
   • Identifying the specific workflows, capabilities, and non-functional requirements needed for a particular industry system
   • Documenting the specific steps performed in each workflow, ensuring the steps (and their sequence) are optimal
   • Designing the physical architecture, including the selection of infrastructure components and application of key design considerations
   • Developing a well-defined testing strategy to evaluate the system’s behavior and end-user experience under specific conditions
   • Completing a test which includes applying load to the system to identify how well it responds to both design load and future increases, based on the defined strategy, workflows, and architecture
2. Configuration test studies — a focused evaluation of a certain configuration, such as virtual machine size, instance type, network configuration, or application configuration. These kinds of test studies:
   • Often start with the same initial physical architecture that was used for a system test study
   • Establish the context for the test, including the system type, workflows, and capabilities
   • Describe the system configuration being tested and the test methods used
   • Control other system components to ensure meaningful result
   • Complete a test of the system to identify whether the configuration changes make a meaningful difference

The intent of providing test studies is to help organizations design and implement well-architected systems. Specifically, test studies aim to provide an example of:

• How reference architectures can be used to support the design process
• A starting point for designing a certain kind of system, offering context and an example of a configuration that worked under specific conditions and for specific purposes.
• Information to understand potential implications and impacts of making certain design and configuration choices
• A testing approach that can be used to assess your organization’s systems before they are implemented in a production environment.

Test study gallery

Test studies are a brand-new addition to the ArcGIS Architecture Center. There are currently eight available:

• Network Information Management System: Gas Utility (SQL Server)
• Network Information Management System: Gas Utility (SAP HANA)
• Network Information Management System: Electric Utility (Oracle)
• Network Information Management System: Electric Utility (PostgreSQL)
• Network Information Management System: Performance and Usability Comparison with SAP HANA deployed in RISE with SAP Private Cloud Edition
• Evaluating GPU and CPU Impact On Desktop Editing Workflows
• Evaluating impact of adding mobile capabilities to a foundational network information management system
• Evaluating the impact of visibility range on system performance and user experience

More to come soon!

Anatomy of a test study

All system test studies presented in the ArcGIS Architecture Center have four primary elements:

1. Overview
2. Tested workflows
3. Physical architecture
4. Test methods and results

Overview

The overview section introduces the parameters of the test study. It also outlines some of the key capabilities the system delivers, the software components included to enable those capabilities, the respective software versions that were used, and an overview of the data that were tested.

Tested workflows

The tested workflows section outlines the specific steps each user persona conducted for each of the defined workflows in the reference architecture. Workflows in this context refer to a series of tasks that a user would perform using the system to achieve a specific business process or objective.

Developing a workflow as a detailed, stand-alone set of steps allows them to be repeated, iterated on to tweak operations, or adjusted to improve performance or user experience. Once each workflow is clearly defined, it is used consistently throughout the remainder of testing to establish the performance baseline and conduct load tests.

Physical architecture

The physical architecture section is where you will find a diagram of the physical design. It illustrates the core software components of the system and how they were implemented within the deployment pattern used (e.g. Windows/Linux, Kubernetes). The diagram also defines the specific physical characteristics, such as machine counts, machine types, and disk size that are allocated to each system component.

Accompanying the diagram is a high-level description of the resourcing and configuration considerations applied to the design, organized around the six pillars of the ArcGIS Well-Architected Framework.

Test methods and results

The test methods and results section explains the testing approach taken, including the test scope, workflow pacing, and testing tools used. Additionally, it includes the test results and an evaluation for multiple test scenarios, such as for:

1. Design load (as defined in the test study)
2. 4x design load
3. 8x design load

Workflow duration charts are also shared to give context for what end-user experience is like under each load scenario and to help inform your design based on relative load each workflow places on the system.

More to come!

These test study resources are an exciting addition to the ArcGIS Architecture Center. We hope they help you successfully design and implement business systems with ArcGIS. We will continue to publish more test studies, so stay tuned for new additions to the site, and let us know what you’d like to see next through the “Tell us what you think” links at the bottom of each page!