How to Use APM to Enhance Performance Testing

16 Pages·2017·0.2 MB·English

DevOps Guide:
How to Use APM to Enhance
Performance Testing
CHAPTER 1:
This short eBook discusses how combining performance test
automation with application performance management
Introduction
(APM) solutions can be a powerful enabler to implementing
DevOps within the enterprise.
Implementing DevOps is rapidly becoming the norm for
businesses looking to improve the quality and speed of release
in response to demands of the Digital Revolution. Increasing
the speed and quality of release delivery is key to remaining
competitive and agile to deliver on-demand business change.
When partnered with performance testing, APM is well-placed
to help ensure core applications continue to provide a great
customer experience, even under conditions of peak load.
But first, let’s look at performance testing: what it is and why it’s
needed more than ever.
DevOps Guide: How to Use APM to Enhance Performance Testing 2
Performance Testing
OVERVIEW
Performance testing is a practice designed to simulate and measure how
stable and responsive a software application is under a particular workload.
It can also serve to investigate, measure, validate, or verify other quality
attributes of the system, such as scalability, reliability, and resource usage.
There are different performance tests, each with their own purpose:
Load Test (or Workload Test)
• Designed to simulate the current “peak day in production”
Stress Test
• Understanding how a system reacts to extreme load conditions—like
200% of current production peak, for example
Capacity Test
• Sees if the footprint and number of servers are right-sized
Device, FE, BE, end-to-end, and front-end optimization, focused on:
• Collapsing calls
• Reducing latency
• Minimizing bandwidth and size of resources (css, JS, images)
Endurance or Soak Test
• Runs for an extended time to see if anything fails
DevOps Guide: How to Use APM to Enhance Performance Testing 3
Performance testing is enabled by automation and is impractical as a Technical
manual testing exercise. The key capability that automation provides is • Management and provision of suitable performance
the ability to record and script end-user activity, then iteratively replay test environments
this activity at scale against a target environment, all while measuring • Performance automation tooling selection appropriate for
application and system performance. technical fit, skill set, and budget (ideally via POC)
• Monitoring approach (ideally based on APM)
There are many performance test automation solutions available in today’s • Data collection, analysis, correlation, and reporting
IT marketplace (licensed or open-source) to suit budget and requirement.
Many of these integrate directly with APM solutions like AppDynamics. Once captured, these NFRs enable the creation of test assets that are a true
representation of application behavior, including:
Performance testing is its own discipline and is distinct from other forms
of software testing. It is commonly referred to as “non-functional,” as the • Scripted use cases that represent realistic end-user activity on
focus is on testing application characteristics other than pure function (i.e., browsers or mobile devices. (Alternatively, they could represent
scalability and availability under load). API calls from service consumers.)
• An accurate workload model that defines:
Effective performance testing relies on capturing essential non-functional • The types of performance tests required
requirements (NFRs), including: • The distribution of scripted use cases amongst performance
tests and the scale of load to be generated
Business • A test environment that is appropriate for the level of testing
• Identifying scope (strategic, tactical) required. For end-to-end testing, this should be a copy or close
• Confirming expected deliverables (ensuring no surprises) approximation of production. For other testing requirements,
• Confirming client platforms to test (desktop, mobile, service such as component performance testing in dev, this could be an
consumers, batch) environment of sufficient scale to test service-level agreement
• Identifying key use cases, test data requirements, and test (SLA) compliance for a discrete subset of application functionality.
scenarios for each client platform
• Identifying the key performance indicators (KPIs) that must be
captured to accurately measure application performance
• Building an accurate workload model (essential)
• Deciding whether performance test resourcing will be sourced
in-house or outsourced to a third party
DevOps Guide: How to Use APM to Enhance Performance Testing 4
MONITORING AND ANALYSIS CURRENT PERFORMANCE TESTING TRENDS
Of course, simulating end-user traffic is only half the picture. We need to be In response to the Digital Revolution and business requirements,
able to capture and understand application performance under load, as performance test automation solutions have evolved over time to
well. Performance automation solutions have traditionally provided metrics accommodate:
such as:
• Increased release velocity, a catalyst for change
• Response time of complete and nominated subsets of scripted use • Smaller, more frequent releases
cases. For example, a user may log in, carry out a search, then add an • Faster time to market for new features and fixes
item to their basket. Each of these user actions can typically be timed. • Reduction in pain of “giant monolithic deployments”
• The number of concurrent users at any point during a test execution. • Customer demands: faster, better, less patience
This metric is typically correlated with response time to demonstrate
how increasing load impacts application response time. Tooling vendors now typically offer:
• The number and type of errors that occur during a test run. The
appearance of errors at a certain level of load is an important tell-tale • Focused support for modern web development technologies, such as
sign of problems with application scalability. Node.js, AngularJS, SPF, and WebSockets
• Integration with established functional test solutions, such as Selenium
While these metrics are great at identifying when a problem occurs, they • Integration with test automation servers, such as Jenkins as part of CI
don’t necessarily pinpoint the root cause. To identify root cause, we need and CD
to capture performance data from the full stack, including the network, • Integration with APM
hosting servers, and databases.
The last evolution is key, as integration with APM solutions like AppDynamics
Performance automation solutions have long provided integration with provides unique insight and understanding as to why an application fails to
monitoring point solutions like Windows Performance Monitor or SNMP. scale or respond under load—and more importantly, how this impacts the
And while this does give additional insight into application performance, it end user.
is not full-stack and only reports on the metrics selected for the servers or
software components being monitored.
DevOps Guide: How to Use APM to Enhance Performance Testing 5
CHAPTER 2:
Undetected performance defects at any stage of the
systems development life cycle (SDLC) can impact an
Performance
application’s ability to deliver the level of service and
availability that the business and the customer expect.
Challenges
“Shift left” is a popular term to describe a change to the
traditional software testing approach. In response to
the widespread adoption of agile and DevOps, shift left
promotes testing early and often, not just in QA but also
in dev. This provides some important advantages over a
traditional QA-only approach to testing:
• The earlier you detect defects in the SDLC, the easier
and cheaper it is to fix them.
• By finding and fixing performance defects earlier, you
reduce the amount of testing required, increasing the
speed and quality of delivery.

With that in mind, the next question becomes, “How do you
stop performance defects from creeping into the SDLC in
the first place?” Let’s look at some common reasons why
performance defects occur in the following IT stages:
• Design
• Development
• Testing
• Deployment
• Scalability
DevOps Guide: How to Use APM to Enhance Performance Testing 6
DESIGN TESTING
For design, this responsibility clearly lies with business and IT stakeholders, Performance test automation has evolved from basic replay engines to
as well as anyone else who has input into application design. sophisticated test and monitoring frameworks. There’s certainly a lot of
choice in the marketplace, including established open-source offerings
Put simply, if the application isn’t designed from the ground up to scale, like JMeter.
then it will be very unlikely to satisfy business or end-user requirements for
performance. What’s more, these problems may not be detected until late The good news is that most businesses—regardless of size—now use some
in the SDLC, when it typically can only be corrected with time-consuming form of performance automation tooling within QA.
and costly remedial action.
The bad news is that performance testing is often implemented as a purely
DEVELOPMENT tactical function, lacking formal process and management when it should
be an integral part of strategic IT planning. This can result in incomplete
The traditional model for development has teams of coders assigned to performance NFR capture and release decisions, based on incomplete or
groups of work items and tasks. Areas of performance risk include: misleading test results.
• Lack of or minimal control over coding standards DEPLOYMENT
• Developers not having the bigger application picture
• Developers not typically taking performance SLAs into account When it comes to deployment decision-making, there is potential for
when coding considerable performance risk.
• Developers not conducting performance testing
• Developers not talking to Ops If those responsible for production deployment don’t have visibility into
appropriate performance metrics for the release candidate under
In reality, much of the above is changing, due in no small part to the consideration, then a critical piece is missing from the data that informs
widespread adoption of an agile development style. However, the picture is their decision.
still patchy within many organizations on the journey from waterfall to agile.
In other words, there should be a clear indication of whether the application
Performance considerations are often only considered at a high level even has met any performance SLAs related to response time, scalability, and
within agile-sprint planning and are frequently deferred in response to availability at anticipated peak load.
pressure to deploy.
DevOps Guide: How to Use APM to Enhance Performance Testing 7
SCALABILITY
An application that doesn’t scale is highly vulnerable to unexpected peaks This temporary fix may provide some breathing space, but it inevitably
in load, which can result in partial or complete loss of availability. Even if it incurs increasing costs, as expanding numbers of cloud instances need to
stays available, end-user performance may degrade to such an extent that be provisioned to support demand.
the application may as well be unavailable.
Lack of scalability is more often than not a result of a design miss, so it is
The emergence of the cloud has enabled many applications that were not unlikely to be trivial (or inexpensive) to correct.
originally designed with scalability in mind to continue to provide a service
to end users under increasing load.
Fig 1 – Application Performance Challenges
System Under Test
EUE
Mobile Users
Capacity
Web Server Tier
Network Issues
EUE
Web Users
Code Quality
App Server Tier
Contention
Database Cluster
Config Issues
EUE
LAN Users (No) SQL
Performance
DevOps Guide: How to Use APM to Enhance Performance Testing 8
CHAPTER 3:
Now that we’ve covered the performance risks and
how they’re present across the SDLC, let’s explore
The Process
how we can address these risks using performance
testing best practices and application performance
management (APM).
DevOps Guide: How to Use APM to Enhance Performance Testing 9
Application Performance Management (APM)
A LITTLE HISTORY
APM has evolved from simple code profilers commonly used in
development to full-stack analysis of software applications across the
SDLC—from last-mile to first-mile with increasing focus on understanding
end-user experience.
The transactional view of user actions (from browser click to database
call) provided by APM is the most effective way to triage and resolve
application performance problems as part of performance testing or
production monitoring.
APM is rapidly becoming the solution of choice for centralized performance
and availability management, as well as the monitoring of core applications
and end users.
More importantly, this cross-SDLC capability makes APM a natural
choice for a partnership with performance test automation.
Let’s revisit the IT stages we discussed in Chapter 2 to see how APM
and performance testing can help mitigate performance risk and
enable DevOps.
DevOps Guide: How to Use APM to Enhance Performance Testing 10