How Can the Cloud Improve your App's Quality? Part I

Should your quality assurance processes and tools change if you are building an application for the cloud? The answer is obvious - "it depends." Despite having one of the clearest definitions amongst recent buzzwords, there are many types of clouds and many ways to integrate them with your application or development processes. If you have an experienced team that can make informed choices, then these options are a great thing for quality. The team can take advantage of the cloud as a tool to assure quality, rather than viewing it as an extra aspect that needs to be tested.

What is the cloud?

There are a few clear definitions of the cloud that I find useful.

The first one from Gartner only describes a single cloud use case, but is a great way to promote cloud quality:

Virtual lab environment. These customers typically try to provide self-service infrastructure to a group of technical users, such as developers, scientists or engineers, for the purposes of test and development, or scientific computing or other batch computing. (From Gartner's "Magic Quadrant for Public Cloud Infrastructure as a Service")

The self-service nature is very important. If you only need an environment for 30 minutes to run a spike, than it is wasteful if you need to spend more than 30 minutes budgeting or scheduling the creation of that environment. One of my colleagues quipped: “I can spin up a VM in minutes, destroy it whenever, and repeat this process as frequently as I need. Anything less is just yesterday's infrastructure. If I have to email someone, open a ticket, submit a request to get a VM, etc. then it's yesterday's infrastructure.”

The other important definition is the official definition of cloud computing from the National Institute of Standards and Technology. The definition (as summarized by Martin Fowler) contains:

• Five essential characteristics: on demand self-service, broad network access, resource pooling, rapid elasticity, and measured service.
• Three service models: software, platform, and infrastructure (all as a service).
• Tour deployment models: private, community, public, and hybrid.

Cloud Quality Advantages

Let’s see how we can we take advantage of these cloud characteristics for the 8 dimensions of quality assurance. In Part I we’ll examine how the cloud helps with Performance, Features, Reliability and Conformance. In Part II we’ll describe the positive effects of the cloud on Durability, Serviceability, Aesthetics and Perceived Quality.

#1 Performance

Performance refers to a product's primary operating characteristics and involves measurable attributes. I believe performance is the #1 business feature. There have been many studies¹ showing that even a small reduction in performance can have a big business impact. The cloud helps you improve your performance in several ways.

a. Resource pooling and rapid elasticity give you extra oomph when you need it, without overspending.

Since rapid elasticity is an essential characteristic of cloud computing, you have the option to ramp up compute resources to match spikes or recurring demand trends. For example, you could run more servers during weekdays than on the weekend. You're able to make these fine-tuned adjustments without blowing the budget, because another essential characteristic - measured service - means you are usually billed per-hour (or even in smaller chunks of time) for the resources.

b. Broad Network Access and Content Delivery Networks (CDNs) put you closer to your customers and business collaborators

The broad network access characteristic of the cloud makes it easy to take advantage of high-speed networks, fast DNS services, and CDNs like Akamai to speed up your application. This is especially important for mobile applications, where networks are slower, making every byte or millisecond of latency count. A CDN can help you cache content closer to your users, compress your content, and make sure it supports conditional requests. All of these can significantly speed up your site.

c. “Testing as a Service” companies let you quickly ramp up and run performance tests

There are many cloud services offered to help you test or increase the performance of your site. It is difficult and expensive to build your own performance test framework that can rapidly ramp up to simulate a large number of users. Often homegrown frameworks run into trouble because they themselves run into network or performance problems before the application they are testing, or because they lack features to simulate users in other parts of the world, or due to slow connections. Instead of building your own, you can use a service like Blitz.io. Blitz can economically simulate 1000 users in the network of your choice for 1 minute and even scale this up to 50000 users for 20 minutes.

If you need to test a more complex workflow, you can use something like BlazeMeter, which lets you run JMeter or Selenium tests from their cloud.

#2 Features

Features are additional characteristics that enhance the appeal and value of the product or service to the user. However, this is just a hypothesis until you put the feature in front of real users and validate its value. Functionality that is well-tested but not useful is not a feature, it is "high-quality bloat". The cloud can help you avoid bloat and quickly test that your features are properly implemented.

a. Cloud tools means you always have enough test power.

In order to quickly test features, we can take advantage of the elastic capabilities of the cloud. You can use tools like [jclouds-jenkins] to make sure your Continuous Delivery pipeline can handle a spike in commits without running out of nodes. You can use tools like Vagrant (with your choice of providers like VMWare or vagrant-rackspace, and provisioner like Puppet, Chef or Ansible) to quickly create a test environment, and destroy it once you're finished. You can also use SaaS testing services, like SauceLabs, Appium, Xamarin, Appium or Soasta so you can speed up your test suite by running tests in parallel or on multiple browsers or mobile devices.

b. The many *aaS models give you build-vs-buy options. IaaS takes complete control, PaaS saves you some technical architecture decisions, DBaaS, Logging-as-a-Service, etc. shift a certain "aspect" to another vendor.

One strategy to avoid bloat is to keep your application focused on solving the business domain, and trusting SaaS partners for related services. A good example is sending mail. It is not easy to quickly process a mail template and send it to a large number of subscribers. In addition to building the capacity to quickly process templates and send emails, you need to process unsubscription requests, spam complaints, email campaign schedules, analytics and much more. Providers like Mailgun let you outsource these problems and focus on your core business. These sorts of providers are available for many ancillary needs, like video encoding through Zencoder or payment processing through PayPal.

c. Easy deployments make A/B, MAB (Multi-Armed Bandit), and other options more viable.

Applications such as Mailgun support analytics and A/B testing, two methods that help ensure quality in the features you deliver. Analytics, A/B and multivariate testing make it possible to run experiments to see what features or content provides the most value. Other cloud services that help you implement these experiments include Google Analytics Content Experiments, Optimizely and VWO. You can also integrate them directly into your application, using libraries like the split gem for Ruby.

If you use these techniques, you can:

• Optimize the feature-to-bloat ratio by using A/B or multivariate tests to prove new functionality provides value before deciding to launch it for everyone.
• Minimize the amount of ancillary code and tests you maintain by utilizing services like Mailgun and Zencoder.
• Quickly test the remaining business features by elastically scaling your test infrastructure or using cloud services from testing vendors.

The reason we recommended so many SaaS solutions in this because they are highly reliable. PayPal has shared their cloud infrastructure is the secret to their reliability.

#3 Reliability

Reliability is the likelihood that a product will not fail within a specific time period, and can be especially critical for certain domains. Reliability may be the dimension of quality where the cloud provides the biggest advantage. Even though individual components on the cloud may fail, the cloud makes it very easy to "design for failure". Thus you will have a resilient application that can survive even the most severe outages.

a. The cloud can let you spread your application across multiple data centers or even providers, ensuring Disaster Recovery.

Most public cloud providers offer services from several data centers across the world. You can take advantage of this to easily build a highly redundant application or create several disaster recovery sites. If that's not enough, you could use RightScale, which helps you manage multiple public clouds across the globe, as well as your own private clouds. If you want to make sure your application never goes down, you could deploy it in several sites across the world on Rackspace, Amazon, and your own private OpenStack instance.

b. The cloud also gives you back-up abilities and unlimited storage, so backup retention is just a cost decision.

The cloud also makes it extremely simple to manage backups. Object Storage services like Amazon S3 and Rackspace Cloud Files provide data redundancy and virtually unlimited storage. You can simply click (or schedule) when you want a backup.

c. Cloud providers and *aaS companies offer monitoring of all types.

There are also many great cloud monitoring systems, so you can detect a deteriorating service before it becomes a major problem. You can set up monitoring alerts for most cloud infrastructure through the vendor, and can also integrate your application with great monitoring and log management services like NewRelic, Loggly or Splunk Storm.

#4 Conformance

Conformance is the precision with which the product or service meets specified standards. The cloud can help with conformance or compliance needs either by letting you avoid the need for compliance to a standard by using a service provider, or by providing consistent infrastructure to help ensure conformance.

a. Use SaaS options to avoid the need for conformance.

Often, you can defer an ancillary feature to a compliant SaaS provider, so you don't need to deal with the requirement yourself. You can use Mailgun's expertise on the CAN-SPAM Act, like ensuring your e-mails have an unsubscribe link and a physical mailing address. You can let PayPal or Braintree Payment Gateway deal with the majority of your PCI-DSS compliance concerns. If you need strong auditing and security to guarantee compliance there are services like Dome9, which provide advanced security and compliance auditing solutions for your cloud setup.

b. Use the caching layer provided by your cloud’s CDN to avoid running into client issues due to non-conformance.

Let's not forget about the less intimidating compliance tests either. Are you conforming with IEEE or W3C standards? If you're configuring your own caching servers you might make a subtle mistake that causes problems or slows performance for some users. If you're using the caching layer provided by your cloud's CDN then it is unlikely they would be configured in a way that causes problems.

Tools like RedBot, which "finds common protocol mistakes", Google Page Speed or YSlow can help even further, and check on caching, content-negotiation and compression.

Check out Part II where I describe the positive effects of the cloud on the other 4 dimensions of quality assurance - Durability, Serviceability, Aesthetics and Perceived Quality.