Testing different types of browsers from the Cloud is a great scenario. Virtual machine images can be configured with all kind of browsers. Team members create an instance from an image the moment that browser needs to be tested, and throws the VM away when done. An optimal Cloud scenario.
Using Azure for a Browser Lab brings some additional capabilities and control benefits. The Virtual Machines (with the browser) that are created by team members are in control of the organization.
Images can contain preinstalled organizational software, which may influence the browser. Or company private browser add-ins needs to be validated and preinstalled on the images. There is also more control on the public add-ins and capabilities of the browser, for example should Adobe Flash be available or not.
The Browser Lab virtual machines on Azure can be connected to the corporate domain. Making it possible to validate corporate login functionality with the corporate domain. And also a connection can be made with the internal bug reporting and test case management system via a secure connection like VPN.
Using Azure for a Browser Lab brings great flexibility, pay per use. Full control of the machine configuration and secure connections to the corporate network for both bug reporting, test cases and test data and for network login and internal website validating.
The limitation for a Cloud based browser lab is the Cloud provider and what con run on it. For most public cloud providers (Azure, Amazon, and Google) the available guest operating systems are limited to Windows and Linux. No iOS platform and no mobile platforms are available. Depending on the needed Test Coverage this can be a blocker for using a public cloud provider.
Next to the limitations a public cloud based browser lab has, is that the flexibility also comes with additional work and responsibilities. Images needs to be maintained and updated when needed.
There are multiple approaches for testing the functionality and end-user experience in browsers on different PC and mobile devices. Choosing the best fit is done based on the existing design and development process and specific risks that need to be covered.
To make a balanced decision for the size of your browser lab and testing effort choices are needed. Making a full browser lab with all browser operation system combinations will get really expensive and time consuming. I think it will get to almost 1000 combinations which need used for you test run.
Many factors influence your browser test lab, only the development team with the business owner can make a proper decision. See also: Browser Compatibility testing- When, Where and how it is done
Workaround the public cloud limitations
As mentioned the public cloud doesn’t support all Operating systems, but based on the needed test coverage it can still full fill the needs for your test lab. Windows and Linux machines are supported and most browsers can be installed on it.
The workaround for the missing mobile platforms and screen scales can be the “Mobile Emulation mode” many browsers have them available via the F12 developer tools.
Chrome has a whole set of capabilities see: https://developer.chrome.com/devtools/docs/device-mode also with sensor settings.
The new Edge Browser can emulate the Windows Phone 10 edge browser.
Test automation frameworks can work with the Mobile Emulation mode and set for example the properties per test run. With Selenium the WebDriver can be put automatic in a mobile emulation mode via:
mobileEmulation.put("deviceName", "Google Nexus 5");
Map<String, Object> chromeOptions = new HashMap<String, Object>();
Taken from: https://sites.google.com/a/chromium.org/chromedriver/mobile-emulation
Still there are limitations on these emulators which needs to take in to account and should fit the needed test coverage. For example, the CPU isn’t emulated and popup menu behavior can be disabled on a device.
Most device simulators won’t work on a Virtual Machine. A device emulator need to run in a virtualized environment, while an Azure Virtual Machine already runs in a virtualized environment.
While Azure brings good capabilities to a Browser Lab for teams to work fast and flexible. But, when the needed test coverage requests all operating systems with all browsers on all type of devices the cloud has its limitations and another solution is needed.
Maintaining physical machines and configuration provides not the flexibility needed for teams. There are also other service providers who provide, probably from their own datacenter, capabilities to run a full Browser Lab on subscription base.
There are many Browser Lab providers on the internet who provide a service using a cloud based like model. http://saucelabs.com/ and https://www.browserstack.com are both offering an enormous list of available platforms and browsers.
Both providing Selenium support for automation.
While with these providers the Browser Labs loses the capability to control your own Images and configuration, they do provide a full stack of operating system and browser combinations which can be used automated.
Another interesting solution is using crowed testing for you browser lab. When using a cloud based test case management system like Visual Studio Online, crowed testers can easily connect to the repository, take a test and execute it on the device they have.
With a capability like defining multiple configurations for a test plan, testers can be assigned to execute them. Per configuration a test report can be created.
Open Device Labs: Why Should We Care?
Making a 1000+ devices, operating system, browser combination physical test lab isn’t really an option, maybe only when you have tons of money and time.
Not all tests require a full lab. Balance the browser lab size based on test coverage and use only what is needed will lower the costs of your lab enormous. For example when the public cloud can support 80% of the tests, you can use it for this 80%, the left over 20% can be consumed from browser labs providers or from a crowed sourced solution.
A cloud based test case and bug reporting system with all kind of platform support is a good way to stay flexible in your choices.
But all depends on the needed test coverage.