In collaboration with Manas Pandey.

Introduction: 

“Quality at Speed” is the new mantra for organisations and demand from customers. It means high quality product must reach end-users but in the shortest time span possible. This is an important spoke in the wheel for reducing time to market for customers as well. Hey-Maas fits right into this equation. It is a framework that completely runs on APIs, giving it a capability to run End-to-End complex automation scenarios, in a short time possible.

“Ease of use” may be invisible, but its absence isn’t. Hey Maas framework is very easy to use making it practically anyone with little/no knowledge of programming to run it with ease. It can be used by customers for User Acceptance Testing (UAT). The framework is well integrated with Command Line Interface (CLI) along with easy executions with using developer IDEs like Eclipse and IntelliJ. In the course of this blog, you will learn how to setup and run the automation suites with aforementioned modes.

Problems with the legacy UI Framework:

Every organisation or a team within the organisation has a right to choose the tool that would fit their day-to-day automation executions. For Apple portal, choosing a UI automation framework using Selenium, Appium tools was an easier prospect to automate our test cases, but it had lot issues on the flip side. Some of the problems we faced were:

1.    Time it took to execute a suite: in fact, every single script which was a combination of selenium and Appium took on an average around 10 to 15 mins to execute. This means our sanity suite took around 1 hour to execute and confidence took approximately 3 hours to execute. On a production elevation day, it used to take around 4-5 hours just to finish sanity and confidence.

2.    Failure Rate: Since the Selenium-Appium tools are more prone to failures compared to API executions, our pass percentage for sanity was around 60% and confidence suite was less than 50%. Failures where for various reasons such as object load lag on the environments, network issues, Proxy errors, device getting disconnected, unforeseen prompts on the Apple device etc.

3.    Frequent version upgrade of the tools:  With every iOS major version release, the Appium version had to be upgraded which landed us into lot of compatibility issues with Appium-XCode-iOS combination.

The Solution:

We analysed the Customer Platform Services (CPS) data over past 2 years and found a very interesting fact - more than 90% of the customer issues were dependent on the way we implemented the feature within our tomcat/batch apps, rather than the end user Apple devices. Invariably, if right configuration of the workflow was pushed to the device, then the workflow worked as expected on the device in most cases.

The above fact made a strong case to remove physical device dependency, along with portal UI dependency and to go for API driven automation framework, with a virtual device enrollment.

Hey-Maas project:

What started out as a MGR idea soon transformed into an action item and need of the hour for Apple automation. Hey-Maas is a simple API only driven framework which uses HTTP Client and Rest Assured java library for API executions, it also used Junit for assertions, It is well integrated to Reports Portal and Slack, for daily executions it is also integrated to CICD Jenkins Pipeline.

The most unique feature of Hey-Maas is it is well integrated with Command Line Interface (CLI) which means using customised set of commands we can trigger automation from Command Prompt, giving even a novice in automation testing an opportunity to run it hassle free in their local machines too.

How to run it from CLI:

Please follow the below steps to run it from CLI:

1. Clone your project to local machine.
2. Build the pom.xml. (In the command prompt run mvn clean package from the repository path)
3. Keep handy the customer id, instance name, billing id and password.
4. Go to the repository folder and run the following (replace <> with the required value):   
   java -jar target/hey-maas.jar <suitename> -e <environment> -u <username> -p <password> -b <billingId>
5. That’s it, Wait for the suite to complete!

On the similar lines, we can run the suite we desire. Suites currently available are:

How to run from Eclipse IDE: 

Please follow the below steps to run it from eclipse IDE:

1. Download and install latest JDK.
2. Download and install latest eclipse IDE.
3. Goto File -> Import ->Existing Maven Project.
4. Once imported, locate pom.xml and do a mvn clean package.
5. Locate App.java, right click go to RUN-> Run Configurations -> Add the following in Java Applcation -> App

run-app-suite -e <environment> -u <username> -p <password> -b <billingid> -rp <reportportal> -rpa <reportportal attributes>

Successful suite run on eclipse:

How to run from Jenkins:

Please follow the below steps to run it from eclipse IDE:

1. Open the Jenkins url and goto Apple -> Apple-virtual-device-sanity.
2. Click on Build with Parameters.
3.  Enter the user name, password, billing id and report portal status.
4. Click on Run, That’s it!

Jenkins pipeline progress of various suites :

Other Features and Integrations:

Reports Portal: 

We have also integrated reports portal which Is an AI driven Test Automation Dashboard. This dashboard gives various insights on how the suites run, passing rate summery, Launches duration, Overall Statistic etc.

Report Portal Dashboard:

Slack Integration:

We have also successfully integrated slack alerts of every suite we run, as this gives us an update of test cases status on a suite to suite basis. Making the automation results accessible to everyone who is present on a channel.

Return on Investment (ROI):

Automated testing is an important aspect for any business to deliver a good return on investment (ROI). The automation ROI helps in finding out whether the change in automation strategy is worth the risk an organisation should take. With Hey-Maas framework we have tried to answer the ROI problem to the best of our abilities, here are the few noteworthy investment returns we could achieve:

1.    The Time factor:  We were able to reduce the sanity and confidence suite execution time drastically from 3 - 4 hours to 1 hour.

2.    The Cost factor: Since our older UI framework was device dependent, a lot of money was used to procure newer MAC Machines and iOS iPhones and iPads. With the Hey-Maas framework we effectively eliminated the physical device dependency by performing a virtual device enrollment. This saved a lot of device procurement cost.

3.    The Pass efficacy factor: Compared to legacy UI automation framework which had pass percentage of less than 50%, With Hey-Maas framework the pass percentage of sanity and confidence stands close to 90%.

4.    The Ease-of-use factor: We all know how challenging it is to execute suites which are selenium and Appium driven. But with Hey-Maas framework we have introduced a contract testing logic where even a non-QA resource can easily pick the automation suite and run it according to his use.