Three sprints ago we had to deliver a story. When that particular story has been finished, we could run some tests on a Native App on an IOS device with the help of Mobile Center and UFT. One of the acceptance criteria was that it should run on Jenkins without any problems. This is a normal acceptance criterion in an agile environment. For me personally, running the test on our build server should be in the definition of done. We want to automate as much as possible to have our feedback loop as short as possible.
The story
With that user story we investigate if it is possible to use mobile center. The question if mobile center is stable enough for IOS is answered after this story. Of course before I start to automate, I do the test that I want to automate manually. That succeeded.
Now it is time to start doing the same with UFT. Because for android some code already existed, it was very easy to adapt it to IOS. After some coding, it was possible to log in on an IOS device with an UFT test. Is looks very promising. The story is nearly finished.
Problems can occur
Image may be NSFW.
Clik here to view.
After several retries on the test machine, it is time to put the test in Jenkins. Jenkins will then start the test and show the results. The first test run was failing. What happened there? After logging in, there was a spinning wheel, that was there for ages. Because the timeout of the test was only 30 seconds, the build failed. After that, I retried it manually, and the login was going well and fast.
I retried the test again with Jenkins. This time the build succeeded. What is that? Something strange is going on. I discovered something after some investigation. If the applications is running and the login is performed again, in that case the Jenkins job will succeed. If the application is not started, then Jenkins has to start the application. In this case, the build fails because of the hanging login procedure.
We where thinking about fixes. What could be solutions?
- Is it a problem with the application?
- Is it a problem with Jenkins? Maybe the plugin for UFT/Mobile center?
- Is it an IOS only problem?
- Some timing problems?
How to solve the problem
Also on android?
Let’s try the same test on android. Is that failing as well on Jenkins? I changed the device and launched the job on Jenkins. After a while, it was clear to me. The test will go green. So the problem is not present on android devices. That narrowed the search. And I could look only at IOS from now onwards.
Was it a timing problem?
I placed some sleeps after the clicking of the buttons of the login. But that did not solve the problem. It was always the same result, a failing build job. I abandoned this idea.
Jenkins plugin problem?
Maybe it was a problem of the HPE plugin in Jenkins? How can I see if the plugin is the problem? Write our own Visual Basic script that starts the test. After some time, the script was stable enough to launch the test. Let’s try it on our build server. After a big change, the first press on the build link in Jenkins is always exiting.
Is it going well? Is it failing? After some minutes, the result of the test was visible. The build was again not green. Some retries confirmed this. The build never gets green when the application was not running before. Just like before. The problem is still not fixed.
Do other Apps have the same problem?
Maybe it was an application problem. So I tried to automate another Native Application. The first application I tried was an application where I only needed to click on. Logging in was not possible. This time, the Jenkins job was passing. For me that shows that there is not really a problem with the infrastructure. Is must be the application.
After this application, I tried another application. In this application it is possible to log in. Again, I tried to log in manually. This succeeded. I created a script that could log in with HP UFT and Mobile Center. The next step now is running this test on the build server.
I was confident now that the test job will succeed now. Because it is the app that has a problem, no? Again, I was wrong. The first time the build was already failing with exact the same result as with the original application.
Really, a solution?
I start changing the timeout of a function check_label_exist. This function returns True or False if a label is present on the screen. Is has a parameter, a timeout. If after the timeout the item is not present, the function will return False.
I noticed something. I tried a timeout of 5 seconds. The test on Jenkins failed. After 7 seconds, the login succeeded. But because 7 is greater than 5, the test failed. So I increased the timeout to 10 seconds. Now the login succeeded after 15 seconds. Is this coincidence ?
I tried more timeouts, and saw always the same result. The login is finished after the timeout.
At that time it was time for our standup. I explained this very strange behaviour. Some ideas came out the heads of the team members. One of them was to create our own loop. Maybe that is worth a try.
The hours after the standup, I created an extra loop around the check_label_exist. This time, our build server shows a better color as result of the test run. The test was Green!! I was hopeful. I retried it again. Again a build that is successful. Each time I tried, the build was green.
After refactoring the code the build still was succeeding. The loop is now inside the check_label_exist function. So now I am very happy. The build is still green after 11 retries. The solution is found and our story is finally finished.