Codeless Automation: Framework and visual testing, Gherkin, and version control- by David Lay

In software testing, there are tests that need to be run frequently. Test automation is good for that. Test automation is useful for monitoring the behavior of systems with thousands of features, which would be time consuming to run manually. Automated testing is also great for tests that need to be run frequently, simple tests, tests that use multiple data entries for the same application and critical items where frequent testing using the same variables is needed. Automated testing can greatly speed up the testing process and free up people on the team to do other tasks. Automated testing is by no means a one size fits all solution, however. There are situations where automated testing is not ideal. Any test case in which setting up the automation takes longer than just testing it manually would take is not a good test for automation. For example, if there was an exceptionally complex test that only needs to be run a few times a year and takes about 2 hours to run manually versus a few days to automate, this would be a test where manual testing is preferable. Similarly, you would not want to automate a UI level test if you can go further down into the system and automate at the unit or even the service level.

There is a hierarchy as to which tests are ideal for automation. Think of it as a pyramid. At the bottom is the unit level of tests. Automation is most preferable at this level. In the middle is the service level. This is a higher level up and is less ideal for automation. At the very top of the pyramid is the user interface (UI) level. This level is automated least often. The unit level is pretty quick to automate and desirable to run after each new build. Unit level tests should be run first. Service level test are slower and more complicated than unit tests so they aren't automated as often but the are still faster and more reliable than UI tests so you will still want to use them and there may be times in which you will want to automate them. Tests at the UI level are extremely complicated and brittle and they need to be changed every time the UI changes, so automation of UI level tests is ideally done as little as possible, and instead it is vastly more preferable to automate at the unit or service level.

Automation has benefits such as increasing a company's return on investment (when it is appropriate), preventing human error from getting in the way of test results, speeds up the testing process dramatically, allows for better code refractoring, makes cross browser testing easier, and allows for a greater variety of test data than could be generated by a human.

Converting manual tests into automation utilizes tools. Before any test can be automated, it needs to be run at least once manually. This is needed in order to ensure a test will work. Once this has been done, the method of automation can be chosen. Cucumber is one tool which uses the language Gherkin. Gherkin is a simple programming language easy enough to understand by non programmers and uses operators like given (the precondition), when (the action) and then (the result). Additional operators can be used such as and or but. And includes another condition and but is an exclusionary operator to preclude a condition. Such as Given user is on Home Page, And the user is logged in, When the user clicks the log out button, Then the user is logged out. Gherkin automation cases use the given, when, then structure. The and and but operators are optional. A browser plugin called Selenium IDE is another tool. It uses data that is recorded. The user sets Selenium IDE to record and then runs the test manually. Sikuli is another tool that records tests in a similar fashion. The difference between Selenium IDE and Sikuli, however, is that Selenium IDE works with the source code to perform the steps while Sikuli reads the visuals on the screen. Sikuli is useful for test environments that are closed source (meaning the source code is not accessible) such as Flash files or tests that are done outside of the browser or that require a lot of clicking around. Sikuli has the weakness that it is visual dependent, so if there are any changes in the visuals of the environment, the automation needs to be done all over again. Cypress is a framework tool that can create and run test cases with programming language. It is mostly used for testing front end web apps. It only runs in Chrome. Applitools works with Cypress or Selenium to do visual testing.

Roles of test automation can be summed up to the three amigos, otherwise known as a specification workshop. The three amigos are business, development and testing. The role of business or is the person who defines the problem. Development involves coming up with solutions to the problem. The testing role is to question the solution.

Shift left is a model of agile software development that solves the issue that there can be many cycles of code fix that still does not satisfy the requirements by the time the sprint ends, which can sometimes happen with the traditional top down model. Rather than starting with the product owner moving to the developer and then moving to the qa engineer trying to get all the bugs before the sprint ends, shift left has the product owner, developer, and qa engineer all translate user stories into feature files. The QA engineer will write automated reusable tests at the same time that thedeveloper codes new user stories from the feature files, which can be the basis for unit tests. The qa engineer's automated tests can be used by the developer to validate code. This process ends with finished code faster and more efficiently than working with the traditional top down model. Shift left has everyone working together rather than feeding from the product owner, down the ladder. This process helps everyone to deliver the right thing faster. One method of shift left is behavior driven development, or BDD. BDD is centered around the Gherkin language for the development of acceptance tests. Gherkin is a simple enough language that it is understood by both programmers and non programmers alike. Gherkin can be used for writing feature files. The Cucumber tool can be used to generate glue code from Gherkin feature files. Manual, smoke, integration, regression and multiple repeating end to end tests are great uses for BDD. BDD is not good for exploratory testing, contract tests, performance tests or load tests. Although BDD can be used for API tests, it is not ideal. Gherkin is the language using the given-when-then syntax. In addition to the given when then gherkin code, there is also a Java file written by the automation engineer as a to hold the automation code together. This often has the definitions of the steps which are mapped to the Gherkin feature file. Where there is a step definition file, the engineer will aos add a runner file to tell Cucumber where to find the feature files and step definition files. The step definition file and runner files make up what is known as the glue code. User stories written with Gherkin start with the feature and background, followed by the scenario. The feature has the name after Feature:. The background will have a precondition started with a Given statement. This is followed by the scenario, which is written in the usual Given-When-Then syntax, with the possible inclusion of And and/or But if needed. The scenario describes what should happen. To summarize the terms, the feature is a description of the desired outcome of the scenario, the background is descriptive text adding context to the scenario, the scenario is the particular situation to be run and the scenario is broken down into the Given-When-Then syntax formula. Given is used to denote a precondition. When is used to denote the action or operation performed by the user. The expected outcome is Then. Both the When and Then sections can be followed by And and But operators, but these are not necessary for the test to run. And is an operator for adding an additional action (When) or desired result (Then). But is an exclusionary operator to suggest an additional action (When) is not performed or an additional result (Then) is not achieved. The core keywords for a scenario operation are Given, When and Then, because these are needed in order to run the scenario. And and But are not needed, but can be used if the testing parameters require more specificity, so they aren't considered core operators.

Sometimes one might have a large number of values to use for a scenario and it may become tedious to keep copynig and pasting the scenario to use all of them, so a scenario outline may be more preferable to simplify things. A scenario outline is a placeholder template that can store all of the values to be tested in a table. The scenario outline has steps that are never directly run, but after naming all of the variables and placing each variable value to be run in an Examples table directly below, the outline can then be run.a scenario outline can save time and effort when there are many variables to be tested. A scenario outline will work so long as the exact tests that need to be run are identical for each variable.
There are different team members on a test automation project. The product owner or business analyst, the developers, and the testers. This is not too different from the three amigos division of roles. The business analyst or product owner can decide which features are to be developed. The product owner knows which features will be tested most frequently, which is also helpful in determining which tests to automate. Developers contribute to test automation as well, as they can create unit tests. If the automation is done right, developers can move faster with their feature development. It should be noted that even with a successful automation project, there is still a need for manual testers. That's where the testers come in. But manual testing is not all they do. The testers have great insight as to automation in that they know which tests to automate and which areas will need the most coverage by automation. Automation does not replace the job of the testers and they are the ones who can discover the most critical of bugs. Testers also work on triaging and maintenance of test scripts.

Considerations for test automation are the tool and the programming language. A tool should be chosen based on who is using it, available support and resources and compatibility with required programming languages, devices and browsers. When selecting a programming language for automation, considerations could include that the developers use the same language, it's easy to learn, and there is good documentation available. A deterrent for a programming language would be if it is unsupported by most test automation tools. Some ways automated tests can interact with a test application.

When designing an automated test suite, one should think about creating tests that can be used again and again and needing to change as little as possible. This can be thought of as futureproofing automated tests. The first way to future proof is running in parallel. Rather than running the tests in a sequential order, this is running more than one test at once. However, this can only be done with tests that are not reliant upon other tests in order to run. Designing tests that can be run independently of one another allows for the use of having multiple tests run in parallel. Another way to future proof tests is clean coding. This means keeping the code short, avoid duplicate code snippets, or programming inefficient waiting within tests. Seeing as automated tests are software that needs regular maintenance, clean code cuts down on maintenance time. Lastly, become familiar with the design patterns. Stick to using the ones that make the most sense for your project.
When you set up an automated test it's important to scale your test or determine the scale of testing that will be done. How many different environments, devices and browsers will be tested? when determining the scale for different environments, environment specific data should be taken into consideration. When choosing browsers to run tests on, the way to approach this is to consider what browsers are needed most for the business needs of the project. You don't always need to test on every browser. Mobile test automation is a bit more complex. Which device operating systems can your automation system run on? iOS, Android or both? Is it feasible to automate on both? What apps are developed by your organization? What devices do you have on hand? Is it more feasible to automate on physical devices or using a cloud service? Are the mobile apps distinctly different from the desktop browsers version? What automation tools are appropriate for mobile automation and what tools do you have on hand? All of these factors are crucial in determining the scale of the test automation.

Git and GitHub are used by coders and software developers. Git exists for version control and collaboration. It has advanced and sophistocated features for collaboration. For example it output of can allow more than one person to work on the same document at once. Git and GitHub aren't the same thing. Git is the actual version control software. Github is a web server that usually works with projects managed with Git. Git is the application while GitHub is a website. You can use one without ever having to use the other. A github repository is a project. It can have multiple files associated with it. The commit button for new files is like a save button. It means I will change the file and make a commit.
Here is a test poem that I made using GitHub to demonstrate the branching that occurs when commits are made on new branches versus on the main or previous branch https://github.com/davidlay85/DLTestpoem1/commits/main
Version control is keeping track of the entire history of things you are working on on the computer. Git and Github do this. All of the previous versions of the file are accessible in Git and GitHub.
The main linear list of commits is on the master branch. A user can continue and make commits on another branch.

Branches may be used to keep certain changes separate from the main branch. A pull request is used to merge a change (or commit) from a new branch into the branch from which the new branch came, or possibly another branch. Even if someone is not a programmer, someone like a QA tester for example, might find GitHub useful for keeping track of the changes that are made by the development team.
Visual testing is for checking the visuals of a test environment and comparing it against the expected visuals. This would be something you would want to use in cases where the web UI automated tests (non visual) in programs like Selenium may show that the site is working properly but due to an unexpected layout, something is not accessible by the user. An example of this could be that an automated test runs the code and shows that all functions work but due to poor layout, the add to cart button on an item in an e commerce store is covered up by another element, so it can't be accessed. Visual testing would be used to pick up on this.
Web based test application < browsers (make sure it works on different browsers) < Selenium web driver < programming framework ie Java or J Unit (python or other frameworks can be used) < version control application like Git < project management tool like Maven < integration environment such as Jenkins. Selenium and Selenium IDE are automation tools or web drivers, J Unit is more like a test execution framework (as is Test NG), Git is a version control system, Maven is a project management tool and Jenkins is a continuous integration environment that can trigger the test in different conditions.

Sikuli is program for screen automation rather than digital automation. Screen automation uses images to perform its automation rather than digital data. This would be used for situations such as if someone needed to automate a Flash program, since the average digital automation tool has difficulty automating things like Flash which is closed source. Screen testing tools like SIkuli are better suited for complicated tasks that are more visual, test environments outside of the browser realm, or closed source environments like Flash or Swift files or things without an API file that require clicking around to use. A drawback is that if the visuals change in an application or test environment, the automation needs to be done all over again because it looks for exact images. Changing visuals throws that off completely, so the test automation will need to be configured all over again.

Codeless automation, while technically not completely codeless, is easier to learn for non programmers than coded automation. It isn't a replacement for manual testing, but in applications in which it can be used, it will make the testing process go considerably faster.

-- DavidLay - 01 Dec 2020