In this blog post, I’ll introduce a nice approach for end-to-end testing that reduces a lot of code and time for your large-scale application.
What is End-to-End testing?
End-to-end testing, or E2E, is a testing approach based on the end user’s perspective and experience. It simulates the real user scenario by testing the whole app to ensure that the application flow behaves as expected.
End-to-end testing does not care about functions, methods, classes, etc. in your code. Rather, it covers the user’s output of your software almost completely. Besides, it can be very helpful for complex apps because they have so many dependent parts that changing one part affects lots of other parts and potentially creates bugs in those ones.
An introduction to automated testing with Cypress
Cypress is an open-source, JavaScript-based automation framework that helps you perform end-to-end testing more easily. It enables QA engineers to perform end-to-end testing without requiring extensive experience and knowledge. As Cypress does not use Selenium, it does not have the common end-to-end testing problems that Selenium-based approaches have. This is because, instead of executing remote commands through the network, it runs commands in the same run-loop as your application. You can see the main difference that Cypress creates below (the image is from the main Cypress website):
One important thing about Cypress is that it’s based on JavaScript. You can compile your code with different languages on the browser, but you just need to use JavaScript to write tests with Cypress.
One of the most interesting things about Cypress is that finding DOM elements is similar to JQuery, which makes writing tests really easy. You can access the DOM and test every part of your application. Therefore, the structure of Cypress makes it easier to work with any front-end framework or website.
One of the other goals of Cypress is to help TDD or test-driven development come true for end-to-end testing. Therefore, it provides you with a great mindset about how to implement your codes, and in fact, it makes you more aware of software output and more capable to do coding while also enabling you to test and develop your app simultaneously.
So, briefly, we can say that Cypress is a great all-in-one tool for end-to-end testing that helps developers and QA engineers test their applications much more easily and in less time. In addition, it can help them prevent lots of bugs and help the marketing and management teams have a better attitude towards application problems and scenarios. However, in this article, I won’t be tackling the configuration and installation of Cypress. If you’re curious about it, you can refer to it here.
Cypress provides you with a perfect Test Runner that will show the tests and assertions in a really nice way. It will help you understand your bugs visually. The screenshot below shows it to you:
Cypress has lots of great features and integration as below:
- API Testing with Cypress
- Run Cypress tests in a GitHub workflow
- Bitbucket integration
- Jira integration
- Gitlab Integration
- Slack integration
- Conditional testing
- Auth0 Authentication
- Google Authentication
- Working with GraphQL
- Continuous integration with CircleCI, Gitlab CI, Bitbucket pipeline, AWS Codebuild
- Component testing for React, Vue, etc
- And lots of other features
Cypress drawbacks
Cypress parallel testing: When your app grows in size, you will need a lot of tests to ensure its functionality. In other words, you need to scale your Cypress tests. However, one common problem that arises is that the duration of testing increases dramatically.
For instance, if you have written many Cypress tests and you add Cypress to CI/CD, and after every PR, you want to have end-to-end testing, then your build process may take more than an hour. If you encounter an error, you have to fix it and then wait another hour to see if it’s fixed or not. If you still have other errors, you have to fix them again and wait repeatedly. Moreover, this amount of time wasted will bother you when you need to merge something quickly and you need it to be fast.
To fix the problem above, you need an idea like “Parallelization Testing”. You have to know how to parallelize test runs and group them to decrease this time. Cypress can run tests in parallel across multiple machines since version 3.1.0, which fixes this problem. For more information, you can refer to this link.
Flaky tests and retrying: Sometimes, end-to-end testing can be really challenging due to the complexity of modern web applications. You may find that some of your Cypress tests fail sporadically, which can be really annoying because they are not real failure tests. If you haven’t worked with Cypress before, this can be especially bothersome. In the Cypress world, this kind of test is called “flaky.”
These kinds of tests may pass on a developer’s machine but fail in a CI environment, or they may pass one time and fail another time. To fix this kind of problem, you can automatically retry failed tests. You can refer to this link for more information. Additionally, Cypress Dashboard can help you detect flaky tests that run in your CI/CD pipeline. You can learn more about it here.
Some complicated scenarios for testing: Sometimes, it’s hard to test certain scenarios in Cypress, and to prove this, you need to write some complex code or sometimes there is no way at all. Examples of this include uploading, downloading, dragging and dropping, etc. There are some NPM packages that can help with some parts of these problems for these types of tests. Additionally, visual testing can help assert some parts of this without coding.
Even though writing code with Cypress reduces the amount of code, it can become challenging for large-scale applications after a while. Therefore, in the following sections, we will suggest some recommendations for this problem using visual testing.
What is visual regression testing?
One of the main reasons why visual testing has come about is that UI bugs are often not caught by test suites, but they are visually wrong. Visual testing helps with even pixel changes and provides a pixel-perfect app. Practically, it is a good substitute for manually testing all UI repeatedly, making it another step for automated testing.
If you consider automating the process of detecting visual UI changes, in fact, you want to implement visual testing. In this process, your goal is to realize if there are any unexpected changes on the UI after committing or Pull Request for specific changes on your codebase. But based on what I’ve experienced with visual testing, I’m able to say that it can be even more than detecting visual changes, and it can help you with assertions as well.
Let’s say you want to test the login form. If a user clicks on the submit button, there are two scenarios. First, there are no errors and the user is directed to the dashboard page. Second, there is at least one error, and the user receives a notification about it. So after clicking the button, and once we are sure that the API response is fetched and the UI loads, we take a screenshot. If we expect to see an error, we take a screenshot with the label “Login User with error”. When reviewing this screenshot, you should always expect to see the UI with an error notification.
But maybe you think it’s not always possible to check this screenshot, and it would be more like manual testing instead of automated testing. However, that’s not really the case. You just need to review this screenshot once and confirm it. After that, if there are any changes on the screenshot, visual testing dashboards or command lines will inform you about them, so you can become aware of pixel changes on the UI. On the other hand, in this scenario, if the user logs in incorrectly, it will also inform you, so you can do an assertion with just one line of code instead of multiple lines.
Another example I would like to explain here is if you have a drop-down and when the user clicks on the button, it should open or close based on the previous state. If you want to test this with visual testing, it would be like this:
So that’s pretty much it. Just for the first time, a master tester should check if it has been generated correctly. After that, if there is a significant change, whether it’s a UI change or wrong behavior of the dropdown, visual testing will inform us, and we need to fix or confirm that change.
However, to take screenshots and have a platform for reviewing them, we need to use Cypress plugins. The best two of them are Percy and Applitools, which I will introduce in the next parts.
Percy
Based on their site, Percy is an all-in-one visual testing and review platform. Percy handles everything from capturing and rendering screenshots to detecting and notifying your team of visual changes.
Based on their site, Applitools helps rapidly write, run, analyze, report, and maintain tests that do cross-browser and cross-device testing. It accelerates functional and visual testing by up to 30x.
Applitools seems to be a better approach for assertion and comparing screenshots, as it uses AI and provides better results for tests. For example, one of the differences between Percy and Applitools is the threshold Applitools has but Percy lacks. Sometimes, adding a few padding or margin to elements can change the pixel of the whole page, and even the whole app, which changes all the screenshots. In this situation, Percy throws errors for all the screenshots, and for example, if you have 300 screenshots, it can be shocking to see all of them fail. But with Applitools, you can specify a threshold, so it won’t throw errors for differences like this, and comparing screenshots has a much better result.
Applitools uses AI for taking and comparing screenshots, and understands the differences between common changes and unexpected changes. In addition, Applitools’ dashboard allows comparison of screenshots in a very nice way, so master testers can compare them more productively. You can integrate Cypress and Applitools from here.
With this kind of visual testing, we can remove lots of code and assertions, and utilize just taking screenshots in most scenarios, which would decrease the amount of code and time for our end-to-end testing, and help us improve our automated testing and QA significantly.
Our website uses cookies