Perform automated testing

Approaches to software testing look similar for different applications. All levels of AI Agent testing and differences between ODF and ODF 2 are described in test automation chapters.

First, let's list all types of automatable tests for complex implementations

Automated test types

Unit tests

Unit tests are built into the code itself. They target the coding standards like how methods and functions are written. Developers usually write these tests. However, automation testers may also write them.

Executing unit tests with no exceptions means that your code will compile and run without any code issues. These tests usually don't target the application's functional aspects as they target code. It's more appropriate to automate them to run as and when required by a developer.

API tests

Applications built on the API architecture can perform API tests. In API testing, testers validate the application's business layer by checking request-response combinations for various APIs on which the application is built. API tests can also be done as a part of integration tests.

Integration tests

An integration test is a means of testing an application by integrating all the modules and checking the application functionality. Integration testing can be done through API testing or the application UI.

UI tests

UI tests are done from the UI layer or the application frontend. They may target the application functionality or its UI elements. Automating the UI to test the functionality is a common practice. However, automating GUI features is one of the most complicated automations.

End-to-end tests

End-to-end testing is the process of testing a piece of software from start to finish as used by actual users. For a web application, you start a browser, navigate to the correct URL, use the application as intended, and verify the behavior. In the case of a desktop application, you start the application, use it, and verify its behavior. If you test an API, you make calls just like real clients would do.

Testing cycle

Acceptance tests

Acceptance tests check if you are building the right thing.  They are written in terms that the business and users can understand. These tests focus on code and the system that is built. You mock out all the external dependencies. If you can't mock these out, you try to create local, in-memory, or implementations that you can control.

Smoke tests

Smoke tests are often performed in the test life cycle. These are post-build tests, usually of a functional nature, executed immediately after any build is done to ensure that the application still functions. This is a small testing suite to be executed multiple times, making sense to automate. Depending on the application type, you can pick a tool for them.

Regression tests

One of the most commonly automated test suites is the regression test suite. Regression is a test performed at the end of testing a new module to ensure that none of the existing modules are affected. It is repeated after each new iteration of testing. The main test cases stay fixed with usually a few new additions after a new iteration. As it is frequently run, almost all test teams try to automate this pack.

Security tests

Security tests can be both functional and non-functional, which involves testing the application for vulnerabilities. Functional tests are composed of authorization tests, whereas non-functional requirements may test for SQL injection, cross-site scripting, and many more.

Performance tests

Performance tests are non-functional tests that target the requirements like testing of load, stress, application scalability.

Described types of tests that can be automated eventually lead to the same test suite results being automated. ODF 2 is designed with a "testability" idea in mind.

For AI Agent automation projects, ODF provides several APIs allowing various types of automated testing. There is no strict mapping between the classical testing pyramid and our frameworks. See overall level relations in the picture below.

Unit testing

As the name implies, the phase tests individual components or units of software. Unit testing is the first phase of testing, and it can be automated. Automated unit tests are written to test the code level. Bugs are identified in the functions, methods, and routines written by developers. Developers have access to the source code and write unit tests to break the production code. Due to the presence of unit tests, whenever the code compiles, all unit tests run and tell the result if the functionality works as expected. If any unit test fails, there is a bug present in the production code.

All the above is 100% applicable for Use Cases implementation projects. Developers are recommended to add two types of automated unit tests:

1. Regular JUnit tests
2. JUnit-based Bot Task tests

Developer teams are welcome to define code coverage standards for their project. Test code-coverage validation and reporting tools like SonarQube or JaCoCo can be leveraged by development teams for transparent continuous unit test management.

note

Test-coverage tools are not part of ODF. You should add them on top.

Integration

As the name implies, an integration test means testing the application by integrating all the modules and checking the application functionality. Integration testing can be done either through API testing or the application UI.

In AI Agent implementations based on ODF, case integration testing is approached via API. ODF provides the SPOKE framework, which is API allowing to execute Business Process Bundles, mock any part of it on any Control Tower environment, and assert results in various ways.

Continuous Integration on top of test automation

Developers working in the Continuous Integration (CI) paradigm merge their changes with the main code as often as possible. That only sounds simple. Complex AI Agent implementation is composed of various components built using a multiplicity of technologies. Thus, there is a strong need for reliable frameworks for developers to operate in and integrate and validate the changes. The key component of the CI ecosystem is the test environment, which reduces the testing time by automatically spotting the most serious bugs in the early stages of software delivery. Automated code testing smoothes out the process. Frequently merging small pieces of code is a good way to avoid future conflicts. All team members have access to the latest code base and can ensure the compatibility of committed code on the go. Running regular integration testing is crucial to maintaining software consistency.

ODF provides a recommendation of CI pipeline build-out based on the stack:

• Git
• Maven
• Nexus
• Jenkins

note

The standard Work.AI infrastructure does not provide a CI server. You should configure it additionally with assistance from the IT team.

Automated testing benefits for AI Agent implementation

It is definitely an extra cost to build an environment, maintain, and run automated tests. Why do we do it? You can combine all benefits into two major outcomes:

1. Automated tests help you release faster by reducing the amount of manual testing needed for each release. That is why automated tests and CI are essential if you release more than once every two weeks. You may find that release cycles are getting longer because it takes more and more time to test increasingly complex systems. This is a signal that you need automated testing.

2. By providing immediate feedback, automated tests give developers the confidence to make changes. If you spend your time fixing numerous small things but don't have the confidence to make significant changes, you need automated testing. However, while automated tests will tell you if you break something that used to work, they are not very good at finding bugs in new features. Bugs usually come from a lack of understanding or perspective that will also be present in the tests.

Now, let's look deeper into automated testing benefits.

• A faster feedback cycle. Without test automation, feedback for newly developed features can take a while. Test automation helps you reduce the feedback cycle and bring faster validation for phases in your implementation. Test automation is especially useful because it helps you detect problems or bugs early on during the development phase, which increases the team's efficiency.

• The team saves time. By automating your testing procedure, your team has to spend less time validating newly developed features. It also improves communication with other departments like marketing, design, or product owners who rely on the results of these tests. In addition, these departments can easily check logs of automated tests and see what's happening.

• Reduced business expenses. When using an automated test environment, your company saves money as fewer resources are spent on testing your implementation. The idea is that you should not do any manual testing. Over the course of a whole project, this can make a big difference. But, of course, installing and setting up an automated testing environment takes time and resources.

• Higher test coverage. Manual testing puts limits on how many tests you can verify. Automation allows you to spend time writing new tests and adding them to your automated test suite. This increases the test coverage for your product, so more features are properly tested, resulting in a higher-quality application. Also, automated testing allows developers to write more in-depth tests that test complex Use Cases. Lengthy tests often avoided during manual testing can run unattended.

• Better insights. Automated testing provides better insights than manual testing when some tests fail. This is because automated software testing gives insights into the application and shows you the memory contents, data tables, file contents, and other internal program states. This helps developers determine what's gone wrong.

• Improved accuracy. Even the best testing engineer makes mistakes during manual testing. Especially when testing a complex AI Agent, faults can occur. On the other side, automated tests can execute tests with 100-percent accuracy as they produce the same result every time you run them.

• Automated testing provides more features. An automated test suite can help you with more features, for example, simulating thousands of virtual users interacting with your web application to see how the application behaves. It's impossible to simulate this kind of behavior by doing manual testing. Features like this save developers a lot of time.

• Less stress on the QA team. Implementing an automated testing strategy allows your QA team to spend time on tasks other than manual testing. Besides that, the job description of a QA engineer gets more interesting once you remove the repetitive element of manual testing. For many QA engineers, testing automation creates the opportunity to build new tools to optimize the current testing suite further or extend it with new features.

• Quickly determine the stability of your build. Test automation helps you to automate tests to determine the stability of your build. Often, smoke testing is used to validate stability. However, smoke testing is slow and requires manual input, such as setting up databases with test data. Furthermore, you can automate smoke testing through test automation. You can automatically generate and prepare the right databases to run smoke tests. Therefore, you can quickly determine the build stability. To summarize, the goal is to release a build as fast as possible and automatically validate its stability.

• Eliminate human errors. Manual testing opens up the opportunity for humans to make mistakes. Especially for complex scenarios, it makes sense to use test automation to avoid mistakes. You can still make mistakes, even with test automation. However, the rate of mistakes is significantly lower when using test automation for your test suite.