AutoML QA process
The QA process for AutoML consists of three main parts:
- End-to-end tests on the e2e production-like environment
- Sonar static code analysis
- BlackDuck scan for security and licence issues
End-to-end tests
All e2e tests are run using AuloML Services REST API to mimic external AutoML communication.
The e2e instance is reinstalled every two weeks with the latest PSI version.
All set of tests and runs could be found here: https://jenkins.workfusion.com/job/automl/job/model/job/test-e2e-automl-models/
Smoke tests, Fast tests
Consist of Presentation models (Classification and IE) + All OOTB models run on small data sets (e.g. 200 docs for IE and 700 for classification)
Each test checks expected models quality.
Performance/Load tests
Performance and Load tests execute All OOTB models on large data sets, for example, 2K docs for IE and 7K for classification.
Each test checks expected models quality and whole bunch is limited up to five hours.
Benchmark tests
Benchmark tests are run using JMH tools and generate performance report per component (Annotator, FeatureExtractor) and asserts predefined limits.
Integration tests
The set of local unit tests which conduct local model training with additional checks for corner cases.
Sonar static code analysis
We use SonarQube to perform static code analysis.
Sonar analysis is performed automatically on each PR created in BitBucket.
Current develop branch status could be found here: https://sonar.workfusion.com
Main rule all quality gates should be passed, no Blocker or Critical issues.
BlackDuck scan for security and licence issues.
To check dependencies versions for security and licence issues we use BlackDuck.
The main rule: no High or Critical issues.
Current state of BlackDuck scan is here: https://workfusion.blackducksoftware.com