ML model giving different results in two environments
Machine learning (ML) models are supposed to work the same way in any environment. To figure out what is wrong when a model gives different results in two different environments, follow the steps below.
Check models and environment
The investigation must include the following steps:
To verify whether the models in two environments are the same, check the following:
- Model (trained model) names
- Pipeline (hyper model) names
- Folder sizes
- Model and pipeline versions
Ensure that the document subject to extraction is the same in both environments.
To ensure that the model works and there is no crash, check the ML model logs in the following directory:
<wf_dir>/vds-data/worker/log/automl-model-service/<G.A.V.>/ ....
Check S3 bucket
Check the S3 bucket where the document for extraction is located using the boolean isUrlToDocument(String document) function. The function can return one of the following:
trueif you send the S3 URL to the document in the input data.falseif you send a document proper.
The function can return an incorrect value if the S3 domain name is not standard or the top-level domain name is unknown to it. If you use a document link in the input data, add the following row at the very beginning of the function:
if(true) return true;
As a result, the function should look like this:
boolean isUrlToDocument(String document) {
if(true) return true;
if (!document.startsWith("http") && !document.startsWith("https")
&& !document.startsWith("HTTP") && !document.startsWith("HTTPS")) {
return false;
}
// 8190 is max uri size. That number cover most default configuration cases.
if (document.length() > 8190) {
return false;
}
String[] schemes = ["http", "https" ];
UrlValidator urlValidator = new UrlValidator(schemes, UrlValidator.ALLOW_LOCAL_URLS);
String encodedDocument = UrlEscapers.urlFragmentEscaper().escape(document);
return urlValidator.isValid(encodedDocument);
}
As you can see, the UrlValidator public external library is used above to validate the URL. The library can return a wrong result if the domain name is unknown to it.
View also: