10.2.7

Edit

File storage

ODF 2 can work with Amazon S3-compatible file storages. Under the hood, it uses AWS Java SDK, though with a more convenient high-level API.

Low-level API

All interactions with the S3 storage are via the com.amazonaws.services.s3.AmazonS3 class that is part of AWS Java SDK. To get access to the preconfigured AmazonS3 client, inject com.workfusion.odf2.service.s3.S3Service and call the getAmazonS3() method. Alternatively, you can create the AmazonS3 client with your own settings.

@BotTask
@Requires(S3Module.class)
public class S3ServiceTask implements AdHocTask {

    private final AmazonS3 amazonS3;

    @Inject
    public S3ServiceTask(S3Service s3Service) {
        this.amazonS3 = s3Service.getAmazonS3();
    }

}

You can use S3Module or aggregate the default ControlTowerServicesModule modules to get access to those services. The module is called from the generated Bot Task code, where the platform service wrappers are provided as injectables.

note

By default, S3 credentials are taken from the IA Cloud instance. When running a task from IDE, provide the credentials manually—in a test code or IDE properties if you use Studio.

To customize the AmazonS3 configuration, you can create your own instance of the AmazonS3 client the same way S3Module does it:

@Provides
@Singleton
public AmazonS3 customAmazonS3Client(Scraper scraper, BindingReader bindingReader) {
    S3ConnectionProperties connectionProperties = s3ConnectionProperties(scraper, bindingReader);
    AmazonS3ClientBuilder defaultBuilder = amazonS3ClientBuilder(connectionProperties, bindingReader);
    // apply custom settings to the builder here
    return amazonS3(defaultBuilder);
}

For more details, refer to the source code of the com.workfusion.odf2.service.s3.AbstractS3Module class that already contains predefined builder methods for that.

High-level API

You can avoid using the low-level API for most standard cases. Instead, you can inject an instance of com.workfusion.odf2.service.s3.S3Service. The API provided by this class hides the complexity of underlying AmazonS3 and provides methods for common cases.

S3Service is quite small. It contains the following public methods:

  • getBucket(String bucketName) gets an API object that represents a specific S3 bucket.
  • getObjectByUrl(String url) retrieves an object as a byte array from S3 by a URL.
  • parseUrl() processes a string with the S3 URL extracting relevant parts from it.
  • getAmazonS3() gets the configured AmazonS3 client associated with S3Service.
final S3Bucket myBucket = s3Service.getBucket("my-bucket");

final byte[] documentData = s3Service.getObjectByUrl("https://s3-instance.com/documents/doc.pdf");

final S3Url s3Url = s3Service.parseUrl("https://s3-instance.com/documents/doc.pdf");

final AmazonS3 amazonS3 = s3Service.getAmazonS3();

The com.workfusion.odf2.core.webharvest.service.s3.S3Bucket instance is used to manipulate files within the corresponding bucket. It contains the following public methods:

  • getBucketName() returns the current bucket name.
  • get(String s3key) retrieves an object by a specified key from the current bucket.
  • put(byte[] data, String s3key) uploads a new object to the current bucket.
  • delete(String s3Key) deletes an object by a specified key in the current bucket.
  • listObjects() returns a list of summary information about objects in the current bucket.
  • copy(String sourceKey, String destinationKey, CannedAccessControlList cannedACL) copies a source object to a new destination within the current bucket.

Download binary content from S3

final S3Bucket myBucket = s3Service.getBucket("my-bucket");
final byte[] fileContent = myBucket.get(document-key);

Upload binary content to S3

final S3Bucket myBucket = s3Service.getBucket("my-bucket");
final byte[] fileContent = "some content".getBytes(StandardCharsets.UTF_8);
myBucket.put(fileContent, "/some/file-name.txt");

When using this method, you can guess the MIME type of an uploaded file from its filename.

Upload binary content to S3 with additional options

An overloaded version of put() allows you to specify additional options and properties for the file upload.

final S3Bucket myBucket = s3Service.getBucket("my-bucket");
final byte[] fileContent = "some content".getBytes(StandardCharsets.UTF_8);
myBucket.put(fileContent, "/some/file.name", 
    Optional.of(CannedAccessControlList.PublicRead), // The access control list for uploaded file can be specified here
    Optional.of("text/plain"),                       // The MIME type can be explicitly specified here
    Optional.of("inline"),                           // The Content-Disposition HTTP header for the file can be specified here (see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition)
    Optional.of(Duration.ofHours(6)));               // If this parameter is specified, presigned URL with specified expiration time is generated for the file.

List objects

To list all keys in the current bucket:

final S3Bucket myBucket = s3Service.getBucket("my-bucket");
final List<S3ObjectSummary> objects = s3Bucket.listObjects();
final List<String> keys = objects.stream()
    .map(S3ObjectSummary::getKey)
    .collect(Collectors.toList());

To restrict the response to keys beginning with the specified prefix:

final S3Bucket myBucket = s3Service.getBucket("my-bucket");
final List<S3ObjectSummary> objects = s3Bucket.listObjects("my-prefix");

To get more control over the request settings, use com.workfusion.odf2.service.s3.S3ListObjectsRequest:

final S3Bucket myBucket = s3Service.getBucket("my-bucket");

final S3ListObjectsRequest request = new S3ListObjectsRequest()
        .withPrefix("prefix")
        .withMaxKeys(100)
        .withBeforeModifiedDate(Date.from(LocalDateTime.of(2022, 1, 1, 18, 0).toInstant(ZoneOffset.UTC)));

final List<S3ObjectSummary> objects = s3Bucket.listObjects(request);
note

As S3 buckets can contain a virtually unlimited number of keys, the complete results of a list query can be extremely large. That is why Amazon S3 usually limits each response to a maximum of 1,000 objects. S3Bucket provides built-in pagination that tries to return all objects within a bucket. Use this feature with high caution as S3Bucket doesn't know how many objects are stored inside a bucket; therefore, the listObjects method might take an unpredictable amount of time.

To get all objects from the bucket using pagination, set requestAllKeys to true:

final S3Bucket myBucket = s3Service.getBucket("my-bucket");
final S3ListObjectsRequest request = new S3ListObjectsRequest().withRequestAllKeys(true);
final List<S3ObjectSummary> objects = s3Bucket.listObjects(request);

Copy object

final S3Bucket myBucket = s3Service.getBucket("my-bucket");
myBucket.copy("sourceKey", "destinationKey", CannedAccessControlList.Private);
← OCRSecrets Vault →