9.0

Edit

Perform advanced configuration

HPO workflow

  1. Define an initial subset of the components.
  2. Run the evaluation process and get results (experiment).
  3. Train a neural network to change or predict components with its parameters.
  4. Repeat Steps 2 and 3 until one of the following two conditions is met:
    1. New experiments stop improving results.
    2. Time limit is met, for example, 10 hours.

HPO job hierarchy

System requirements baseline

The formula for resources requirements:

MainJob + ParallelFields * (ParallelExperiments * Experiment)

  • Memory:

    • MainJob = 16 GB
    • Experiment = 10 GB
    • 16 + ParallelFields * (ParallelExperiments * 10)

  • CPU:

    • MainJob = 2 CPU
    • Experiment = 2 CPU*
    • 2 + ParallelFields * (ParallelExperiments * 2)

  • Recommended Infrastructure requirements:

    • RAM: 28 GB
    • CPU: 7 CPU

Environment

Best Result

Max Parallel

128 RAM

32 CPU

1 HPO

hpo.maxParallelModels = 1

hpo.maxParallelExperiments = 8

1 HPO

hpo.maxParallelModels = 2

hpo.maxParallelExperiments = 4

256 RAM

64 CPU

1 HPO

hpo.maxParallelModels = 3

hpo.maxParallelExperiments = 8

1 HPO

hpo.maxParallelModels = 6

hpo.maxParallelExperiments = 4

or

2 HPO (2 Models in parallel)

hpo.maxParallelModels = 3

hpo.maxParallelExperiments = 4

Memory requirements for evaluation models

During evaluation, multiple experiments are running to define the best feature combination.

During extraction, there is already a trained model. Therefore, extraction requires less resources than evaluation.

HPO evaluation

  • A single experiment requires 10 GB of memory.
  • MainJob requires 16 GB of memory.

To define an approximate required amount of memory, the following formula should be used:

Amount of parallel experiments: (ClusterCapacity - SystemRequirements - ParallelFields * MainJob) / Experiment

  • ClusterCapacity: Total available memory size of the ML (VDS) Server/Cluster
  • SystemRequirements = 28 GB
  • MainJob = 16 GB
  • Experiment = 10 GB

128 GB Cluster: (128 - 28 - 16) / 10 ≈ 8 (experiments in parallel)

256 GB Cluster: (256 - 28 - 16) / 10 ≈ 21 (experiments in parallel)

It is recommended to use from 4 to 8 parallel experiments for each field, number of experiments and time limit could be configured.

For example, if there are 10 fields to be processed:

10*8 = 80 (number of experiments) or 10*4 = 40 (number of experiments)

128 GB Cluster: 80/8 = 10 iterations, each iteration may take up to 10 hours, up to 100 training hours.

256 GB Cluster: 80/21 ≈ 4 iterations, each iteration may take up to 10 hours, up to 40 training hours.

Fixed configuration evaluation

Once subset of the components is found, model could be fixed via configuration and during the next re-training HPO will not be used. In this case training for each field will require:

  • MainJob: 16 GB of memory
  • WorkingJob: 10 GB of memory

The approximate time is around 15 minutes for each field.

Memory requirements for Extraction models

There are two configurable parameters:

  • RAM per Job (default is 8 GB; less is not recommended)
  • Max Iinstances per Job (default is 20)

Actual requirements depend on the following factors:

  • Number of Extract job instances executing in parallel.

  • ML model size (models with 20+ fields require additional RAM per Job).

  • Number of documents sent for extraction (per hour or day).

  • SLA for one document extraction (for small SLAs, add more Job Instances).

Field extraction

Parallel extract: (ClusterCapacity - SystemRequirements) / ExtractJob

  • ClusterCapacity: Total available memory size of the ML (VDS) Server or Cluster.
  • SystemRequirements = 28 GB
  • ExtractJob = 8 GB

The extract time may vary depending on a document size and model complexity, usually it takes from 1 to 60 seconds per document for the most frequently used use cases.

For example, 128 GB Cluster: 100/8 ≈ 12 documents could be extracted in parallel

CPU requirements

Recommended configuration options:

  • 128 GB – 32 Core CPU
  • 256 GB – 64 Core CPU

CPU is shared across all jobs. It is recommended to have 2 CPU per MainJob and 2 CPU per 4-8 experiments.

For example, if 32 CPU is used on the 256 GB environment instead of 64 CPU, it may affect effective time of by 30%-40%

← PreviousNext →