Edit

Run Search Engine

Search Engine is a framework for optimizing hyper model parameters and finding an optimal set of components: Annotators, Feature Extractors (FEs), Post-Processors, algorithms, and their parameters for model training.

Here are the best prerequisites for a Search Engine experiment:

• To run Search Engine, use a set of gold labelled training data.
• In many cases, it's better to have a baseline implementation to optimize. The model should be able to train and produce results using the training data set described above with an arbitrary set of FEs.
• One of the primary purposes of Search Engine is to find an optimal subset of FEs from an initial superset. Search Engine uses the initial FE set as the candidate space to search and find the best subset of available FEs.

Algorithm details

Search Engine is a proprietary implementation of the best approaches from the recent research around the hyperparameter optimization problem.

At a high level, Search Engine performs the following iterations:

1. Trains an approximation function of the hyperparameter and FE space. A deep learning model is used as the approximation function.
2. Conducts MCMC sampling against the approximate function using Metropolis-Hastings acceptance testing.
3. Executes actual experiments with the base ML model using the top-scoring results found by sampling against the approximation function.

These steps are repeated for a number of iterations until no better result can be found.

Approximation function

In most cases, it is impossible to run a full grid search of experiments due to the ample space of hyperparameters and FEs.

tip

If a native (brute force) search includes 20 FE choices with 5 model parameters that can each take one of three values, the output would be 2^20 + 3^5 = 1,048,819 experiments.

Obviously, this is way too many experiments to run. Even if each experiment took only one second, this would still require over 12 days of computing time. To get results within limited reasonable time, a function is used to model the full space of FEs and hyperparameters. Then, the same function is used to search within the space.

Initialization

To use the approximation function, you must train it with samples from the full search space. To do this, you obtain a number of initial random samples using the base ML model. Each initial sample uses a different random subset of FEs and thus gets different scores. You can represent each initial sample as a binary vector, where each dimension corresponds to one of the FEs. The "on" dimension of the binary vector represents an FE used for the current sample, whereas a zero dimension corresponds to an "off" FE that is not used in the current experiment.

After you perform a number of initial seed experiments, you train the approximation model using the binary vector representations that correspond to each experiment with the base ML model score as the dependent variable.

Deep Learning Model

Currently, a multi-layer perceptron is used as the approximation function since the search space is highly non-linear. The deep learning model may change in the near future.

Metropolis-Hastings sampling

The approximation function is trained with the last set of samples obtained from the model. It generates a preset number of random binary vectors and scores them against the approximate function. The top N of these samples are selected for MCMC sampling using the Metropolis-Hastings (MH) algorithm. Each sample consists of flipping one dimension of the binary vector from "on" to "off" or vice versa and then scoring against the approximation function. As per the standard MH algorithm, all samples that result in higher scores are immediately accepted, and the samples that result in lower scores are only accepted with a certain probability.

Simulated annealing

Simulated annealing is used to speed up the search via sampling. It allows the search to jump in larger steps to a different area of the space by raising the acceptance probabilities by "temperature". This results in more acceptances relative to the current "temperature". After each iteration, the "temperature" is dropped to reduce the size of sampling steps around the search space.

Running Search Engine

Search Engine experiments run in a distributed cluster that can be hosted in a cloud or on premise.

Search Engine output

Search Engine experiments produce a number of files with optimization details. Pay attention to the output to the <path/to/results>/work/pre-eval/ parent directory.

note

Note that there may be subfolders under this directory if multiple models are optimized in one Search Engine run.

• statistics.csv contains per-iteration statistics, such as top score, average score, standard deviation, and so on.
• extractors.json is the current best set of FEs in JSON format.
• parameters.json is the complete serialized configuration used for the current best experiment, including Tokenizers and Annotators.
• scores.csv is the "on/off" configuration of all experiments with the corresponding score from the approximation function.
• best.csv or best.json is the "on/off" configuration of FEs used for the best experiment. Each "on" FE is listed in extractors.json.

Finally, after the Search Engine cycle is complete, there is an output directory located at <path/to/results>/output. The directory contains the following artifacts:

• model is a directory with the final best model.
• avg-evaluation-results.txt is the P/R/F1 evaluation score for the best model.
• evaluation-results.zip is an archive of the detailed evaluation output for the best model. You can load the output to Tableau.

Search Engine results

Search Engine Results
Data set information	Document type	Invoices
	Structure details	Contain tables
	Number of documents	102 training and 45 test instances (test set not used)
	Fields of interest	Invoice number Total amount Currency Supplier name Supplier site Date of sale Bill to entity name
Search Engine configurations	Number of initial experiments	(usually set to 8)
	Batch size	(Usually set to 8)
	Maximum number of experiments "maxNumOfExp"	(Default: 10000)
	Maximum experiments with the same best score "maxExpWithSameBestScore"	(default: 250)
	Time limit "timeLimit"	(default: 0)
	Target score "targetScore"	(Default: 1)
	Scoring function "calcScoreFunc"	(Default: average F1 score "AvgF1Score")
Experiments and cross-validation stats per field (ran on Mesos)	Experiment 1 parameters	maxExpWithSameBestScore = 100, timeLimit = 1 * 60 * 60 (1 hr), all other parameters are set to default values.
	Experiment 2 parameters	maxExpWithSameBestScore = 100, timeLimit = 5 * 60 * 60 (5 hrs), all other parameters are set to default values.
	Experiment 3 parameters	maxExpWithSameBestScore = 100, timeLimit = 10 * 60 * 60 (10 hrs), all other parameters are set to default values.
	Experiment 4 parameters	maxExpWithSameBestScore = 100, timeLimit = 10 * 60 * 60 (10 hrs), maxNumOfExp =100, all other parameters are set to default values.
	Experiment 5 parameters	maxExpWithSameBestScore = 100, timeLimit = 10 * 60 * 60 (10 hrs), maxNumOfExp =10, all other parameters are set to default values.

F1 results dependence from certain experiment parameters

timeLimit parameter

Field	timeLimit, hours
	1	5	10
invoice_number	0.74251497	0.804469274	0.808988764
total_amount	0.633540373	0.646706587	0.715151515
currency	0.926553672	0.961748634	0.950276243
supplier_name	0.589041096	0.590604027	0.631578947
address	0.518518519	0.533333333	0.630630631
date	0.826815642	0.829545455	0.89010989

The charts are as follows:

maxNumOfExp parameter

Field	maxNumOfExp
	10	100
invoice_number	0.815642458	0.813186813
total_amount	0.617283951	0.695121951
currency	0.961748634	0.961748634
supplier_name	0.559440559	0.617449664
address	0.533333333	0.592592593
date	0.837988827	0.895027624

The charts are as follows:

Experiment statistics

Experiments 1-5

Experiment 1

TAG	EXTRACTED	GOLD	CORRECT	PRECISION	RECALL	F1	PRC_AUC	RPC_OVER_80_AUC	RPC_OVER_90_AUC	ROC_AUC	MSE
total_amount	60	101	51	0.85	0.504950495	0.633540373	0.747931948	0.118559052	0.056673241	0.647293447	0.222599166
supplier_site	37	71	28	0.756756757	0.394366197	0.518518519	0.880376984	0.129960317	0.061833333	0.825	0.149624729
supplier_name	49	97	43	0.87755102	0.443298969	0.589041096	0.695292837	0.101673789	0.043942308	0.563333333	0.173916082
invoice_number	67	100	62	0.925373134	0.62	0.74251497	0.574978014	0.097479957	0.047755439	0.504545455	0.225837091
date_of_sale	80	99	74	0.925	0.747474747	0.826815642	0.782082492	0.142082492	0.063954185	0.675	0.227778821
currency	84	93	82	0.976190476	0.88172043	0.926553672	0.372999006	0.05445734	0.019427578	0.163461538	0.43010207

Experiment 2

TAG	EXTRACTED	GOLD	CORRECT	PRECISION	RECALL	F1	PRC_AUC	RPC_OVER_80_AUC	RPC_OVER_90_AUC	ROC_AUC	MSE
invoice_number	79	100	72	0.911392405	0.72	0.804469274	0.959130442	0.162970491	0.075460565	0.757417582	0.30762344
total_amount	66	101	54	0.818181818	0.534653465	0.646706587	0.750566848	0.119770644	0.057221804	0.644786325	0.227074048
currency	90	93	88	0.977777778	0.946236559	0.961748634	0.180694855	0.021028189	0.005254301	0.113333333	0.377720307
supplier_name	52	97	44	0.846153846	0.453608247	0.590604027	0.885223716	0.125123187	0.057222222	0.708888889	0.208105927
supplier_site	34	71	28	0.823529412	0.394366197	0.533333333	0.764543651	0.126793651	0.061666667	0.666666667	0.139600005
date_of_sale	77	99	73	0.948051948	0.737373737	0.829545455	0.385418385	0.065418385	0.026396203	0.31547619	0.243142672

Experiment 3

TAG	EXTRACTED	GOLD	CORRECT	PRECISION	RECALL	F1	PRC_AUC	RPC_OVER_80_AUC	RPC_OVER_90_AUC	ROC_AUC	MSE
invoice_number	78	100	72	0.923076923	0.72	0.808988764	0.771750607	0.135656658	0.066329735	0.655769231	0.294603721
total_amount	64	101	59	0.921875	0.584158416	0.715151515	0.541452149	0.068118816	0.021661279	0.334358974	0.322171748
currency	88	93	86	0.977272727	0.924731183	0.950276243	0.161210728	0.012988506	0.001254301	0.093333333	0.153991647
supplier_name	54	98	48	0.888888889	0.489795918	0.631578947	0.723223257	0.114199447	0.052266547	0.568857809	0.471390668
supplier_site	40	71	35	0.875	0.492957746	0.630630631	0.716344955	0.090085034	0.043214286	0.450892857	0.186276342
date_of_sale	83	99	81	0.975903614	0.818181818	0.89010989	0.199241728	0.039241728	0.019241728	0.19375	0.173656444

Experiment 4

TAG	EXTRACTED	GOLD	CORRECT	PRECISION	RECALL	F1	PRC_AUC	RPC_OVER_80_AUC	RPC_OVER_90_AUC	ROC_AUC	MSE
invoice_number	82	100	74	0.902439024	0.74	0.813186813	0.956262316	0.161173285	0.074434468	0.756629318	0.301598868
total_amount	63	101	57	0.904761905	0.564356436	0.695121951	0.993808065	0.193808065	0.094768949	0.940659341	0.271826265
currency	90	93	88	0.977777778	0.946236559	0.961748634	0.160430305	0.012208083	8.42E-04	0.086666667	0.18232055
supplier_name	51	98	46	0.901960784	0.469387755	0.617449664	0.53751161	0.086154467	0.039225895	0.402424242	0.514236262
supplier_site	37	71	32	0.864864865	0.450704225	0.592592593	0.699217372	0.082557319	0.035833333	0.358333333	0.233378536
date_of_sale	82	99	81	0.987804878	0.818181818	0.895027624	0.191836264	0.031836264	0.011901623	0.094117647	0.249336567

Experiment 5

TAG	EXTRACTED	GOLD	CORRECT	PRECISION	RECALL	F1	PRC_AUC	RPC_OVER_80_AUC	RPC_OVER_90_AUC	ROC_AUC	MSE
invoice_number	79	100	73	0.924050633	0.73	0.815642458	0.766905697	0.128676149	0.059371351	0.589230769	0.279723248
total_amount	61	101	50	0.819672131	0.495049505	0.617283951	0.764624862	0.133566439	0.063961433	0.679191919	0.172975566
currency	90	93	88	0.977777778	0.946236559	0.961748634	0.180694855	0.021028189	0.005254301	0.113333333	0.367470382
supplier_name	46	97	40	0.869565217	0.412371134	0.559440559	0.758425706	0.12675904	0.056457071	0.605555556	0.175961041
supplier_site	34	71	28	0.823529412	0.394366197	0.533333333	0.760376984	0.119960317	0.056833333	0.658333333	0.138980216
date_of_sale	80	99	75	0.9375	0.757575758	0.837988827	0.585104158	0.105104158	0.045497677	0.508333333	0.38315795