General Description

WF-ETL stands for Workfusion ETL Tool.

The main purpose of the WF-ETL tool is to migrate data from SPA 9.2 to IAC 10.1.x, including:

Data stores.
User/group/role/permission.

Subsystem	Source RDBMS	Source Database	Source Schema	Target RDBMS	Target Database	Target Schema
Control Tower	MySQL	wfdb	-	MS SQL	workfusion	ct
Data Store	PostgreSQL	wf_datastore	public	MS SQL	workfusion	ds

Concepts: migration of data stores

Conceptually, the process consists of the following steps:

Transferring of metadata, which is stored in Control Tower (to display data stores on UI).
Creating data stores in the target database.
Pumping of data into the target data stores.

Metadata is stored in the tables (MySQL):

FieldSchema
DataStore (not to be confused with DATA_STORE)
DataStoreColumn
Answer

Data is stored in PostgreSQL wf_datastore.public. Every data store consists of three parts:

ds_<name> - main table of data store.
ds_<name>_archive - tables with records, which are removed from the main table.
ds_<name>_ad_trg - trigger on the main table to move deleted records to the archive table.

Creating data stores on the target is implemented as auto generation of create table statements. Data pumping is going "table-by-table". Preliminary, a list of pre-existing tables of the target database is stored to avoid the table name collision: if a table already exists on the target it will be skipped during ETL. The list is saved in ds.etl_9_10_skip_list.

Concepts: migration of User/Group/Role/Permission

List of tables to migrate:

Account
ApplicationUser
UserGroup
UserGroupApplicationUserLink
Role
UserGroupRoleLink
RolePermissionLink
ApplicationUserRoleMap

Setup

Files to deploy

Installation

important

Deploy the scripts to the target database server to re-use MySQL/PostgreSQL libraries.

It is expected that the target database is bootstrapped: no extra changes have been done before the migration:

No new data stores were created.
No new role/permission were added.

You can skip this behavior with the option --skip-check.

Assume that DBTOOLHOME=/opt/workfusion/db-tools

Upload and unzip the Python virtual environment into $DBTOOLHOME/venv.
```
tar xzvf db-tools-env.tar.gz
```
Upload and unzip the WF-ETL tool archive into $DBTOOLHOME/wfetl.
Create a directory for logs: $DBTOOLHOME/log.
Fill out correct values in $DBTOOLHOME/wfetl/wfetl.ini. See the template in the section "Configuration" below.
Change a current directory to $DBTOOLHOME/venv.
Run source ./environment.sh to configure Python's virtual environment.
Change a current directory to $DBTOOLHOME/.

Run python -m wfetl, python -m wfetl ds -h or python -m wfetl sec -h to see the help message:

python -m wfetl

usage: wfetl [-h] {ds,sec} ...

command-line ETL processing to migrate data from SPA 9.2 to IAC 10.1

optional arguments:
-h, --help  show this help message and exit

ETL actions:
{ds,sec}

WorkFusion, Database Tools, v.20.08.13-1

python -m wfetl ds -h

usage: wfetl ds [-h] [-c CONFIG] [-i] [-l {debug,info,warning,error}] [-p]
                [-t] [-v] [-e [{meta,ddl,data} [{meta,ddl,data} ...]]] [-r]

optional arguments:
-h, --help            show this help message and exit
-c CONFIG, --config CONFIG
                        path to the configuration file
-i, --init            initialize ETL dictionary
-l {debug,info,warning,error}, --loglevel {debug,info,warning,error}
                        logging level
-p, --pre-check       ping sites and pre-check database types
-s, --skip-check      skip bootstrap check of target database
-t, --tidy-up         remove all of staff used by ETL
-v, --validate        validate results
-e [{meta,ddl,data} [{meta,ddl,data} ...]], --etl [{meta,ddl,data} [{meta,ddl,data} ...]]
                        run ETL to migrate metadata, table structures and data
-r, --revert          remove all of new data stores and clean up meta data

python -m wfetl sec -h

usage: wfetl sec [-h] [-c CONFIG] [-i] [-l {debug,info,warning,error}] [-p]
                [-t] [-v] [-e [{user,role,perm} [{user,role,perm} ...]]]

optional arguments:
-h, --help            show this help message and exit
-c CONFIG, --config CONFIG
                        path to the configuration file
-i, --init            initialize ETL dictionary
-l {debug,info,warning,error}, --loglevel {debug,info,warning,error}
                        logging level
-p, --pre-check       ping sites and pre-check database types
-s, --skip-check      skip bootstrap check of target database
-t, --tidy-up         remove all of staff used by ETL
-v, --validate        validate results
-e [{user,role,perm} [{user,role,perm} ...]], --etl [{user,role,perm} [{user,role,perm} ...]]
                        run ETL to migrate users, roles and/or permissions

Configuration

The ini-file contains sections for the following databases presented on both source and target databases:

[source_ct]
db_type=mysql
db_host=<source-db-host-name>
db_port=<port>
db_name=<database-name>
db_user=<user-name>
db_pass=<user-password>

[source_ds]
db_type=postgresql
db_host=<source-db-host-name>
db_port=<port>
db_name=wf_datastore
db_user=<user-name>
db_pass=<user-password>

[source_ws]
db_type=postgresql
db_host=<source-db-host-name>
db_port=<port>
db_name=virtualizer
db_user=<user-name>
db_pass=<user-password>

[target_ct]
db_type=mssql
db_host=<target-db-host-name>
db_port=<port>
db_name=workfusion
db_user=<ct-user-name>
db_pass=<ct-user-password>

[target_ct]
db_type=mssql
db_host=<target-db-host-name>
db_port=<port>
db_name=workfusion
db_user=<ds-user-name>
db_pass=<ds-user-password>

[target_ws]
db_type=mssql
db_host=<target-db-host-name>
db_port=<port>
db_name=workfusion
db_user=<ws-user-name>
db_pass=<ws-user-password>

Logging

The File name format for log files is wfetl.YYYY-MMDD-HHMI-SS.log, where:

YYYY - year
MM - month
DD - day of month
HH - hours (in 24-hours format)
MI - minutes
SS - seconds

For instance: wfetl.2020-0227-1813-37.log

Usage: Data Stores

Examples

Sample of the whole list of options:

python -m wfetl ds -c wfetl/conf/wfetl.ini -p -i -e meta ddl data -v -l info -t

Check connections:

python -m wfetl ds -c wfetl/conf/wfetl.ini -p

Validate post-ETL state:

python -m wfetl ds -c wfetl/conf/wfetl.ini -v

Post-ETL clean-up:

python -m wfetl ds -c wfetl/conf/wfetl.ini -t

Description of log output

Pre-check and initialize

All sites are available:

python -m wfetl ds -c wfetl/conf/wfetl.ini -p -i

[2020-09-24 17:24:50,029] wfetl.db_site        INFO     Ping DB - OK: site:[source_ds], host:[wfinst-401-92-db1.workfusion.com]
[2020-09-24 17:24:50,035] wfetl.db_site        INFO     Ping DB - OK: site:[target_ws], host:[localhost]
[2020-09-24 17:24:50,036] wfetl.db_site        INFO     Ping DB - OK: site:[target_ds], host:[localhost]
[2020-09-24 17:24:50,424] wfetl.db_site        INFO     Ping DB - OK: site:[source_ws], host:[wfinst-401-92-db1.workfusion.com]
[2020-09-24 17:24:50,691] wfetl.db_site        INFO     Ping DB - OK: site:[source_ct], host:[wfinst-401-92-db1.workfusion.com]
[2020-09-24 17:24:50,694] wfetl.db_site        INFO     Ping DB - OK: site:[target_ct], host:[localhost]
[2020-09-24 17:24:50,694] wfetl.db_site        INFO     Pre-check - OK: [source_ct] is mysql     
[2020-09-24 17:24:50,694] wfetl.db_site        INFO     Pre-check - OK: [source_ds] is postgresql
[2020-09-24 17:24:50,696] wfetl.db_site        INFO     Pre-check - OK: [target_ct] is mssql     
[2020-09-24 17:24:50,696] wfetl.db_site        INFO     Pre-check - OK: [target_ds] is mssql     
[2020-09-24 17:24:50,711] wfetl.ds_loader      WARNING  Table [etl_9_10_skip_tables] already exists

The last row can look like:

...
[2020-09-24 18:07:27] wfetl.db_util        ERROR    Non-bootstrap database

ETL (data loading)

python -m wfetl ds -c wfetl/conf/wfetl.ini -p -e meta ddl data

[2020-09-24 17:24:51,542] wfetl.ds_loader      INFO     Total number of tables to copy:[2630]
[2020-09-24 17:24:51,542] wfetl.ds_loader      INFO     === COPY-TABLE === 1 of 2630: table name: [src:ds_WFML_Job_Data_archive][tgt:ds_WFML_Job_Data_archive]
[2020-09-24 17:24:51,552] wfetl.db_util        DEBUG    schema:[ds], table:[ds_WFML_Job_Data_archive]
[2020-09-24 17:24:52,151] wfetl.db_util        DEBUG    copy-table-data(): SQL: select * from "ds_WFML_Job_Data_archive" where "ds_WFML_Job_Data_id" >= %s [last-id:76]
[2020-09-24 17:24:52,555] wfetl.ds_loader      INFO     Process time: Total:[1.012]secs. Last table:[1.012]secs
[2020-09-24 17:24:52,555] wfetl.ds_loader      INFO     === COPY-TABLE === 2 of 2630: table name: [src:ds_WFML_Job_Data][tgt:ds_WFML_Job_Data]
[2020-09-24 17:24:53,115] wfetl.ds_loader      DEBUG    Next [fieldschema].[id] = [1]
[2020-09-24 17:24:53,667] wfetl.ds_loader      DEBUG    [copy-ds-meta-ds] ds-id-src:[7]
[2020-09-24 17:24:53,667] wfetl.ds_loader      DEBUG    [copy-ds-meta-ds] ds-id-tgt:[1]
[2020-09-24 17:24:53,960] wfetl.ds_loader      DEBUG    [copy-ds-meta-ds-column] id-delta:[1]
[2020-09-24 17:24:54,232] wfetl.ds_loader      DEBUG    [copy-ct-meta-answer] fs-id-src:[7], fs-id-tgt:[1]
[2020-09-24 17:24:54,509] wfetl.db_util        DEBUG    schema:[ds], table:[ds_WFML_Job_Data]
[2020-09-24 17:24:54,895] wfetl.db_util        DEBUG    copy-table-data(): SQL: select * from "ds_WFML_Job_Data" where "ds_WFML_Job_Data_id" >= %s [last-id:79]
[2020-09-24 17:24:55,289] wfetl.ds_loader      INFO     Process time: Total:[3.747]secs. Last table:[2.734]secs
[2020-09-24 17:24:55,290] wfetl.ds_loader      INFO     === COPY-TABLE === 3 of 2630: table name: [src:ds_uc1_data_validation_archive][tgt:ds_uc1_data_validation_archive]
[2020-09-24 17:24:55,299] wfetl.db_util        DEBUG    schema:[ds], table:[ds_uc1_data_validation_archive]
[2020-09-24 17:24:55,690] wfetl.db_util        DEBUG    copy-table-data(): SQL: select * from "ds_uc1_data_validation_archive" where "ds_uc1_data_validation_id" >= %s [last-id:1]
[2020-09-24 17:24:56,085] wfetl.ds_loader      INFO     Process time: Total:[4.543]secs. Last table:[0.795]secs
[2020-09-24 17:24:56,086] wfetl.ds_loader      INFO     === COPY-TABLE === 4 of 2630: table name: [src:ds_uc1_data_validation][tgt:ds_uc1_data_validation]
[2020-09-24 17:24:56,621] wfetl.ds_loader      DEBUG    Next [fieldschema].[id] = [2]
[2020-09-24 17:24:57,157] wfetl.ds_loader      DEBUG    [copy-ds-meta-ds] ds-id-src:[61]
[2020-09-24 17:24:57,158] wfetl.ds_loader      DEBUG    [copy-ds-meta-ds] ds-id-tgt:[2]
[2020-09-24 17:24:57,431] wfetl.ds_loader      DEBUG    [copy-ds-meta-ds-column] id-delta:[28]
[2020-09-24 17:24:57,697] wfetl.ds_loader      DEBUG    [copy-ct-meta-answer] fs-id-src:[64], fs-id-tgt:[2]
[2020-09-24 17:24:57,967] wfetl.db_util        DEBUG    schema:[ds], table:[ds_uc1_data_validation]
[2020-09-24 17:24:58,358] wfetl.db_util        DEBUG    copy-table-data(): SQL: select * from "ds_uc1_data_validation" where "ds_uc1_data_validation_id" >= %s [last-id:2]
[2020-09-24 17:24:58,750] wfetl.ds_loader      INFO     Process time: Total:[7.208]secs. Last table:[2.664]secs
[2020-09-24 17:24:58,752] wfetl.ds_loader      INFO     === COPY-TABLE === 5 of 2630: table name: [src:ds_type_of_quantities_archive][tgt:ds_type_of_quantities_archive]
[2020-09-24 17:24:58,762] wfetl.db_util        DEBUG    schema:[ds], table:[ds_type_of_quantities_archive]
[2020-09-24 17:24:59,141] wfetl.db_util        DEBUG    copy-table-data(): SQL: select * from "ds_type_of_quantities_archive" where "ds_type_of_quantities_id" >= %s [last-id:1]
[2020-09-24 17:24:59,536] wfetl.ds_loader      INFO     Process time: Total:[7.993]secs. Last table:[0.784]secs
[2020-09-24 17:24:59,538] wfetl.ds_loader      INFO     === COPY-TABLE === 6 of 2630: table name: [src:ds_type_of_quantities][tgt:ds_type_of_quantities]
[2020-09-24 17:25:00,072] wfetl.ds_loader      DEBUG    Next [fieldschema].[id] = [3]
[2020-09-24 17:25:00,648] wfetl.ds_loader      DEBUG    [copy-ds-meta-ds] ds-id-src:[31]
[2020-09-24 17:25:00,648] wfetl.ds_loader      DEBUG    [copy-ds-meta-ds] ds-id-tgt:[3]
[2020-09-24 17:25:00,917] wfetl.ds_loader      DEBUG    [copy-ds-meta-ds-column] id-delta:[377]
[2020-09-24 17:25:01,197] wfetl.ds_loader      DEBUG    [copy-ct-meta-answer] fs-id-src:[32], fs-id-tgt:[3]
[2020-09-24 17:25:01,460] wfetl.db_util        DEBUG    schema:[ds], table:[ds_type_of_quantities]
[2020-09-24 17:25:01,842] wfetl.db_util        DEBUG    copy-table-data(): SQL: select * from "ds_type_of_quantities" where "ds_type_of_quantities_id" >= %s [last-id:4]
[2020-09-24 17:25:02,237] wfetl.ds_loader      INFO     Process time: Total:[10.695]secs. Last table:[2.7]secs
...

Summary sample:

[2020-09-24 17:18:48,312] wfetl.ds_loader      INFO     Migration stats: Planed:[2], Failed:[0], Proc:[2], Skip:[0], Load: DDL:[2], Meta[0], Data:[2], BadIdx:[0], SkipIdx:[4]

INFO Total number of tables to copy:[6]
means there are 6 tables in the source database to migrate.

WARNING SKIPPED. Table pre-exists on target: [not_ds_data]
the table with the name not_ds_data was created before migration.

INFO === COPY-TABLE === 3 of 6: table name: [src:ds_t1][tgt:ds_t1]
regular copying of a new table.

DEBUG CREATE TABLE [ds].ds_t1 ( ... - head of create table statement.
DEBUG create trigger ds_t1_ad_trg on ds_t1 - head of create trigger statement.
See a log file for the whole SQL statements.

INFO Process time: Total:[8.446]secs. Last table:[7.429]secs
Cumulative processing time is 8.446 secs,
Time to process ds_t1 is 7.429 secs.

INFO Migration stats: Planed:[6], Failed:[0], Proc:[6], Skip:[4], Load: DDL:[1], Meta[1], Data:[1], BadIdx:[0]

Number of tables to migrate: Planned:[6]
Tables processed with errors: Failed:[0]
Processed tables: Proc:[6]
Skipped due to name collision: Skip:[4]
Created new tables: DDL:[1]
Meta data is transferred for Meta:[1] table.
Data was pumped for Data:[1] table.
Number of not-created indexes: BadIdx:[0] Number of skipped indexes: SkipIdx:[2]

Known issues

Not all types of indexes can be migrated from PostgreSQL to MSSQL. The only B-Tree indexes can be migrated (including multi-column indexes). HASH-indexes, conditional indexes and other PostgreSQL specific types of indexes are being skipped during migration.

Log file will contains entries like:

[2020-09-21 15:23:55,190] wfetl.ds_loader      WARNING  SKIPPED index:[public.not_ds_data_2.not_ds_data_2_idx_ncol_hash]
[2020-09-21 15:23:55,190] wfetl.ds_loader      DEBUG    CREATE INDEX not_ds_data_2_idx_ncol_hash ON public.not_ds_data_2 USING hash (ncol)
[2020-09-21 15:23:55,191] wfetl.ds_loader      WARNING  SKIPPED index:[public.not_ds_data_2.not_ds_data_2_idx_scol]
[2020-09-21 15:23:55,191] wfetl.ds_loader      DEBUG    CREATE INDEX not_ds_data_2_idx_scol ON public.not_ds_data_2 (scol) WHERE ((scol)::text ~~ 'a%'::text)

Validation

python -m wfetl ds -c wfetl.ini -v

[2020-03-03 14:36:47] wfetl.ds_loader      INFO     === Validation ===
[2020-03-03 14:36:48] wfetl.ds_loader      INFO     [ds_t1_archive] - start validation
[2020-03-03 14:36:49] wfetl.ds_loader      INFO         Skipped. Pre-exists on target
[2020-03-03 14:36:49] wfetl.ds_loader      INFO     [ds_t1] - start validation
[2020-03-03 14:36:50] wfetl.ds_loader      INFO         Rows on source:[3], on target:[3]
[2020-03-03 14:36:50] wfetl.ds_loader      INFO         Meta Data: Field Schema - OK
[2020-03-03 14:36:51] wfetl.ds_loader      INFO         Meta Data: DataStore    - OK
[2020-03-03 14:36:51] wfetl.ds_loader      INFO     [not_ds_data] - start validation
[2020-03-03 14:36:52] wfetl.ds_loader      INFO         Skipped. Pre-exists on target
[2020-03-03 14:36:52] wfetl.ds_loader      INFO     [ds_Global_Variables_archive] - start validation
[2020-03-03 14:36:52] wfetl.ds_loader      INFO         Skipped. Pre-exists on target
[2020-03-03 14:36:52] wfetl.ds_loader      INFO     [ds_Global_Variables] - start validation
[2020-03-03 14:36:53] wfetl.ds_loader      INFO         Skipped. Pre-exists on target
[2020-03-03 14:36:53] wfetl.ds_loader      INFO     === Validation Summary ===
[2020-03-03 14:36:53] wfetl.ds_loader      INFO     [   32] - tables on target before migration (listed in:[etl_9_10_skip_tables])
[2020-03-03 14:36:53] wfetl.ds_loader      INFO     [    5] - tables on source
[2020-03-03 14:36:53] wfetl.ds_loader      INFO     [    0] - missed on target
[2020-03-03 14:36:53] wfetl.ds_loader      INFO     [    1] - passed (equal row count on both source and target)
[2020-03-03 14:36:53] wfetl.ds_loader      INFO     [    0] - different row count
[2020-03-03 14:36:53] wfetl.ds_loader      INFO     [    0] - no meta data for Field Schema
[2020-03-03 14:36:53] wfetl.ds_loader      INFO     [    0] - no Answers for Field Schema
[2020-03-03 14:36:53] wfetl.ds_loader      INFO     [    0] - no meta data for DataStore
[2020-03-03 14:36:53] wfetl.ds_loader      INFO     [    0] - no columns for DataStore
[2020-03-03 14:36:53] wfetl.ds_loader      INFO     ===

Per-table messages:

Skipped. Pre-exists on target
Table has been presented on the target before migration. No action done.

WARNING Table is absent on target
table is presented on the source but absent on the target.

INFO Rows on source:[3], on target:[3]
Good state: number of rows is equal on both source and target.

WARNING Rows on source:[3], on target:[1]
Numbers of rows are not the same on source and target.

INFO Meta Data - skipped (is not a data store)
Table is not a data store. No checks for meta data.

WARNING Meta Data: Field Schema - absent
No records in the table ct.fieldschema

WARNING Meta Data: DataStore - absent
No records in the table ct.datastore

Usage: User/Group/Role/Permission

Initialization

python -m wfetl sec -c wfetl.ini -p -i

[2020-03-17 12:54:32] wfetl                DEBUG    Process SEC actions
[2020-03-17 12:54:32] wfetl.db_site        INFO     db ping conn - OK site:[source_ct], host:[wfinst-401-92-db1.workfusion.com]
[2020-03-17 12:54:32] wfetl.db_site        INFO     db ping conn - OK site:[target_ct], host:[rcm-3661-1012-ut-wfan-10024-mssql1.workfusion.com]
[2020-03-17 12:54:32] wfetl.sec_loader     INFO     Pre-check: source (CT) is MySQL      - OK
[2020-03-17 12:54:32] wfetl.sec_loader     INFO     Pre-check: target is MS SQL          - OK
[2020-03-17 12:54:32] wfetl.sec_loader     DEBUG    ETL: init stats
DBG (tab-stats):
cnt_src  cnt_tgt  table_name_src
      1        1  u'Account'
     30        1  u'ApplicationUser'
      7        0  u'UserGroup'
     33        0  u'UserGroupApplicationUserLink'
      6        3  u'Role'
      7        0  u'UserGroupRoleLink'
    137       84  u'RolePermissionLink'
     30        1  u'ApplicationUserRoleMap'

Data loading

python -m wfetl sec -c wfetl.ini -e user role perm

[2020-03-17 12:54:47] wfetl                DEBUG    Process SEC actions
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL users
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL License. start...
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL License. done
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL Account. start...
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL Account. done
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL AppUser. start...
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL AppUser. done
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL UserGroup. start...
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL UserGroup. done
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL UserGroup-x-ApplicatonUser. start...
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL UserGroup-x-ApplicationUser. done
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL Role. start...
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL Role. done
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL AppUsr-x-Role. start...
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL AppUsr-x-Role. done
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL permissions
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL RolePermissionLink. start...
[2020-03-17 12:54:47] wfetl.sec_loader     DEBUG    ETL RolePermissionLink. done

Validation

python -m wfetl sec -c wfetl.ini -v

[2020-03-17 13:19:33] wfetl                DEBUG    Process SEC actions
[2020-03-17 13:19:33] wfetl.sec_loader     DEBUG    Validate ETL results
[2020-03-17 13:19:33] wfetl.sec_loader     INFO     table_name                     |  src:ini |  tgt:ini |  tgt:now |  tgt-src |  tgt-tgt
[2020-03-17 13:19:33] wfetl.sec_loader     INFO     Account                        |        1 |        1 |        1 |        0 |        0
[2020-03-17 13:19:33] wfetl.sec_loader     INFO     ApplicationUser                |       30 |        1 |       30 |        0 |       29
[2020-03-17 13:19:33] wfetl.sec_loader     INFO     ApplicationUserRoleMap         |       30 |        1 |       30 |        0 |       29
[2020-03-17 13:19:33] wfetl.sec_loader     INFO     Role                           |        6 |        3 |        6 |        0 |        3
[2020-03-17 13:19:33] wfetl.sec_loader     INFO     RolePermissionLink             |      137 |       84 |      132 |       -5 |       48
[2020-03-17 13:19:33] wfetl.sec_loader     INFO     UserGroup                      |        7 |        0 |        7 |        0 |        7
[2020-03-17 13:19:33] wfetl.sec_loader     INFO     UserGroupApplicationUserLink   |       33 |        0 |       33 |        0 |       33
[2020-03-17 13:19:33] wfetl.sec_loader     INFO     UserGroupRoleLink              |        7 |        0 |        7 |        0 |        7

src:ini - number of rows on source.
tgt:ini - initial number of rows on target.
tgt:now - current number of rows on target.
tgt-src - difference between number of rows on target and on source. Can be negative because not all of source records may be migrated.
tgt-tgt - difference between initial amount of rows and final amount of rows on target. Show how many new rows ahve appeared on target.

Cleaning up

python -m wfetl sec -c wfetl.ini -t

[2020-03-17 14:23:49] wfetl                DEBUG    Process SEC actions
[2020-03-17 14:23:49] wfetl.sec_loader     INFO     Table is dropped:[etl_9_10_row_stats]

Actually, the file ./etl_9_10_rows_stats is being removed during this step.

Examples

Migrate Data Stores:

python -m wfetl -c conf/wfetl.ini -p -i -e meta data ddl -p

Migrate Users and Roles:

python -m wfetl -c /conf/wfetl.ini -p -i -e user role perm -p

Logs are located in the $DBTOOLHOME/log/ directory.

WorkFusion Documentation

10.2

Hardware

Install Linux components

Keycloak administrator guide

Workfusion Tools for ETL

General Description

Concepts: migration of data stores

Concepts: migration of User/Group/Role/Permission

Setup

Files to deploy

Installation

important

Configuration

Logging

Usage: Data Stores

Examples

Description of log output

Pre-check and initialize

ETL (data loading)

Known issues

Validation

Usage: User/Group/Role/Permission

Initialization

Data loading

Validation

Cleaning up

Examples