10.2.2

Edit

Troubleshoot installation issues

BI services drop connection to APP

Expand to learn more

Symptoms

When BI services are running, they drop the connection to the Master server.

Resolution

To resolve the issue, execute the following script:

sysctl -w net.ipv4.tcp_tw_recycle=0

As stated by the tcp(7) manual, the net.ipv4.tcp_tw_recycle option is problematic for public-facing servers as it won’t handle connections from two different computers behind the same NAT device.

SSL check cannot verify CA certificate

Expand to learn more

Symptoms

An SSL check cannot verify the CA certificate.

SSL check is unable to verify CA certificate
[wfsadmt@wfapp workfusion]$ openssl s_client -showcerts -connect wfbi.example.com:443 
...
SSL handshake has read 1698 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 7DB33295ED365F489D1D953076A04C95D1A9A76E14648526666AD3A0B90FFE5F
Session-ID-ctx:
Master-Key: ECB17FE663720814E973AC3A576A32E217757174CDBC9F39C4E2A423AF6D0E22D45A2809A5AC972D44AE4B627A2627C8
Key-Arg : None

Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)

Resolution

Add CA.cert to the OpenSSL system main file:

# openssl version -d
OPENSSLDIR: "/etc/pki/tls"

cat CA.cert >> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

ls -lah /etc/pki/ca-trust/extracted/pem/
lrwxrwxrwx. 1 root root 49 Apr 18 2017 cert.pem -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

vds-standalone/vds-master installation fails with kernel.exec-shield

Expand to learn more

Symptoms

During the vds-standalone or vds-master installation, Ansible fails on the Connected with NFS step.

Resolution

Follow the steps below:

  1. Open the /etc/sysctl.conf file or files in the /etc/sysctl.d/ folder.

  2. Find the kernel.exec-shield option and change its value to 0:

    kernel.exec-shield=0

  3. Upload the options from the /etc/sysctl.conf file to the Linux kernel once again:

    sysctl -p /etc/sysctl.conf

  4. Restart nfs-server.service:

    systemctl restart nfs-server.service

  5. Rerun the Ansible installation step:

    ./install.sh install vds-standalone

No access to RHEL repositories

Expand to learn more

Symptoms

  • Servers have no access to public RHEL repositories.

  • yum fails with the following traceback:

    See traceback

    Traceback (most recent call last):
File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 375, in user_main
    errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 170, in main
    base.getOptionsConfig(args)
File "/usr/share/yum-cli/cli.py", line 262, in getOptionsConfig
    self.conf
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1078, in <lambda>
    conf = property(fget=lambda self: self._getConfig(),
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 420, in _getConfig
    self.plugins.run('init')
File "/usr/lib/python2.7/site-packages/yum/plugins.py", line 188, in run
    func(conduitcls(self, self.base, conf, **kwargs))
File "/usr/share/yum-plugins/rhnplugin.py", line 121, in init_hook
    login_info = up2dateAuth.getLoginInfo(timeout=timeout)
File "/usr/share/rhn/up2date_client/up2dateAuth.py", line 228, in getLoginInfo
    login(timeout=timeout)
File "/usr/share/rhn/up2date_client/up2dateAuth.py", line 195, in login
    li = server.up2date.login(systemId)
File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__
    return rpcServer.doCall(method, *args, **kwargs)
File "/usr/share/rhn/up2date_client/rpcServer.py", line 204, in doCall
    ret = method(*args, **kwargs)
File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
    return self.__send(self.__name, args)
File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1
    ret = self._request(methodname, params)
File "/usr/lib/python2.7/site-packages/rhn/rpclib.py", line 384, in _request
    self._handler, request, verbose=self._verbose)
File "/usr/lib/python2.7/site-packages/rhn/transports.py", line 188, in request
    return self._process_response(fd, connection)
File "/usr/lib/python2.7/site-packages/rhn/transports.py", line 195, in _process_response
    fd = resp.decode(fd)
File "/usr/lib/python2.7/site-packages/rhn/transports.py", line 442, in decode
    self.read(fd)
File "/usr/lib/python2.7/site-packages/rhn/transports.py", line 420, in read
    max_mem_size=self.max_mem_size)
File "/usr/lib/python2.7/site-packages/rhn/transports.py", line 528, in _smart_read
    chunk = fd.read(l)
File "/usr/lib64/python2.7/httplib.py", line 610, in read
    self.close()
File "/usr/lib64/python2.7/httplib.py", line 555, in close
    self.fp.close()
File "/usr/lib/python2.7/site-packages/rhn/SSL.py", line 133, in close
    self._really_close()
File "/usr/lib/python2.7/site-packages/rhn/SSL.py", line 148, in _really_close
    if self._connection.state_string() == 'SSL negotiation finished successfully':
File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1135, in __getattr__
    return getattr(self._socket, name)
AttributeError: '_socketobject' object has no attribute 'state_string'

Resolution

To resolve the issue, follow the steps below:

  1. As a root user, open /usr/lib/python2.7/site-packages/rhn/SSL.py.

  2. Change line #144.

    Original line #144:

    if self._connection.get_state_string() == b'SSL negotiation finished successfully':

    Line #144 after the required change:

    if self._connection.state.string() == b'SSL negotiation finished successfully':

Analytics setup fails

Expand to learn more

In some cases, the installation of Analytics components can fail on the Ensure Analytics Server installed task with the following error message:

  • Trial activation for Tableau server was not applied correctly. Refer to the documentation for further steps.
  • There is not enough license to promote user 3 to the system admin. (errorCode=10009)

This issue relates to the Tableau software and appears when the Tableau trial activation fails with errors.

VDI installation on RPA Server

Expand to learn more

For VDI installations, you need to maintain RDP sessions active.

Mind that Bot Manager can open an RDP session but for some seconds only, and then the session is disconnected. In this case, you have a unit without an active session—hence, no active UI to make RPA work incorrectly. To avoid opening an RDP connection, start Bot Manager with the noRdpCheck profile:

  1. Go to the Master server where Bot Manager is installed.

  2. In the /opt/workfusion/bot-manager/run-bot-manager.sh file, change the --spring.profiles.active value from master to noRdpCheck. After editing, you should have:

    --spring.profiles.active=production,noRdpCheck

  3. Save the changes and restart Bot Manager.

    wfmanager restart bot-manager

Single-point installation fails with error "sudo: no tty present"

Expand to learn more

To fix the problem, follow the steps below:

  1. Go to INSTALL_DIR/wf_installer/.

  2. In the install.sh script, change export ANSIBLE_SSH_PIPELINING=true to false with any available text editor.

    vi install.sh
# the vi editor opens

export ANSIBLE_SSH_PIPELINING=false

  3. Save the changes to the install.sh file.

  4. Continue the installation.

Installation fails on the preinstall step

Expand to learn more

Symptoms

If the installation fails on the Configure sysctl parameters for Elasticsearch step, make sure that the etc/sysctl.conf configuration file contains no invalid parameters.

Resolution

Try to load the sysctl settings by running the following command on the host where you are going to install Elasticsearch:

sudo sysctl -p

The command must finish without errors. Otherwise, edit the /etc/sysctl.conf file and remove invalid settings.

For example, installation fails with the following error:

    TASK [root-steps : Configure sysctl parameters for Elasticsearch: vm.max_map_count=262144] ***
    fatal: [int.mphyEA.com]: FAILED! => changed=false
      msg: |-
        Failed to reload sysctl: vm.max_map_count = 262144
        sysctl: cannot stat /proc/sys/kernel/exec-shield: No such file or directory

As the sysctl settings are loaded from etc/sysctl.conf, the following error appears:

sudo sysctl -p
sysctl: cannot stat /proc/sys/kernel/exec-shield: No such file or directory

To fix the issue, remove kernel.exec-shield=0 from /etc/sysctl.conf. The option exec-shield is always enabled and no longer tunable in RHEL7.

No events in Kibana GUI after environment has been off for several days

Expand to learn more

Elasticsearch has an index policy configured to freeze all indexes after a predefined period (currently, three days). If the environment has been off for more than three days, no new events could be written to frozen indexes, which breaks the message search in the Kibana GUI:

If you see no data in search, verify whether indexes are indeed frozen:

  1. Go to Management > Index Management and find the Filebeat index tagged with the date when your environment was shut down. If it is frozen (see the picture below), unfreeze it.

  2. To unfreeze a particular index, select its checkbox, click the appearing Manage Index button, and choose Unfreeze index.

When all services are restarted, one of the Marathon applications cannot be started

Expand to learn more

Symptoms

When all services are stopped and started, one of marathon-apps services cannot be started, whereas in the status output says it is RUNNING.

The first symptom of the issue is the difference in uptime. Whereas most services show similar uptime, there is one that has a higher value:

If you observe this behavior after you stopped and started services, check the mesos-agent log file for the following message: 'Killing un-reregistered executor'. Use the Kibana GUI or CLI on each Master server:

cat /opt/workfusion/logs/mesos-agent.log | grep 'Killing un-reregistered executor'

Resolution

If the behavior exists on the particular Master server where such zombie service was found, restart mesos-agent:

wfmanager restart mesos-agent

After that, the uptime for zombie starts counting from 0 again, which indicates that it is now unfrozen.

Restore application removed via Marathon

Expand to learn more

Symptoms

You have accidentally removed an application via the Marathon GUI and want to restore it back.

Resolution

To restore the destroyed application, do the following:

  1. Log in to the Master server via SSH as the application user.

    ssh -i <path-to-private-key.pem> <username>@<hostname.master.com>

  2. Run the command:

    wfmanager deploy <application>
# here, workfusion

    Output:

    [INFO] ACTION=deploy, APP=workfusion
[INFO] Invoking command: marathon-cli deploy workfusion
{"version":"2020-10-29T09:59:05.432Z","deploymentId":"ce408b61-1baf-4011-85c7-91df658f5b9a"}

As a result, the destroyed application is restored within several minutes.

← Fix failed Kibana authentication via self-signed certificatesMount new partition →