Updating Apache Versions for Job Manager

Purpose

This document is written to help you understand how to upgrade your underlying Apache and Apache Tomcat versions within Job Manager for customers with On Premise installations.

Important Points to Note before you start

• These tasks should only be performed by an appropriately skilled professional.
• All changes should be throughly tested in Test before committing changes to production.
• The upgrade of HTTPD and Tomcat can be scheduled to run concurrently with an Oracle database upgrade.
• If performed concurrently, Job Manager cannot be restarted until the database upgrade is complete.
• Check with OneAdvanced support around tested and supported versions of Apache and Apache Tomcat for your version of Job Manager
• Please note that the configuration of TLS certificates has been altered for Apache HTTPD to bring inline with recommended practices for more recent version of Apache HTTPD.
• The text file, D:\tpp\apache\apache-2.4.58\conf\ssl\README-creating-certificate-files.txt, describes how certificate file, fullchain.pem, is produced.
• If in any doubt, please contact OneAdvanced Support before continuing.

Installation files

Download your required installation files (Apache HTTPD and Tomcat) to the Job Manager Host Server.

Certificate check

Before upgrading Apache HTTPD you should check the certificates in use by navigating to your Job Manager console in a browser and checking the certificates reported:

In the above screenshot we see that the certificate for host has been signed by intermediate certificate DigiCert Global G2 TLS RSA SHA256 2020 CA1.

Using openssl we can check which certificates the server is including in its handshaking with HTTPS clients:

D:\tpp\apache\Apache2.4.56\bin>openssl s_client -connect localhost:443 -servername <<YOUR JOB MANAGER HOST SERVER NAME HERE>>
CONNECTED(00000130)
depth=1 C = US, O = DigiCert Inc, CN = DigiCert Global G2 TLS RSA SHA256 2020 CA1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = GB, L = London, O = <<ASSIGNED AGENT>>, CN = <<URL>>
verify return:1
---
Certificate chain
 0 s:C = GB, L = London, O = <<ASSIGNED AGENT>>, CN = <<URL>>
   i:C = US, O = DigiCert Inc, CN = DigiCert Global G2 TLS RSA SHA256 2020 CA1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 20 00:00:00 2023 GMT; NotAfter: Apr 19 23:59:59 2024 GMT
 1 s:C = US, O = DigiCert Inc, CN = DigiCert Global G2 TLS RSA SHA256 2020 CA1
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 30 00:00:00 2021 GMT; NotAfter: Mar 29 23:59:59 2031 GMT

The s_client output shows that a chain of two certificates are returned by the server, the certificate for the host itself and the certificate for the intermediate CA, DigiCert Global G2 TLS RSA SHA256 2020 CA1.

The certificate chain is correctly configured on this host for the old version of Apache HTTPD

Checking the configuration related to HTTPS, we find the following directives related to certificates:

SSLCertificateFile "${SRVROOT}/conf/ssl\<<ENDPOINTCER>>.cer"
SSLCertificateKeyFile "${SRVROOT}/conf/ssl\private.key"
SSLCACertificateFile "${SRVROOT}/conf/ssl\intermediate.cer"

Although the HTTPD server is currently returning a correct certificate chain, it is configured to do so by way of a separate Certificate Authority (CA) chain file. Newer versions of Apache HTTPD recommend we combine the host’s and the intermediate CAs' certificates into a single file.

Make sure a README-creating-certificate-files.TXT is placed alongside the certificate files in the new Apache installation with the following content:

Apache HTTPD is configured to use two files related to TLS: 
- private.key 
     - Contains the private key associated with the server's certificate. 
- fullchain.pem 
    - Contains the chain of the server's host TLS certificate and any Intermediate CA certificates necessary to establish trust back to a root CA certificate. 

When certificates are updated from time-to-time, manual steps may be needed to recreate fullchain.pem. 

The file private.key was already available from a previous Apache HTTPD installation. 

The file fullchain.pem was created by combining the content of the endpoint certificate file, <yourURL>.cer, with the content of intermediate.cer to a new text file. 

The order of certificates in fullchain.pem must be the host certificate, followed by any intermediate CA certificates in the order necessary to establish a trust chain back to a root CA certificate. 

The root CA certificate should not be added to fullchain.pem.

OpenSSL versions

If OpenSSL is installed on your servers, newer versions are bundled with Apache HTTPD, which will be deployed as part of the upgrade process.

Upgrade activities for the Job Manager host

• Shutdown Job Manager and its modules by running the /app/kirona/admin/stopall_jmLive_tomcatservice.bat batch file.
• Shutdown Apache HTTPD by running the /app/kirona/admin/stopApache.bat batch file.
• Uninstall the Apache HTTPD service
  • Navigate to the HTTPD bin directory and run the httpd -k uninstallcommand.
• Uninstall the Tomcat services for Job Manager and modules
  • Navigate to the /tpp/tomcat/KIRONALIVE/bin/servicescripts directory and run the following:

  JmLiveService.bat uninstall
  integrationLiveService.bat uninstall

Install Apache HTTPD

• Extract contents of directory httpd-2.4.58-win64-VS17.zip:/Apache24 to path /tpp/apache/apache-2.4.58.
• Copy files <YourURL>.cer and private.key from /tpp/apache/Apache2.4.56/conf/ssl to /tpp/apache/apache-2.4.58/conf/ssl.
• Use Notepad++ (or another appropriate comparison tool) to compare existing configuration file /tpp/apache/Apache2.4.56/conf/httpd.conf to new configuration file /tpp/apache/apache-2.4.58/conf/httpd.conf and apply relevant changes to the new configuration file.
• Use Notepad++ (or another appropriate comparison tool) to compare existing configuration file /tpp/apache/Apache2.4.56/conf/extra/httpd-ssl.conf to new configuration file /tpp/apache/apache-2.4.58/conf/extra/httpd-ssl.conf and apply relevant changes to the new configuration file.
  • Note: The host and intermediate CA certificates have been combined into a single file, fullchain.pem. This must be reflected in the new configuration file.
  • Note: Ensure ServerName in httpd-ssl.conf matches the subject in the host certificate.
• Run httpd from the command line to confirm no errors in configuration.
• Visit your Job Manager Dashboard url in a web browser and confirm It works! is displayed.
• On the command line, navigate to the /tpp/apache/apache-2.4.58/bin directory and run the following command to confirm the certificate chain is present in HTTPS handshaking:
  • openssl s_client -connect localhost:443 -servername <YourURL>
• Check logs files for any errors.
• Stop the httpd process by using Control-C.
• Install the Apache service by running httpd -k install.
• Clear any log files

Install Apache Tomcat

• Extract directory apache-tomcat-9.0.82-windows-x64.zip:/apache-tomcat-9.0.82 as /tpp/tomcat/apache-tomcat-9.0.82.
• Navigate to directory /tpp/tomcat/KIRONALIVE/bin/servicescripts and modify the three batch scripts used to install the Job Manager and modules' Windows Services to refer to the new version of Apache Tomcat.
• This can be done using Notepad++'s Find-and-Replace functionality. Find 9.0.80 and replace with 9.0.82.
• For each batch script, run the following with the 'install' command line argument:

jmLiveservice.bat install 
integrationLiveservice.bat install

• For each tomcat container under /tpp/tomcat/KIRONALIVE (i.e. JM and the modules), clean up the log directories by removing old logs, keeping the last 10 days of logs.
• Remove any old logs under /tpp/tomcat/KIRONALIVE/logs.
  • Note: Log rotation is applied to this directory, but there may be lots of old files that can be cleaned up.