Pablo Alvarez

This article intends to cover what you need to secure your search in Solr and how it's connected to WebFOCUS. Sometimes, some of our sensitive data is stored in the WebFOCUS Repository or at the WebFOCUS Reporting Server Application folders (hold files that are not stored temporarily, for example) and Solr allows you to find almost everything under WebFOCUS, so when performing a search, the connection between Solr and WebFOCUS is not encrypted by default and uses HTTP protocol. With this article, we want to enhance your security by adding an extra SSL layer to the communications between Solr and WebFOCUS, enabling SSL in Solr and WebFOCUS, and making available the connection between them. You'll find a troubleshooting section and a tips section at the end. I do recommend to read the entire article, and, of course, if you have any doubts or comments about it, we're here to help! To enable SOLR to work under a secure connection, you’ll need to have a valid certificate with a fully qualified domain name to be used (FQDN), we’ll follow all the steps to create one on your own and use it to secure and encrypt your connection between our WebFOCUS Client and the SOLR service. Even if I used this under Windows, the same steps apply to Linux, also, I used the WSL (Windows Subsystem for Linux) on my box to perform some of the steps, but that’s not necessary if you have the required components or you’ve been already provided with a valid certificate, so you don’t need to generate a new one. The very first step was to create the certificate and the key for my machine. Here’s where I used the WSL (CentOS) to use the OpenSSL commands. But if you don’t want to install WSL, the easiest way to get OpenSSL on your Windows box is by installing Git for Windows and running the Git Bash utility that comes with it (it’ll open a command prompt window in which you can execute Linux commands) or PowerShell for Windows. If you have WSL, you may need to install the OpenSSL package, so you’ll need to follow your distribution updater package to get it. sudo apt install openssl Or sudo yum install openssl Note: You may want to upgrade your repositories and packages before installing it (sudo apt update && sudo apt upgrade / sudo yum update && sudo yum upgrade). Once the package is installed, you can run the OpenSSL command that will generate the key file and the certificate file we will use later To get the FQDN: $myFQDN = (Get-WmiObject Win32_ComputerSystem).DNSHostName + "." + (Get-WmiObject Win32_ComputerSystem).Domain >> Write-Host $myFQDN And to generate the key and certificate files: openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout keyfile.key -out certificatefile.crt -subj "/C=US/ST=Oregon/L=Portland/O=Company Name/OU=Org/CN=yourfqdn" -addext "subjectAltName=DNS :yourfqdn,DNS:*.tibco.com,IP:10.0.0.1" The output will be similar to these screenshots: You’ll need to change the values (highlighted in red) to match your company name, organization unit, country, etc… (not really needed so that you can use the ones in the sample). Still, the most important part is the Common Name (CN), which’s the one that needs to match the URL you’ll be using on your browser to access WebFOCUS, and it has to contain the FQDN. The key thing here is that Solr expects to have a valid domain to create a real secure connection between environments. Note: I’ve also added as DNS the entire tibco.com domain, that way, this will work even if I add my region, like: https://machinename.emea.tibco.com Now that we have created the key and the certificate, we need to store it under a keystore, so the next step is creating it based on them: openssl pkcs12 -export -in tibco-pc1t3xv0.crt -inkey tibco-pc1t3xv0.key -out tibco-pc1t3xv0.p12 You’ll be prompted for a password, so don’t forget which one you’ve selected as it’ll be used later. Lastly, we need this recently created certificate to be “trusted” by our environment, in case you can’t certify it with a CA (Certificate Authority like VeriSign, IdenTrust, DigiCert, Let's Encrypt, GoDaddy, or similar) or already have a certified one by these authorities, which is the recommended for Production environments, and even more if they are publicly available. Your server has a JDK installed (requirement for WebFOCUS) so, in order to validate it ‘internally’ we’ll use the cacerts keystore that comes with the JDK installation. You can check it’s content by executing the following command from the command prompt: For Java11: keytool -v -list -cacerts For Java 8: keytool -v -list -keystore <path to cacerts>\cacertsfile.ext Note: Keytool is a tool also provided in the JDK, so it has to be in the PATH variable of the OS to be able to execute it anywhere, if you don’t have it there, you can use the entire path to it to make it work. You’ll be prompted for a password, and the default one is ‘changeit’ (without quotes). You’ll see a bunch of certificates scrolling on your screen (usually around ~99). So if you also want to review them, just send that output to a text file you can check later: Java11: keytool -v -list -cacerts > C:\Temp\cacerts_content.txt Java8: keytool -v -list -keystore <path to cacerts>\cacertsfile.ext > C:\Temp\cacerts_content.txt The cacerts file is usually stored under %JAVA_HOME%\lib\security (or $JAVA_HOME/lib/security in Unix/Linux environments), but depending on how you have your environment configured, you may be using the one that comes with WebFOCUS (C:\ibi\WebFOCUS93\jdk) or Tomcat (C:\ibi\tomcat\jdk), not adding the -keystore parameter we’re making sure that we are importing our certificate in the cacerts file that the OS is reading. The command to import our certificate into that cacert file is: keytool -import -alias {aliasname} -cacerts -file {certificatename.cer} So you should be using something like: Java11: keytool -import -alias mylocalcert -cacerts -file tibco-pc1t3xv0.crt Java8: keytool -import -alias WF8207SSL -keystore <path to cacerts>\cacerts -file <path to crt>\WF8207SSL.crt keytool -import -alias mylocalcert -cacerts -file tibco-pc1t3xv0.crt Again, you’ll be prompted for the cacerts password and it’ll also ask you if you trust the certificate you are importing (obviously, you trust it, as you are the one who has created it). Now our certificate is also trusted, and these files will now be used in our WebFOCUS installation. I recommend applying them to Tomcat first, so we’ll be able to access WebFOCUS using SSL, then apply it to Solr, and finally connect both. So, even if it’s not needed, as long as you want to secure the connection between the WebFOCUS Client and Solr, you’ll probably want to secure the WebFOCUS Client as well, so now that we have the certificate, the key, and the keystore, we should be able to make that happen. For Tomcat, is as simple as adding the following block to the server.xml file located under Tomcat’s ‘conf’ folder: <!-- IBI SSL Port --> <Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol" maxThreads="150" maxPostSize="-1" URIEncoding="UTF-8" SSLEnabled="true" scheme="https" secure="true" keystoreFile = "C:\ibi\certs\tibco-pc1t3xv0.p12" keystoreType = "PKCS12" keystorePass = "keystorepasswd" ciphers="TLS_RSA_WITH_AES_128_CBC_SHA" clientAuth="false" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2"/> <!-- IBI SSL Port END --> After that, you just need to restart Tomcat and you should be able to access via http (if you didn’t disable the 8080 port) and https. Note: If you still don’t get the ‘Certificate is valid’, just keep going, there’s an ‘Other Tips’ section at the end of the Article that may help you with this as well. If you don’t use the FQDN, you’ll see that the connection will appear as insecure (as the URL doesn’t match the certificate) even it’s using SSL: If you have any issues at any of the steps we’ve taken, don’t hesitate to write me at pablo.alvarez@cloud.com If you are using a different Application Server, you can also ping me or open a Support Ticket case. Do let me know if you want me to write another article about certificates by sending me an email requesting it 😉 Now, we should focus on the Solr configuration. You have a ‘solr.in.*’ file (.sh for Unix/Linux, .cmd for Windows) under C:\ibi\WebFOCUS93\Solr\solr\bin (or ../ibi/WebFOCUS93/Solr/solr/bin if you are using a Linux/Unix environment), edit it and uncomment the following lines (and make sure the values matches the ones for your environment): set SOLR_SSL_ENABLED=true set SOLR_SSL_KEY_STORE=C:\ibi\certs\local\tibco-pc1t3xv0.p12 set SOLR_SSL_KEY_STORE_PASSWORD=keystorepasswd set SOLR_SSL_TRUST_STORE=C:\ibi\certs\local\tibco-pc1t3xv0.p12 set SOLR_SSL_TRUST_STORE_PASSWORD=keystorepasswd set SOLR_SSL_NEED_CLIENT_AUTH=false set SOLR_SSL_WANT_CLIENT_AUTH=false set SOLR_SSL_CLIENT_HOSTNAME_VERIFICATION=false set SOLR_SSL_CHECK_PEER_NAME=true set SOLR_SSL_KEY_STORE_TYPE=PKCS12 set SOLR_SSL_TRUST_STORE_TYPE=PKCS12 I also recommend uncommenting and changing the following line to get a more detailed log the first time to debug the issues you could face, and once it’s working properly, set it back to the default value (INFO) or even comment it out: set SOLR_LOG_LEVEL=INFO If all the previous steps were correctly performed, you should be able to restart the Solr service now and be able to access the Solr Dashboard via https: https://servername.company.ext:8983/solr The latest configuration part is to tell WebFOCUS that the connection between them is now being made using SSL, so you’ll need to access your WebFOCUS Client Administration Console Configuration and change the Solr Url to match the one you used to access the Dashboard: And that should be all, you are now connected in an even more secure way and performing secure searches within your data and WebFOCUS: TROUBLESHOOTING During the investigation on how to enable SSL (and make it work with WebFOCUS), -John Calappi (who was also an important part on this Technical Article and you can contact him as well 😜), and I found some issues. Even if we were able to start Solr with SSL, WebFOCUS wasn’t able to communicate with it and the message you receive is as follows: This is why I recommend you configure the DEBUG mode on Solr, to try to figure out why it’s not working. Our magnify_search.log files from WebFOCUS showed this messages: [2024-04-17 12:45:21,648] INFO main - {} - SolrSearchClientFactory.init() initializing Solr client with: url: https://tibco-pc1t3xv0:8983/solr, username: , collection: ibi-protected [2024-04-17 12:45:21,650] INFO main - {} - createSolrClient(): created SolrClient for url: https://tibco-pc1t3xv0:8983/solr [2024-04-17 12:45:21,672] ERROR main - {} - testClient(): error: IOException occurred when talking to server at: https://tibco-pc1t3xv0:8983/solr [2024-04-17 12:45:21,673] ERROR main - {} - createSolrClient(): Error testing Solr client [2024-04-17 12:45:21,673] ERROR main - {} - Error! Creating SolrClient Even the Solr service was started and we were able to access the Solr Dashboard using SSL. Checking the Solr logs in DEBUG mode showed us that the ‘handshake’ wasn’t being performed. The ‘handshake’ is when you try to communicate with the endpoint securely, but the endpoint doesn’t trust you and doesn’t perform the handshake. 2024-04-08 12:38:40.340 WARN (main) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@20b9d5d5[provider=null,keyStore=file:///C:/ibi/certs/local/mokochino_pkcs12.jks,trustStore=file:///C:/Program%20Files/Java/jdk-11/lib/security/cacerts] 2024-04-08 12:38:42.993 DEBUG (qtp1489193907-23) [ ] o.e.j.i.s.SslConnection fill NOT_HANDSHAKING 2024-04-08 12:38:43.401 DEBUG (qtp1489193907-27) [ ] o.e.j.i.s.SslConnection DecryptedEndPoint@569cafab{l=/10.98.96.15:8983,r=/10.98.96.15:60038,OPEN,fill=-,flush=-,to=414/120000} stored fill exception => javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown We got these messages when the certificate wasn’t added to the cacerts file, and also when the keystore we used didn’t have the .key attached to it, so it was considered a ‘public’ certificate (like having a low fence or an open padlock, both are secure methods, but used incorrectly, so not secure at all). In order to have a proper handshake between products, the keystore needs to have the required files to accept the connections. OTHER TIPS Make sure that you are adding your certificate to the proper cacerts file, sometimes, there are several Java JDK releases installed and each one has its own cacerts file. For example, when you install WebFOCUS, sometimes it also includes its own jdk (C:\ibi\WebFOCUS93\jdk) or even under Tomcat you can have another one (C:\ibi\tomcat\jdk), Linux/UNIX environments doesn’t usually have them, but Windows does. In that case, I usually delete those ones and use the mklink command to create symbolic links (similar to the Linux ones, not Windows shortcuts) just in case that there’s some hardcoded path pointing there. You could use it as follows: mklink /D {destination} {source} Sample: mklink /D C:\ibi\WebFOCUS93\jdk C:\Progra~1\Java\jdk11 It is also recommended to add your new certificate into your client OS, so any browser you use can also trust this one on their side. From your Windows OS, double-click on your .crt file (you can copy the file from the server or the content of it as it’s just a plain-text file) and you’ll see something like this: Click on the Install Certificate… and place it under the Trusted Root Certification Authorities: After finishing this process, you’ll see that certificate as valid: Another option to have SSL in your WebFOCUS Installation that Ben Naphtali also suggested is to put in front of our install a Load Balancer like NGINX and have that using SSL. That way, even if you don’t have SSL configured in WebFOCUS, you won’t be able to access it without going first to NGINX in a secure way (NGINX will manage and redirect the petitions to WebFOCUS and Solr). NGINX, Apache WebServer using the mod_proxy module or mod_jk, or any Load Balancer should work for this. As said before, let me know if you want me to write another Technical Article about how to properly install the Load Balancer in front of your WebFOCUS install to secure it, this will also allow you to have different WebFOCUS Clients to handle HighAvailability features in production environments. You could also enable SSL in the WebFOCUS Reporting Server Side, but that part is perfectly described in the Security & Administration Manual, so you just need to follow that one to make it work. Happy & Secure connections! Pablo Alvarez