Has anyone else heard about this lo4j 2 vulnerability ...

[Warren Hinchliffe]

Has anyone else heard about this lo4j 2 vulnerability

 

 

 

abc.net.au 11 Dec 21

 

 

 

 

'Internet's on fire': Users around the world at risk amid the 'most critical...

 

Within12 hours of being disclosed, a bug is "fully weaponised", sparking a race between hackers and the cybersecurity companies trying to head them off.

 

 

 

 

 

 

 

 

 

 

Description of the Vulnerability (CVE-2021-44228)

The Apache log4j library allows for developers to log various data within their application. In certain circumstances, the data being logged originates from user input. Should this user input contain special characters and be subsequently logged within the context of log4j, the Java method lookup will finally be called to execute the user-defined remote Java class in the LDAP server. This will in turn lead to RCE on the victim server that uses the vulnerable log4j 2 instance.

Executive Summary

On Dec. 9, 2021, a remote code execution (RCE) vulnerability in Apache log4j 2 was identified being exploited in the wild. Public proof of concept (PoC) code was released and subsequent investigation revealed that exploitation was incredibly easy to perform. By submitting a specially crafted request to a vulnerable system, depending on how the system is configured, an attacker is able to instruct that system to download and subsequently execute a malicious payload. Due to the discovery of this exploit being so recent, there are still many servers, both on-premises and within cloud environments, that have yet to be patched. Like many high severity RCE exploits, thus far, massive scanning activity for CVE-2021-44228 has begun on the internet with the intent of seeking out and exploiting unpatched systems. We highly recommend that organizations upgrade to the latest version (2.15.0-rc2) of Apache log4j 2 for all systems.

[Warren Hinchliffe]

Here is a link to TIBCOs article on the issue.

https://support.tibco.com/s/article/Apache-Log4J-Vulnerability-and-Impact-to-TIBCO-Products-and-Services

[Bill Dykema]

Not seeing anything on WebFOCUS specifically yet: https://www.tibco.com/support/notices/2021/12/apache-log4j-vulnerability-update

[Warren Hinchliffe]

Im hoping that TIBCO is like a duck in water, not much above the surface, and frantically moving under.

[Toby Mills]

Im getting lots of questions out of the blue about it.

I need to understand the vulnerability better to understand how to protect us from that (if possible).

Will post if I figure something out that we can do.

[Toby Mills]

warren.hinchliffe:

 

We highly recommend that organizations upgrade to the latest version (2.15.0-rc2) of Apache log4j 2 for all systems.

 

 

Im going to read up on this My customers are pretty picky about this.

They only mention log4j 2. My 8207 uses that of course, but my older 8105m is just log4j.

If I can find and try the POC, Ill know if weve really got a problem.

[Warren Hinchliffe]

I dont think log4j v1.x is affected

[Toby Mills]

Looks like its not.

And the link to download the newer jars for log4j 2 are here:

Download Apache Log4j 2

To find your current version thats installed on your machine, looks like you can go to:

Windows:

ibiWebFOCUS82webappswebfocusWEB-INFlib

Search on things starting with log4j. Heres what I see in 8207.28

 

image.png1257501 47.9 KB

 

When you unzip the newer 2.15 version, you see extra files for each of these in my picture. Theres always a javadoc and a sources.

heres a quick shot of Beyond Compare between the new 2.15 (on the left) and my 8207.28 on the right:

 

image.png1647653 48.4 KB

 

You can see there will be quite a few extra files we wouldnt need that will be packaged up in the new log4j2 2.15 zip.

Havent actually tried swapping these files in yet. I do have a spare client I could experiment on.

But first- if I dont set off too many alarms, I always like to demonstrate the vulnerability first, then try to fix it, then re-try the test that hopefully will show the vulnerability is gone.

Thats it for now

[Warren Hinchliffe]

TIBCO has release a HOTFIX for the vulnerability.

https://support.tibco.com/s/article/TIBCO-WebFOCUS-8207-28-05-HF-001-is-available

[Toby Mills]

I just saw this too. Better that it comes from them than me trying to fix it up myself.

Ill try to install tomorrow.

[Warren Hinchliffe]

Me too, the dangers ate too much to not do it.

[Toby Mills]

This is the multi step process we were asked to do up till hotfix 3. Hotfix 4 was the first client side fix I applied because I liked having the installer. The installer used to take overnight to generate. I know they were in a hurry.

I would wager, aside from myself and a few dedicated WF admins (and those of us who worked at IBI in the past), most people have not done this lengthy multi step (human error prone) sort of hotfix.

Most times, a hotfix isnt all that important and you can elect to skip it. Not this one though.

I really feel for the support staff at TIBCO over the next few days.

To be safest, I think Ill update to the recently posted 8207.28.05 and then apply the hotfix. The speed of turnaround makes me think its best not to try to lay down the hotfix over older 8207.28s. I suspect itd work, but Ill just spend the extra time reconfiguring. I think I have about 8 to 12 VMs to update to the 8207.28.05 and then apply hotfixes. Its gonna be a long day. Im super lucky that were not in production yet with all the new machines.

[Warren Hinchliffe]

I applied the hotfix and seems it has broken ReportCaster

Getting the error Error: Could not find or load main class ibi.broker.SCHScheduler

Im really worried now

[David Hall 5]

Careful notation on the process of applying the available hotfix

 

it is a hotfix to 8207.28.05.

You cannot apply the hotfix to older releases of 8207

so, you must

update the WF to 8207.28.05

go through the process of manually applying the hotfix 01 to the 8207.28.05

Also, note: there is a separate hotfix for the WFRS but the jar files are not named the same

I have asked if this is done on purpose. Waiting to find out

 

If you need a fix for an earlier release, you are going to need to open a case and request it

[David Hall 5]

On this issue, the following information from the apache web site

https://logging.apache.org/log4j/2.x/index.html

CVE-2021-44228

For those who cannot upgrade to 2.15.0, in releases >=2.10, this vulnerability can be mitigated by setting either the system property log4j2.formatMsgNoLookups

or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true.

For releases from 2.7 through 2.14.1 all PatternLayout patterns can be modified to specify the message converter as %m{nnolookups} instead of just %m.

For releases from 2.0-beta9 to 2.10.0, the mitigation is to remove the JndiLookup class from the classpath:zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

I asked this specific question in a case, waiting to hear a reply

Setting an environment variable to me seems the fastest way to do this.

We are currently testing the hotfix install.

Seeing the note on RC, problems causes me concerns

If that fails on us, we will open a case, but we are also going to test the environment variable suggestion by Apache.

8206 is currently using log4j 2.11

8207 (not the hotfix) is using Log4j 2.14

So based on the suggestion it might be worth someone trying

[David Hall 5]

For those who are interested, or Security team put the following on the Windows Server properties

System environments settings

variable name LOG4J_FORMAT_MSG_NO_LOOKUPS

variable value true

[Toby Mills]

Thanks @david.hall and @warren.hinchliffe

Im was considering the use of the system variable David just showed but didnt know exactly where to put it yet so the example of an environment variable helps.

The server names are the same but the dates are different (the readme notes this).

Im going to compare the jars to see if I can find a missing class thats throwing off caster.

[David Hall 5]

You can add the system variable either from the GUI or you could also add it via regedit

[Warren Hinchliffe]

I considered the variable, and asked about it in my case, but got not response at all.

Add at your own risk.

At this point I have applied the fix to .04, so ended up with a Frankenstein install. Its all on me, as I didnt see the line for .05 install.

Had to back out ReportCaster fix to get it to work again, but cant back out the Client or Reporting server on Prod, its out of the maintenance window.

It does seem to be working though.

Im hoping its be cause there were only two non log4j jar files. one for RC, the other goes under utilities.

Desperately hoping for the .04 fix.

[David Hall 5]

I also asked about a the system variable. Heard nothing

However this is the recommendation from Apache.

Our security people forced this on all servers. so that says something

One note on the jar files. The WFRS jar files in the Reporting server hotfix are not listed as 2.15, but are listed as 2.11 . There is disclaimer that the same name is on purpose, but why maintain the wrong release level name. I asked about this and have not hear anything.

Just curious has anyone applied the Hotfix and the RC server worK after the manual copypastereplace

[Toby Mills]

Hey @warren.hinchliffe, make sure you have the webfocus-caster-server-8207.28.05.jar file back in the caster lib folder. That appears to be the place the class resides.

[Warren Hinchliffe]

I have put the old RC jar files back in for now and its working, but I guess vulnerable again.

As for the LOG4J_FORMAT_MSG_NO_LOOKUPS env variable. We talked about it in our weekly catch with support and they said (Verbally) (cant remember the words exactly so dont quote me) we do not recommend applying it due to the many ways it may be used.

[Toby Mills]

That sounds like what theyd have to say from a liability perspective.

I hit an interesting thing on my first machine Im in the midst of. First I upgraded to 8207.28.05.

Then Im stepping through the instructions for Solr and saw this difference in filenames. I dont think itll matter, but figured Id put it here so you guys can see.

Readme says

 

image.png844209 5.38 KB

 

But my files are a little older version:

 

image.png1150514 173 KB

 

Probably going to be fine, but I thought Id mention it.

Same versions in this folder (not the 2.13s)

D:ibiWebFOCUS82Solrsolrcontribprometheus-exporterlib

Cant wait to turn everything on and see what blows up

[Warren Hinchliffe]

I saw the same thing. I assume whoever wrote the readme just didnt put the right version on the move.

As they are being replaced, should be no issue. At lease none from our install yet.

[David Hall 5]

Just a heads up on the use of the LOG4J_FORMAT_MSG_NO_LOOKUPS system environment setting

Our company pushed this out to ALL windows systems as a patch, to give us time to wait for the vendor solution.

On the patches and names of jar files. the Reporting Server file you are copying as part of the fix is still named 2.11 and not 2.14

Also TIBCO will be re-issuing the hotfix with logj4 2.16 version shortly. So you may want to hold off on manually doing any copybackupreplace until that is posted