nonce cache thread proliferation

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

nonce cache thread proliferation

massimiliano.masi
Hi All,

I updated from wss4j 1.6 to wss4j 2.0.7. I observe the following behaviour (using wildfly 9 and java 1.8.0_60).

With Nonce Replay Cache, Saml One Time Use Replacy Cache, Timestamp Replay Cache set to true, there is a linear thread proliferation: 1 call, 1 thread open and not closed.

With all those caches set to false, the threads are not proliferating and everything is fine.

Threads are watched as: watch -n 1 “date && cat /proc/<PID>/status | grep Threads”

In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

Thus the linear proliferation observed seems to be a new instance of the ScheduledThreadPoolExecutor.

Thus: 2000 web service requests, 2000 new threads spawned and not closed.

Any advice?

Thanks a lot,

        Massimiliano


--
Anger is a gift, http://www.mascanc.net/

Reply | Threaded
Open this post in threaded view
|

Re: nonce cache thread proliferation

Colm O hEigeartaigh-2
Hi Massimiliano,

This sounds like a serious problem, although possibly EhCache related rather than WSS4J.

Firstly, WSS4J 2.0.7 uses EhCache 2.8.5 not 2.9.0 - could you verify that the same behaviour occurs with 2.8.5?

If you disable the diskstore by changing the EhCache configuration, does it solve the problem?

Colm.

On Fri, Jul 8, 2016 at 1:41 PM, massimiliano.masi <[hidden email]> wrote:
Hi All,

I updated from wss4j 1.6 to wss4j 2.0.7. I observe the following behaviour (using wildfly 9 and java 1.8.0_60).

With Nonce Replay Cache, Saml One Time Use Replacy Cache, Timestamp Replay Cache set to true, there is a linear thread proliferation: 1 call, 1 thread open and not closed.

With all those caches set to false, the threads are not proliferating and everything is fine.

Threads are watched as: watch -n 1 “date && cat /proc/<PID>/status | grep Threads”

In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

Thus the linear proliferation observed seems to be a new instance of the ScheduledThreadPoolExecutor.

Thus: 2000 web service requests, 2000 new threads spawned and not closed.

Any advice?

Thanks a lot,

        Massimiliano


--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: nonce cache thread proliferation

massimiliano.masi
Hi, 

I performed the following tests. 

1) See which executor service is parked, using jconsole and visual vm.

The exception is the following: 

Name: wss4j%002etimestamp%002ecache-%004crxid%0045s%004ccr%004ci%0041g==.data

State: TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c835ce1

Total blocked: 0  Total waited: 21

 

Stack trace:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

java.lang.Thread.run(Thread.java:745)


2) Change to 2.8.5, and observe the output. Same exception, 



3) Set the inmemory only cache. I am not sure here, I recompiled by changing the

cat wss4j-ehcache.xml
<ehcache xsi:noNamespaceSchemaLocation="ehcache.xsd" updateCheck="false" monitoring="autodetect" dynamicConfig="true" name="wss4jCache">
   
   
<!--diskStore path="java.io.tmpdir"/-->
   
<defaultCache
        maxElementsInMemory="500"
        eternal="false"
        overflowToDisk="false"
        overflowToOffHeap="false"
        memoryStoreEvictionPolicy="LRU" >
       
<persistence strategy="none" />
   
</defaultCache>
   
   
<!--defaultCache
        maxEntriesLocalHeap="5000"
        eternal="false"
        timeToIdleSeconds="3600"
        timeToLiveSeconds="3600"
        overflowToDisk="true"
        maxElementsOnDisk="10000000"
        diskPersistent="false"
        diskExpiryThreadIntervalSeconds="120"
        memoryStoreEvictionPolicy="LRU"
        /-->
   
</ehcache>

With the same results. Is there another way to set it up, or to not use ehcache, but another caching mechanism? For instance, I see the MemoryReplayCache that can be used to test. 



In my opinion the issue seems to be related to ehcache. I don’t think that it can be due to my programming mistakes. 

Here is my code: 

WSSConfig config = WSSConfig.getNewInstance();

config.setProcessor(new QName(WSConstants.SAML2_NS, "Assertion"), LocalSAMLTokenProcessor.class);
config.setProcessor(WSSecurityEngine.SAML_TOKEN, LocalSAMLTokenProcessor.class);

WSSecurityEngine secEngine = new WSSecurityEngine();
secEngine.setWssConfig(config);
RequestData rd = new RequestData();
rd.setEnableNonceReplayCache(true);
rd.setEnableSamlOneTimeUseReplayCache(true);
rd.setEnableTimestampReplayCache(true);
rd.setCallbackHandler(cbHandler);
rd.setSigVerCrypto(fakeCrypto);
rd.setDecCrypto(fakeCrypto);
rd.setActor(null);



List<WSSecurityEngineResult> results = secEngine.processSecurityHeader(WSSecurityUtil.getSecurityHeader(envelope, null), 
rd);


Thanks again! 



Il giorno 11 lug 2016, alle ore 11:32, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

This sounds like a serious problem, although possibly EhCache related rather than WSS4J.

Firstly, WSS4J 2.0.7 uses EhCache 2.8.5 not 2.9.0 - could you verify that the same behaviour occurs with 2.8.5?

If you disable the diskstore by changing the EhCache configuration, does it solve the problem?

Colm.

On Fri, Jul 8, 2016 at 1:41 PM, massimiliano.masi <[hidden email]> wrote:
Hi All,

I updated from wss4j 1.6 to wss4j 2.0.7. I observe the following behaviour (using wildfly 9 and java 1.8.0_60).

With Nonce Replay Cache, Saml One Time Use Replacy Cache, Timestamp Replay Cache set to true, there is a linear thread proliferation: 1 call, 1 thread open and not closed.

With all those caches set to false, the threads are not proliferating and everything is fine.

Threads are watched as: watch -n 1 “date && cat /proc/<PID>/status | grep Threads”

In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

Thus the linear proliferation observed seems to be a new instance of the ScheduledThreadPoolExecutor.

Thus: 2000 web service requests, 2000 new threads spawned and not closed.

Any advice?

Thanks a lot,

        Massimiliano


--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/

Reply | Threaded
Open this post in threaded view
|

Re: nonce cache thread proliferation

massimiliano.masi
In reply to this post by Colm O hEigeartaigh-2
Hi, 

An additional point: it seems not to be related to wildfly. The same code executed in a junit
produces the same results: 





Il giorno 11 lug 2016, alle ore 11:32, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

This sounds like a serious problem, although possibly EhCache related rather than WSS4J.

Firstly, WSS4J 2.0.7 uses EhCache 2.8.5 not 2.9.0 - could you verify that the same behaviour occurs with 2.8.5?

If you disable the diskstore by changing the EhCache configuration, does it solve the problem?

Colm.

On Fri, Jul 8, 2016 at 1:41 PM, massimiliano.masi <[hidden email]> wrote:
Hi All,

I updated from wss4j 1.6 to wss4j 2.0.7. I observe the following behaviour (using wildfly 9 and java 1.8.0_60).

With Nonce Replay Cache, Saml One Time Use Replacy Cache, Timestamp Replay Cache set to true, there is a linear thread proliferation: 1 call, 1 thread open and not closed.

With all those caches set to false, the threads are not proliferating and everything is fine.

Threads are watched as: watch -n 1 “date && cat /proc/<PID>/status | grep Threads”

In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

Thus the linear proliferation observed seems to be a new instance of the ScheduledThreadPoolExecutor.

Thus: 2000 web service requests, 2000 new threads spawned and not closed.

Any advice?

Thanks a lot,

        Massimiliano


--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/

Reply | Threaded
Open this post in threaded view
|

Re: nonce cache thread proliferation

Colm O hEigeartaigh-2
Hi Massimiliano,

> In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private
>final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
>of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

I'm a bit confused by this. Why is the diskstore shutting down after every invocation? Are you seeing a new cache file created for each invocation?

Colm.

On Tue, Jul 12, 2016 at 2:30 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 

An additional point: it seems not to be related to wildfly. The same code executed in a junit
produces the same results: 





Il giorno 11 lug 2016, alle ore 11:32, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

This sounds like a serious problem, although possibly EhCache related rather than WSS4J.

Firstly, WSS4J 2.0.7 uses EhCache 2.8.5 not 2.9.0 - could you verify that the same behaviour occurs with 2.8.5?

If you disable the diskstore by changing the EhCache configuration, does it solve the problem?

Colm.

On Fri, Jul 8, 2016 at 1:41 PM, massimiliano.masi <[hidden email]> wrote:
Hi All,

I updated from wss4j 1.6 to wss4j 2.0.7. I observe the following behaviour (using wildfly 9 and java 1.8.0_60).

With Nonce Replay Cache, Saml One Time Use Replacy Cache, Timestamp Replay Cache set to true, there is a linear thread proliferation: 1 call, 1 thread open and not closed.

With all those caches set to false, the threads are not proliferating and everything is fine.

Threads are watched as: watch -n 1 “date && cat /proc/<PID>/status | grep Threads”

In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

Thus the linear proliferation observed seems to be a new instance of the ScheduledThreadPoolExecutor.

Thus: 2000 web service requests, 2000 new threads spawned and not closed.

Any advice?

Thanks a lot,

        Massimiliano


--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: nonce cache thread proliferation

massimiliano.masi
Hi, 


Yes, as you can see from the attached images of last mail. 
In particular is a timestamp cache.


Il giorno 18 lug 2016, alle ore 12:41, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

> In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private
>final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
>of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

I'm a bit confused by this. Why is the diskstore shutting down after every invocation? Are you seeing a new cache file created for each invocation?

Colm.

On Tue, Jul 12, 2016 at 2:30 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 

An additional point: it seems not to be related to wildfly. The same code executed in a junit
produces the same results: 





Il giorno 11 lug 2016, alle ore 11:32, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

This sounds like a serious problem, although possibly EhCache related rather than WSS4J.

Firstly, WSS4J 2.0.7 uses EhCache 2.8.5 not 2.9.0 - could you verify that the same behaviour occurs with 2.8.5?

If you disable the diskstore by changing the EhCache configuration, does it solve the problem?

Colm.

On Fri, Jul 8, 2016 at 1:41 PM, massimiliano.masi <[hidden email]> wrote:
Hi All,

I updated from wss4j 1.6 to wss4j 2.0.7. I observe the following behaviour (using wildfly 9 and java 1.8.0_60).

With Nonce Replay Cache, Saml One Time Use Replacy Cache, Timestamp Replay Cache set to true, there is a linear thread proliferation: 1 call, 1 thread open and not closed.

With all those caches set to false, the threads are not proliferating and everything is fine.

Threads are watched as: watch -n 1 “date && cat /proc/<PID>/status | grep Threads”

In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

Thus the linear proliferation observed seems to be a new instance of the ScheduledThreadPoolExecutor.

Thus: 2000 web service requests, 2000 new threads spawned and not closed.

Any advice?

Thanks a lot,

        Massimiliano


--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/

Reply | Threaded
Open this post in threaded view
|

Re: nonce cache thread proliferation

Colm O hEigeartaigh-2

Hi Massimiliano,

I think the key issue here is how are you using WSS4J? Most users use WSS4J with Apache CXF. The WS-Security layer in CXF is responsible for managing the caches, storing them on the message context so that they get picked up on the next invocation, and shutting them down properly when the CXF endpoint comes down etc. WSS4J has no concept of a "context" that stores information for the next request, so it's up to the calling code to handle this.

In the test-code you provided, a cache will simply be created each time RequestData is initialized. After processing, data.getTimestampReplayCache().close() is never called meaning that the cache is not closed. That explains the proliferation of threads.

So in short, create the ReplayCache instance in the calling code once and set it on the RequestData for each request.

Colm.

On Mon, Jul 18, 2016 at 1:23 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 


Yes, as you can see from the attached images of last mail. 
In particular is a timestamp cache.


Il giorno 18 lug 2016, alle ore 12:41, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

> In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private
>final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
>of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

I'm a bit confused by this. Why is the diskstore shutting down after every invocation? Are you seeing a new cache file created for each invocation?

Colm.

On Tue, Jul 12, 2016 at 2:30 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 

An additional point: it seems not to be related to wildfly. The same code executed in a junit
produces the same results: 





Il giorno 11 lug 2016, alle ore 11:32, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

This sounds like a serious problem, although possibly EhCache related rather than WSS4J.

Firstly, WSS4J 2.0.7 uses EhCache 2.8.5 not 2.9.0 - could you verify that the same behaviour occurs with 2.8.5?

If you disable the diskstore by changing the EhCache configuration, does it solve the problem?

Colm.

On Fri, Jul 8, 2016 at 1:41 PM, massimiliano.masi <[hidden email]> wrote:
Hi All,

I updated from wss4j 1.6 to wss4j 2.0.7. I observe the following behaviour (using wildfly 9 and java 1.8.0_60).

With Nonce Replay Cache, Saml One Time Use Replacy Cache, Timestamp Replay Cache set to true, there is a linear thread proliferation: 1 call, 1 thread open and not closed.

With all those caches set to false, the threads are not proliferating and everything is fine.

Threads are watched as: watch -n 1 “date && cat /proc/<PID>/status | grep Threads”

In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

Thus the linear proliferation observed seems to be a new instance of the ScheduledThreadPoolExecutor.

Thus: 2000 web service requests, 2000 new threads spawned and not closed.

Any advice?

Thanks a lot,

        Massimiliano


--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: nonce cache thread proliferation

Colm O hEigeartaigh-2
FYI I merged a fix to disable creating ReplayCaches internally. To use this functionality you need to create the cache yourself (or use CXF) + set the instance on RequestData:

https://issues.apache.org/jira/browse/WSS-584

Colm.

On Mon, Jul 18, 2016 at 3:22 PM, Colm O hEigeartaigh <[hidden email]> wrote:

Hi Massimiliano,

I think the key issue here is how are you using WSS4J? Most users use WSS4J with Apache CXF. The WS-Security layer in CXF is responsible for managing the caches, storing them on the message context so that they get picked up on the next invocation, and shutting them down properly when the CXF endpoint comes down etc. WSS4J has no concept of a "context" that stores information for the next request, so it's up to the calling code to handle this.

In the test-code you provided, a cache will simply be created each time RequestData is initialized. After processing, data.getTimestampReplayCache().close() is never called meaning that the cache is not closed. That explains the proliferation of threads.

So in short, create the ReplayCache instance in the calling code once and set it on the RequestData for each request.

Colm.

On Mon, Jul 18, 2016 at 1:23 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 


Yes, as you can see from the attached images of last mail. 
In particular is a timestamp cache.


Il giorno 18 lug 2016, alle ore 12:41, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

> In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private
>final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
>of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

I'm a bit confused by this. Why is the diskstore shutting down after every invocation? Are you seeing a new cache file created for each invocation?

Colm.

On Tue, Jul 12, 2016 at 2:30 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 

An additional point: it seems not to be related to wildfly. The same code executed in a junit
produces the same results: 





Il giorno 11 lug 2016, alle ore 11:32, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

This sounds like a serious problem, although possibly EhCache related rather than WSS4J.

Firstly, WSS4J 2.0.7 uses EhCache 2.8.5 not 2.9.0 - could you verify that the same behaviour occurs with 2.8.5?

If you disable the diskstore by changing the EhCache configuration, does it solve the problem?

Colm.

On Fri, Jul 8, 2016 at 1:41 PM, massimiliano.masi <[hidden email]> wrote:
Hi All,

I updated from wss4j 1.6 to wss4j 2.0.7. I observe the following behaviour (using wildfly 9 and java 1.8.0_60).

With Nonce Replay Cache, Saml One Time Use Replacy Cache, Timestamp Replay Cache set to true, there is a linear thread proliferation: 1 call, 1 thread open and not closed.

With all those caches set to false, the threads are not proliferating and everything is fine.

Threads are watched as: watch -n 1 “date && cat /proc/<PID>/status | grep Threads”

In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

Thus the linear proliferation observed seems to be a new instance of the ScheduledThreadPoolExecutor.

Thus: 2000 web service requests, 2000 new threads spawned and not closed.

Any advice?

Thanks a lot,

        Massimiliano


--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com



--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: nonce cache thread proliferation

massimiliano.masi
Hi, 

Yes, I am not importing CXF, I am using wss4j as standalone library.

Many thanks for the fix, I will try it ASAP. Is it available already in 2.0.9?




Il giorno 18 lug 2016, alle ore 18:05, Colm O hEigeartaigh <[hidden email]> ha scritto:

FYI I merged a fix to disable creating ReplayCaches internally. To use this functionality you need to create the cache yourself (or use CXF) + set the instance on RequestData:

https://issues.apache.org/jira/browse/WSS-584

Colm.

On Mon, Jul 18, 2016 at 3:22 PM, Colm O hEigeartaigh <[hidden email]> wrote:

Hi Massimiliano,

I think the key issue here is how are you using WSS4J? Most users use WSS4J with Apache CXF. The WS-Security layer in CXF is responsible for managing the caches, storing them on the message context so that they get picked up on the next invocation, and shutting them down properly when the CXF endpoint comes down etc. WSS4J has no concept of a "context" that stores information for the next request, so it's up to the calling code to handle this.

In the test-code you provided, a cache will simply be created each time RequestData is initialized. After processing, data.getTimestampReplayCache().close() is never called meaning that the cache is not closed. That explains the proliferation of threads.

So in short, create the ReplayCache instance in the calling code once and set it on the RequestData for each request.

Colm.

On Mon, Jul 18, 2016 at 1:23 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 


Yes, as you can see from the attached images of last mail. 
In particular is a timestamp cache.


Il giorno 18 lug 2016, alle ore 12:41, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

> In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private
>final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
>of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

I'm a bit confused by this. Why is the diskstore shutting down after every invocation? Are you seeing a new cache file created for each invocation?

Colm.

On Tue, Jul 12, 2016 at 2:30 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 

An additional point: it seems not to be related to wildfly. The same code executed in a junit
produces the same results: 





Il giorno 11 lug 2016, alle ore 11:32, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

This sounds like a serious problem, although possibly EhCache related rather than WSS4J.

Firstly, WSS4J 2.0.7 uses EhCache 2.8.5 not 2.9.0 - could you verify that the same behaviour occurs with 2.8.5?

If you disable the diskstore by changing the EhCache configuration, does it solve the problem?

Colm.

On Fri, Jul 8, 2016 at 1:41 PM, massimiliano.masi <[hidden email]> wrote:
Hi All,

I updated from wss4j 1.6 to wss4j 2.0.7. I observe the following behaviour (using wildfly 9 and java 1.8.0_60).

With Nonce Replay Cache, Saml One Time Use Replacy Cache, Timestamp Replay Cache set to true, there is a linear thread proliferation: 1 call, 1 thread open and not closed.

With all those caches set to false, the threads are not proliferating and everything is fine.

Threads are watched as: watch -n 1 “date && cat /proc/<PID>/status | grep Threads”

In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

Thus the linear proliferation observed seems to be a new instance of the ScheduledThreadPoolExecutor.

Thus: 2000 web service requests, 2000 new threads spawned and not closed.

Any advice?

Thanks a lot,

        Massimiliano


--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com



--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/

Reply | Threaded
Open this post in threaded view
|

Re: nonce cache thread proliferation

Colm O hEigeartaigh-2
Yes, it is available in 2.0.9. The "fix" is that to enable caching, you must create the ReplayCache instances yourself and set them on the RequestData Object.

Colm.

On Mon, Aug 1, 2016 at 2:44 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 

Yes, I am not importing CXF, I am using wss4j as standalone library.

Many thanks for the fix, I will try it ASAP. Is it available already in 2.0.9?




Il giorno 18 lug 2016, alle ore 18:05, Colm O hEigeartaigh <[hidden email]> ha scritto:

FYI I merged a fix to disable creating ReplayCaches internally. To use this functionality you need to create the cache yourself (or use CXF) + set the instance on RequestData:

https://issues.apache.org/jira/browse/WSS-584

Colm.

On Mon, Jul 18, 2016 at 3:22 PM, Colm O hEigeartaigh <[hidden email]> wrote:

Hi Massimiliano,

I think the key issue here is how are you using WSS4J? Most users use WSS4J with Apache CXF. The WS-Security layer in CXF is responsible for managing the caches, storing them on the message context so that they get picked up on the next invocation, and shutting them down properly when the CXF endpoint comes down etc. WSS4J has no concept of a "context" that stores information for the next request, so it's up to the calling code to handle this.

In the test-code you provided, a cache will simply be created each time RequestData is initialized. After processing, data.getTimestampReplayCache().close() is never called meaning that the cache is not closed. That explains the proliferation of threads.

So in short, create the ReplayCache instance in the calling code once and set it on the RequestData for each request.

Colm.

On Mon, Jul 18, 2016 at 1:23 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 


Yes, as you can see from the attached images of last mail. 
In particular is a timestamp cache.


Il giorno 18 lug 2016, alle ore 12:41, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

> In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private
>final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
>of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

I'm a bit confused by this. Why is the diskstore shutting down after every invocation? Are you seeing a new cache file created for each invocation?

Colm.

On Tue, Jul 12, 2016 at 2:30 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 

An additional point: it seems not to be related to wildfly. The same code executed in a junit
produces the same results: 





Il giorno 11 lug 2016, alle ore 11:32, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

This sounds like a serious problem, although possibly EhCache related rather than WSS4J.

Firstly, WSS4J 2.0.7 uses EhCache 2.8.5 not 2.9.0 - could you verify that the same behaviour occurs with 2.8.5?

If you disable the diskstore by changing the EhCache configuration, does it solve the problem?

Colm.

On Fri, Jul 8, 2016 at 1:41 PM, massimiliano.masi <[hidden email]> wrote:
Hi All,

I updated from wss4j 1.6 to wss4j 2.0.7. I observe the following behaviour (using wildfly 9 and java 1.8.0_60).

With Nonce Replay Cache, Saml One Time Use Replacy Cache, Timestamp Replay Cache set to true, there is a linear thread proliferation: 1 call, 1 thread open and not closed.

With all those caches set to false, the threads are not proliferating and everything is fine.

Threads are watched as: watch -n 1 “date && cat /proc/<PID>/status | grep Threads”

In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

Thus the linear proliferation observed seems to be a new instance of the ScheduledThreadPoolExecutor.

Thus: 2000 web service requests, 2000 new threads spawned and not closed.

Any advice?

Thanks a lot,

        Massimiliano


--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com



--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: nonce cache thread proliferation

massimiliano.masi
Hi,  

I think I am doing something wrong here (2.0.9). 

I created (to debug) my own instance of ReplayCache, in a static block: 

private static ReplayCache noncecache;

private static ReplayCache samlcache;

private static ReplayCache tscache;

static {

noncecache = new MicroCacheReplayCache();
samlcache = new MicroCacheReplayCache();
tscache = new MicroCacheReplayCache();
}


and then executed as: 

WSSecurityEngine secEngine = new WSSecurityEngine();
secEngine.setWssConfig(config);
RequestData rd = new RequestData();



rd.setNonceReplayCache(noncecache);
rd.setTimestampReplayCache(tscache);
rd.setSamlOneTimeUseReplayCache(samlcache);
rd.setCallbackHandler(cbHandler);
rd.setSigVerCrypto(fakeCrypto);
rd.setDecCrypto(fakeCrypto);
rd.setActor(null);


List<WSSecurityEngineResult> results = secEngine.processSecurityHeader(WSSecurityUtil.getSecurityHeader(envelope, null), 
rd);

But only the constructor is called. The contains() or the add() methods are never called. Am I missing something obvious?


Il giorno 02 ago 2016, alle ore 10:33, Colm O hEigeartaigh <[hidden email]> ha scritto:

Yes, it is available in 2.0.9. The "fix" is that to enable caching, you must create the ReplayCache instances yourself and set them on the RequestData Object.

Colm.

On Mon, Aug 1, 2016 at 2:44 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 

Yes, I am not importing CXF, I am using wss4j as standalone library.

Many thanks for the fix, I will try it ASAP. Is it available already in 2.0.9?




Il giorno 18 lug 2016, alle ore 18:05, Colm O hEigeartaigh <[hidden email]> ha scritto:

FYI I merged a fix to disable creating ReplayCaches internally. To use this functionality you need to create the cache yourself (or use CXF) + set the instance on RequestData:

https://issues.apache.org/jira/browse/WSS-584

Colm.

On Mon, Jul 18, 2016 at 3:22 PM, Colm O hEigeartaigh <[hidden email]> wrote:

Hi Massimiliano,

I think the key issue here is how are you using WSS4J? Most users use WSS4J with Apache CXF. The WS-Security layer in CXF is responsible for managing the caches, storing them on the message context so that they get picked up on the next invocation, and shutting them down properly when the CXF endpoint comes down etc. WSS4J has no concept of a "context" that stores information for the next request, so it's up to the calling code to handle this.

In the test-code you provided, a cache will simply be created each time RequestData is initialized. After processing, data.getTimestampReplayCache().close() is never called meaning that the cache is not closed. That explains the proliferation of threads.

So in short, create the ReplayCache instance in the calling code once and set it on the RequestData for each request.

Colm.

On Mon, Jul 18, 2016 at 1:23 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 


Yes, as you can see from the attached images of last mail. 
In particular is a timestamp cache.


Il giorno 18 lug 2016, alle ore 12:41, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

> In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private
>final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
>of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

I'm a bit confused by this. Why is the diskstore shutting down after every invocation? Are you seeing a new cache file created for each invocation?

Colm.

On Tue, Jul 12, 2016 at 2:30 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 

An additional point: it seems not to be related to wildfly. The same code executed in a junit
produces the same results: 





Il giorno 11 lug 2016, alle ore 11:32, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

This sounds like a serious problem, although possibly EhCache related rather than WSS4J.

Firstly, WSS4J 2.0.7 uses EhCache 2.8.5 not 2.9.0 - could you verify that the same behaviour occurs with 2.8.5?

If you disable the diskstore by changing the EhCache configuration, does it solve the problem?

Colm.

On Fri, Jul 8, 2016 at 1:41 PM, massimiliano.masi <[hidden email]> wrote:
Hi All,

I updated from wss4j 1.6 to wss4j 2.0.7. I observe the following behaviour (using wildfly 9 and java 1.8.0_60).

With Nonce Replay Cache, Saml One Time Use Replacy Cache, Timestamp Replay Cache set to true, there is a linear thread proliferation: 1 call, 1 thread open and not closed.

With all those caches set to false, the threads are not proliferating and everything is fine.

Threads are watched as: watch -n 1 “date && cat /proc/<PID>/status | grep Threads”

In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

Thus the linear proliferation observed seems to be a new instance of the ScheduledThreadPoolExecutor.

Thus: 2000 web service requests, 2000 new threads spawned and not closed.

Any advice?

Thanks a lot,

        Massimiliano


--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com



--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/

Reply | Threaded
Open this post in threaded view
|

Re: nonce cache thread proliferation

Colm O hEigeartaigh-2
Which cache are you expecting to get invoked? The SAML cache only gets invoked if there is a "OneTimeUse" attribute on a SAML Assertion. The UsernameToken cache gets invoked only if there is a nonce in the UsernameToken. The Timestamp cache gets invoked only if there is both a Signature + Timestamp, where the Signature signs the Timestamp.

Colm.

On Tue, Aug 2, 2016 at 12:35 PM, massimiliano.masi <[hidden email]> wrote:
Hi,  

I think I am doing something wrong here (2.0.9). 

I created (to debug) my own instance of ReplayCache, in a static block: 

private static ReplayCache noncecache;

private static ReplayCache samlcache;

private static ReplayCache tscache;

static {

noncecache = new MicroCacheReplayCache();
samlcache = new MicroCacheReplayCache();
tscache = new MicroCacheReplayCache();
}


and then executed as: 

WSSecurityEngine secEngine = new WSSecurityEngine();
secEngine.setWssConfig(config);
RequestData rd = new RequestData();



rd.setNonceReplayCache(noncecache);
rd.setTimestampReplayCache(tscache);
rd.setSamlOneTimeUseReplayCache(samlcache);
rd.setCallbackHandler(cbHandler);
rd.setSigVerCrypto(fakeCrypto);
rd.setDecCrypto(fakeCrypto);
rd.setActor(null);


List<WSSecurityEngineResult> results = secEngine.processSecurityHeader(WSSecurityUtil.getSecurityHeader(envelope, null), 
rd);

But only the constructor is called. The contains() or the add() methods are never called. Am I missing something obvious?


Il giorno 02 ago 2016, alle ore 10:33, Colm O hEigeartaigh <[hidden email]> ha scritto:

Yes, it is available in 2.0.9. The "fix" is that to enable caching, you must create the ReplayCache instances yourself and set them on the RequestData Object.

Colm.

On Mon, Aug 1, 2016 at 2:44 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 

Yes, I am not importing CXF, I am using wss4j as standalone library.

Many thanks for the fix, I will try it ASAP. Is it available already in 2.0.9?




Il giorno 18 lug 2016, alle ore 18:05, Colm O hEigeartaigh <[hidden email]> ha scritto:

FYI I merged a fix to disable creating ReplayCaches internally. To use this functionality you need to create the cache yourself (or use CXF) + set the instance on RequestData:

https://issues.apache.org/jira/browse/WSS-584

Colm.

On Mon, Jul 18, 2016 at 3:22 PM, Colm O hEigeartaigh <[hidden email]> wrote:

Hi Massimiliano,

I think the key issue here is how are you using WSS4J? Most users use WSS4J with Apache CXF. The WS-Security layer in CXF is responsible for managing the caches, storing them on the message context so that they get picked up on the next invocation, and shutting them down properly when the CXF endpoint comes down etc. WSS4J has no concept of a "context" that stores information for the next request, so it's up to the calling code to handle this.

In the test-code you provided, a cache will simply be created each time RequestData is initialized. After processing, data.getTimestampReplayCache().close() is never called meaning that the cache is not closed. That explains the proliferation of threads.

So in short, create the ReplayCache instance in the calling code once and set it on the RequestData for each request.

Colm.

On Mon, Jul 18, 2016 at 1:23 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 


Yes, as you can see from the attached images of last mail. 
In particular is a timestamp cache.


Il giorno 18 lug 2016, alle ore 12:41, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

> In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private
>final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
>of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

I'm a bit confused by this. Why is the diskstore shutting down after every invocation? Are you seeing a new cache file created for each invocation?

Colm.

On Tue, Jul 12, 2016 at 2:30 PM, massimiliano.masi <[hidden email]> wrote:
Hi, 

An additional point: it seems not to be related to wildfly. The same code executed in a junit
produces the same results: 





Il giorno 11 lug 2016, alle ore 11:32, Colm O hEigeartaigh <[hidden email]> ha scritto:

Hi Massimiliano,

This sounds like a serious problem, although possibly EhCache related rather than WSS4J.

Firstly, WSS4J 2.0.7 uses EhCache 2.8.5 not 2.9.0 - could you verify that the same behaviour occurs with 2.8.5?

If you disable the diskstore by changing the EhCache configuration, does it solve the problem?

Colm.

On Fri, Jul 8, 2016 at 1:41 PM, massimiliano.masi <[hidden email]> wrote:
Hi All,

I updated from wss4j 1.6 to wss4j 2.0.7. I observe the following behaviour (using wildfly 9 and java 1.8.0_60).

With Nonce Replay Cache, Saml One Time Use Replacy Cache, Timestamp Replay Cache set to true, there is a linear thread proliferation: 1 call, 1 thread open and not closed.

With all those caches set to false, the threads are not proliferating and everything is fine.

Threads are watched as: watch -n 1 “date && cat /proc/<PID>/status | grep Threads”

In fact, ehcache is using a new ScheduledThreadPoolExecutor (version 2.9.0 net.sf.ehcache.store.disk.DiskStorageFactory, line 151) and it is assigned to a private final (non static) variable. Analyzing the JVM with VisualVM we observed that, when the shutdown()
of the ExecutorService was called, the active threads were parked, causing a new creation of another executor service for a new session.

Thus the linear proliferation observed seems to be a new instance of the ScheduledThreadPoolExecutor.

Thus: 2000 web service requests, 2000 new threads spawned and not closed.

Any advice?

Thanks a lot,

        Massimiliano


--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com



--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

--
Anger is a gift, http://www.mascanc.net/




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com