Tested on Mac & Ubuntu

Two things:

1. what happens if you call stop(true)
2. could you create a stack dump when it is hanging?
   Thanks

Hi Christoph John,

1. Same as stop(false), it will hangs about 60s (with no logging) before exit

2. Both stop(false) & stop(true) thread dump is attach during the hangs period

Hmm, how do you exit the program? There is no trace of any QFJ-related code in the thread dump? For reference, could you create a thread dump while the program is running?

Sorry that I provided wrong log that used my customized quickfixJ version that try to shun down all the executor service when I can initiator.stop()

Now, fall back to 1.6.0 original version, problem still exist

Still can't see nothing QFJ-related in the dump, not even user code. Only Java classes. Could you please provide a dump while the process is running?

I am using "jstack -l [pid]" to get the thread dump, is that what you want?

added thread dump of before & after initiator.stop(true)

Yes, jstack is correct.
In your last dump I can still see the QFJ Timer thread being active. I don't know why it wasn't in the first dumps, though.

If your program does exit after 60 seconds it seems that it waits for the thread in the ScheduledThreadPoolExecutor to die. But actually the QFJ Timer thread is a daemon thread and should not keep the JVM from shutting down.
How do you exit your program?

Could you try with a newer / older / just another JVM version? I have not encountered this problem yet.

we are working or Java 1.8

$ java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

is this information helpful?

How do you exit your program?

Could you try with a newer / older / just another JVM version?

does it matter on what jvm version I use to build the quickfixj jar? I am using Java 1.8 too build quickfixj too

I don't think that it matters. IMHO it could be dependent on the JVM that is used to run the program. It is only a guess since I never saw this error. But it could be also dependent on other things in your code/environment/etc. But I assume it is a simple program that is run on the command line and not something that is running in Spring, an app server, an OSGi environment or the like??!

We see the same thing after we upgraded to QuickFix 1.6.1.
The call to initiator.stop hangs. Meanwhile , we see heartbeats are still sent out from the QuickfixJ engine.
I am attaching a thread dump which I hope helps.
The java version -
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

I was able to resolve the problem by adding this statement to IoSessionInitiator.stop():

synchronized void stop() {
if (reconnectFuture != null)

// Shutdown executor created by ioConnector (indirectly).
reconnectTask.ioConnector.dispose();
// ^^^
}

This disposes the executor created by the default constructor of NioSocketConnector (mina) instantiated in the IoConnector constructor. That executor (ExecutorService.newCachedThreadPool) has a 60s keepAliveTime. During shutdown that thread is blocked waiting until the next interval to check for more tasks.

Also spotted this issue both in version 1.6.1 and 1.7.0-SNAPSHOT

I was able to workaround this by avoiding the call to SocketInitiator.Stop()
I am logging out all the sessions manually and setting the initiator to null so it can be garbage collected

This extends to SocketAcceptors as well

 		if (socketInitiator != null) {
-			socketInitiator.stop();
+			Iterator<SessionID> sessionIds = socketInitiator.getSessions().iterator();
+			while (sessionIds.hasNext()) {
+				SessionID sessionId = sessionIds.next();
+				Session.lookupSession(sessionId).logout("user requested");
+			}
+			while (socketInitiator.isLoggedOn()) {
+				try {
+					Thread.sleep(500);
+				} catch (InterruptedException e) {
+					//Obfuscated
+				}
+			}
+			acceptor = null;
 		}
-		//Obfuscated
+		//Obfuscated
+			//Obfuscated
+		}
 		//ObfuscatedApplication = null;

Hi, what about Steve Nardone 's solution a few comments above? Looks cleaner to me. Although I have not tested it yet.

I haven't tried that out yet. I am pulling the QuickFixJ 1.6.1 dependency straight out of the maven repository and not building my own .
I will give it a shot and see if it works. Would avoid an ugly workaround.

Hi Steve Nardone,
I have integrated your suggested solution into the 1.6.x as well as the 1.7.0 branch.

Rohit, Martin Vrábel: could you test if that works for you?

Thanks,
Chris.

We have encountered this issue, too, in 1.6.1, but only on machines where the "is_production_machine" flag is set to true. We have not been able to reproduce the problem outside of production. The production machine is a 16-core RedHat distro based on 2.6. Our test machines tend to be 3.x based on other distros, and no more than 8 cores, generally 2 or 4. Not sure if this is relevant in the reproduction or not.

We built and deployed the latest 1.6.2-SNAPSHOT to our (Marketcetera) repo, which has this fix included, but this did not solve the problem.

The problem seems to occur when you're trying to shutdown the initiator while it's busy receiving messages. I've uploaded a relevant jstack output that shows the problem in 1.6.2-SNAPSHOT.

Hi Colin, thanks for the stack trace. It helps a lot. It seems that this is a similar issue as described in this comment: http://www.quickfixj.org/jira/browse/QFJ-854?focusedCommentId=12737&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12737
It is tried to access the EventHandlingStrategy while it is still locked. I think we can access the strategy outside of the lock and put a END_OF_STREAM message into it. That should resolve the "deadlock".

Edit: I will try to integrate this into 1.6.2-SNAPSHOT in the next days and let you know.

It looks like we were able to work around the problem by manually disconnecting each session in the initiator before stopping it.

I have made the promised changes. Could you please check if it resolves your problem?
Thanks, Chris.

The problem wasn't 100% reproducible. With that caveat, we were unable to reproduce the problem at all with the changes.

Fair enough. Please feel free to report if problem appears again.