Another user also blogged about this issue at: http://www.natesimpson.com/blog/archives/2011/06/15/quickfixj-and-odd-memory-leaks/
While making sure to never throw an exception solved this issue for the referenced case, it also happens for regular resends caused by a server-side issue or temporary network issues.
Just some further results of our investigation: This seems to be related to nextSequenceReset calling SessionState.setNextTargetMsgSeqNum, which causes the messages skipped by the SequenceReset to remain in the SessionState.messageQueue forever.
Is this the same problem as described in QFJ-271 or QFJ-409?
No, we are not seeing any StackOverflowErrors. It's just the SessionState.messageQueue that fills up which messages that will never be processed or garbage-collected.
OK, I assume that QFJ-678 describes the messageQueue in question? Of course, having a bounded queue does not rectify the problem that the queue is not cleared after the ResendRequest has been satisfied.
The messageQueue described in QFJ-678 is indeed a different one. The messageQueue I refer to is not a LinkedBlockingQueue, but the LinkedHashMap referenced in quickfix.SessionState:
private final Map<Integer, Message> messageQueue = new LinkedHashMap<Integer, Message>();
At the moment I am no longer sure whether the SequenceReset handling implemented by QuickFIX/J is actually correct for GapFill mode. When a SequenceReset w/ GapFillFlag=Y and NewSeqNo is received, QuickFix/J will just set a new target msg sequence number (via setNextTargetMsgSeqNum), but completely ignores any messages it may have received with a lower sequence number. I'm not exactly sure whether those should be discarded or processed, but something needs to be done about them. 
Committed as http://sourceforge.net/p/quickfixj/code/1129/
In Session.nextSequenceReset(): make sure that all messages lower than the new seqnum are deleted from the message queue.