Setup: QuickFix/J acceptor listening on a port.

Test case: connect to the port on which the acceptor is listening via telnet and pump the data from /dev/zero (telnet HOST PORT < /dev/zero & )

What happens?
The server is DOS'ed into an eventual OutOfMemoryError

Why?
In MINA,
org.apache.mina.filter.codec.CumulativeProtocolDecoder.decode() accumulates the data in a session buffer until it is processed.
org.apache.mina.filter.codec.demux.DemuxingProtocolCodecFactory.ProtocolDecoderImpl.doDecode() is called to process the data.
It calls org.apache.mina.filter.codec.demux.MessageDecoder.decodable() method to identify which protocol to use.

In QuickFix/J,
quickfix.mina.message.FIXMessageDecoder.decodable() keeps iterating over and over again over ever increasing ByteBuffer, and keeps returning MessageDecoderResult.NEED_DATA.

What I think should be done to fix it:
quickfix.mina.message.FIXMessageDecoder.decodable() should instead return MessageDecoderResult.NOT_OK after the length of the unprocessed data grows beyond certain reasonable length comparable with the HEADER_PATTERN's potential max length. This would cause MINA to throw ProtocolDecoderException and the connection to be shut down.