Uploaded image for project: 'QuickFIX/J'
  1. QuickFIX/J
  2. QFJ-968

SessionState messageQueue causing out of memory exception

    Details

    • Type: Bug
    • Status: Open
    • Priority: Default
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0
    • Fix Version/s: None
    • Component/s: Engine
    • Labels:
      None
    • Environment:
      Java 8

      Description

      The issues is based on my observation and investigation. Please correct me if I misunderstood it, thanks!

      Scenario:

      • We use SocketInitiator to connect to vendor server and consume data.
      • ResendRequestChunkSize is set to 5000.
        Now assume our process is down for several hours, and 10 million messages are queued up on the vendor side. Once we get connected again, we will send about 2000 resend requests with batch size of 5000 to catch up to the latest message. In between of every two batches, vendor sends latest messages to us, which have much higher sequence number, and are added to the messageQueue in SessionState. They will not be processed until all the 10 million messages have been received by us, which means the memory will keep expanding during our "catching up" time. And this will eventually cause out of memory issue.

      Proposed Solution:

      Not limiting messageQueue size could be risky. My understanding is that messageQueue should only store temporary out of order messages, and if there are say 100,000 messages in it, there must be something wrong. Therefore options should be provided to discard newer messages instead of keep eating up the memory.

      My proposal is to:

      1. Change LinkedHashMap to TreeMap. Even it may slow things down a little bit, O(lg n) should work fine given the size of messageQueue is normally very small. Even when it grows up, sequence number ordered entries should be beneficial.
      2. Update enqueue method to limit the size of messageQueue. If the queue is full, then add the new message and remove the entry with highest sequence number.
      3. Update dequeueMessagesUpTo method.

      If you think this could work, I would be happy to submit a PR, thanks!

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Energeek Ryan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: