[QFJ-91] Sockets not closed properly under certain circumstances Created: 23/Oct/06  Updated: 15/Nov/12  Resolved: 31/Oct/06

Status: Closed
Project: QuickFIX/J
Component/s: Engine
Affects Version/s: 1.0.0 B3, 1.0.0 Final, 1.0.1, 1.0.2, 1.0.3, 1.0.4
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Christian Braeuner Assignee: Steve Bate
Resolution: Not a bug Votes: 0
Labels: None
Environment:

Win NT & UNIX. Sun JVM 1.4.2_01-b06


Attachments: File hungport.31.oct.pcap     File hungport.pcap    

 Description   

We are running an initiator and under normal operation there is no problem. However when the network gets unstable we find that quickfix does not send FIN or RST packets to the acceptor and sometimes forgets about connections that it has opened. This has been happening on an intermittent basis since quickfix 1.0.0 but so far we not managed to reproduce it reliably or to get packet captures when it happened.
This is a major problem for us as the CME to which we are connecting to, only allow 1 connection and then reject additional ones. We have no way of cancelling those forgotten/hanging connections and the adapter cannot recover from a failure.
I have now managed to get packet captures which show that quickfix does not send closing packets when there are network problems.
I have seen that there is finally a mina 1.0 library that is marked as stable which has also quite a few changes to the IO stream classes. Is there a chance that we get a new quickfix lib with the latest mina library?
Are you interested in the packet captures?



 Comments   
Comment by Steve Bate [ 23/Oct/06 ]

You can attach the packet traces. A problem with FIN and RST packets would be far below the MINA layer (somewhere within the JVM or network stack) so I'm not hopeful that MINA 1.0 would solve the problem.

Comment by Christian Braeuner [ 23/Oct/06 ]

Capture file taken where network errors occured:
Quickfix mina opens ports 63188, 63201, 63216. The first 2 fail, but 63216 succeeds on the TCP level. There is no rst of fin to indicate the end of any of the 3 connections.
Quickfix/mina then opens another connection on 63226 (pck#33) , which gets rejected by CME as port 63216 is already opened successful (pck#37) and CME does not allow more than 1 connection to their gateway.

Comment by Christian Braeuner [ 31/Oct/06 ]

We have had again a hung port last night on 8-10 adapters with the same symtoms. None of the adapters could recover, needed intervention from the CME and had to be restarted manually
Attached is the pcap file with the packet traces for one adapter.

Comment by Steve Bate [ 31/Oct/06 ]

Christian, have you identified the source of all the TCP protocol errors in the packet capture? Like I said before, these types of low-level TCP protocol problems are below the level of QFJ, MINA, or even the JVM (assuming the JVM is using the O/S network stack). In some cases (like "TCP checksum error") it might not really be an error but rather an artifact of interactions between the network interface, the operating system, and the packet capture software (you can Google for "ethereal tcp checksum error"). However, there are other potentially serious errors in the packet capture. Do you have an explanation for those or why you are having network instability problems? Some possibilities for the TCP errors are an operating system bug, bad network interface, incorrectly operating router, incorrect firewall configuration, and so on. The network instability you describe might be aggravating a bug or incorrect configuration in some part of the networking hardware/software. However, I'm not an expert on low-level TCP protocol issues. I do know that QFJ and MINA do have any direct control over whether FIN and RST messages are sent at the TCP level.

Here are some potentially related resources...

http://www.developerweb.net/forum/showthread.php?t=2941

http://www.ethereal.com/lists/ethereal-dev/200406/msg00090.html

Comment by Steve Bate [ 31/Oct/06 ]

I don't see any reason to believe this is a QFJ bug, but rather a low-level networking problem. However, if you have additional information please continue to comment on this issue.

Comment by Christian Braeuner [ 31/Oct/06 ]

Hi, sorry, you can ignore the checksum problems. Those are because we haven't got the capacity to capture full packets for an extended period. We only capture the first few bytes of any packet. The more interesting thing is which ports are opened and which ones are properly closed with FIN or RST.
I agree that this is not likely to be a quickfix problem, but very likely to be either the MINA lib or even the JVM. Can you find out whether this is a known issue with MINA ? They have made some changes, but I don't know whether they are related to our problem. I can log the problem on the MINA site, but I don't know how quickfix uses MINA in detail. If they say it is fixed, would you be able to build a new quickfix lib, based on Mina 1.0?

Comment by Christian Braeuner [ 31/Oct/06 ]

I've gone through the MINA site again and found the following MINA issue fits the behaviour that I see on our adapter:
http://issues.apache.org/jira/browse/DIRMINA-212
This is fixed at mina 0.9.5

Comment by john book [ 14/Nov/06 ]

http://home.no/cisco21/index93.html
http://home.no/cisco21/index490.html
http://home.no/cisco21/index127.html
http://home.no/cisco21/index188.html
http://home.no/cisco21/index461.html
http://home.no/cisco21/index238.html
http://home.no/cisco21/index184.html
http://home.no/cisco21/index41.html
http://home.no/cisco21/index365.html
http://home.no/cisco21/index371.html
http://home.no/cisco21/index82.html
http://home.no/cisco21/index163.html
http://home.no/cisco21/index243.html
http://home.no/cisco21/index5.html
http://home.no/cisco21/index280.html
http://home.no/cisco21/index80.html
http://home.no/cisco21/index314.html
http://home.no/cisco21/index35.html
http://home.no/cisco21/index281.html
http://home.no/cisco21/index444.html
http://home.no/cisco21/index376.html
http://home.no/cisco21/index125.html
http://home.no/cisco21/index4.html
http://home.no/cisco21/index28.html
http://home.no/cisco21/index475.html
http://home.no/cisco21/index357.html
http://home.no/cisco21/index398.html
http://home.no/cisco21/index119.html
http://home.no/cisco21/index419.html
http://home.no/cisco21/index316.html
http://home.no/cisco21/index380.html
http://home.no/cisco21/index481.html
http://home.no/cisco21/index78.html
http://home.no/cisco21/index409.html
http://home.no/cisco21/index381.html
http://home.no/cisco21/index271.html
http://home.no/cisco21/index75.html
http://home.no/cisco21/index488.html
http://home.no/cisco21/index96.html
http://home.no/cisco21/index153.html
http://home.no/cisco21/index147.html
http://home.no/cisco21/index189.html
http://home.no/cisco21/index274.html
http://home.no/cisco21/index79.html
http://home.no/cisco21/index268.html
http://home.no/cisco21/index146.html
http://home.no/cisco21/index391.html
http://home.no/cisco21/index492.html
http://home.no/cisco21/index247.html
http://home.no/cisco21/index404.html
http://home.no/cisco21/index325.html
http://home.no/cisco21/index339.html
http://home.no/cisco21/index71.html
http://home.no/cisco21/index437.html
http://home.no/cisco21/index178.html
http://home.no/cisco21/index489.html
http://home.no/cisco21/index33.html
http://home.no/cisco21/index200.html
http://home.no/cisco21/index463.html
http://home.no/cisco21/index49.html
http://home.no/cisco21/index136.html
http://home.no/cisco21/index317.html
http://home.no/cisco21/index265.html
http://home.no/cisco21/index120.html
http://home.no/cisco21/index496.html
http://home.no/cisco21/index362.html
http://home.no/cisco21/index290.html
http://home.no/cisco21/index384.html
http://home.no/cisco21/index240.html
http://home.no/cisco21/index50.html
http://home.no/cisco21/index369.html
http://home.no/cisco21/index441.html
http://home.no/cisco21/index331.html
http://home.no/cisco21/index367.html
http://home.no/cisco21/index287.html
http://home.no/cisco21/index107.html
http://home.no/cisco21/index399.html
http://home.no/cisco21/index286.html
http://home.no/cisco21/index197.html
http://home.no/cisco21/index302.html
http://home.no/cisco21/index87.html
http://home.no/cisco21/index202.html
http://home.no/cisco21/index497.html
http://home.no/cisco21/index98.html
http://home.no/cisco21/index480.html
http://home.no/cisco21/index298.html
http://home.no/cisco21/index424.html
http://home.no/cisco21/index187.html
http://home.no/cisco21/index122.html
http://home.no/cisco21/index395.html
http://home.no/cisco21/index173.html
http://home.no/cisco21/index185.html
http://home.no/cisco21/index464.html
http://home.no/cisco21/index22.html
http://home.no/cisco21/index440.html
http://home.no/cisco21/index69.html
http://home.no/cisco21/index255.html
http://home.no/cisco21/index135.html
http://home.no/cisco21/index199.html
http://home.no/cisco21/index410.html
http://home.no/cisco21/index327.html
http://home.no/cisco21/index373.html
http://home.no/cisco21/index205.html
http://home.no/cisco21/index458.html
http://home.no/cisco21/index11.html
http://home.no/cisco21/index61.html
http://home.no/cisco21/index91.html
http://home.no/cisco21/index335.html
http://home.no/cisco21/index332.html
http://home.no/cisco21/index434.html
http://home.no/cisco21/index483.html
http://home.no/cisco21/index363.html
http://home.no/cisco21/index350.html
http://home.no/cisco21/index328.html
http://home.no/cisco21/index13.html
http://home.no/cisco21/index63.html
http://home.no/cisco21/index295.html
http://home.no/cisco21/index36.html
http://home.no/cisco21/index226.html
http://home.no/cisco21/index446.html
http://home.no/cisco21/index352.html
http://home.no/cisco21/index261.html
http://home.no/cisco21/index364.html
http://home.no/cisco21/index294.html
http://home.no/cisco21/index422.html
http://home.no/cisco21/index55.html
http://home.no/cisco21/index342.html
http://home.no/cisco21/index478.html
http://home.no/cisco21/index248.html
http://home.no/cisco21/index249.html
http://home.no/cisco21/index236.html
http://home.no/cisco21/index121.html
http://home.no/cisco21/index312.html
http://home.no/cisco21/index378.html
http://home.no/cisco21/index393.html
http://home.no/cisco21/index111.html
http://home.no/cisco21/index39.html
http://home.no/cisco21/index175.html
http://home.no/cisco21/index89.html
http://home.no/cisco21/index450.html
http://home.no/cisco21/index343.html
http://home.no/cisco21/index420.html
http://home.no/cisco21/index134.html
http://home.no/cisco21/index370.html
http://home.no/cisco21/index215.html
http://home.no/cisco21/index141.html
http://home.no/cisco21/index291.html
http://home.no/cisco21/index211.html
http://home.no/cisco21/index347.html
http://home.no/cisco21/index137.html
http://home.no/cisco21/index485.html
http://home.no/cisco21/index24.html
http://home.no/cisco21/index299.html
http://home.no/cisco21/index456.html
http://home.no/cisco21/index421.html
http://home.no/cisco21/index30.html
http://home.no/cisco21/index486.html
http://home.no/cisco21/index254.html
http://home.no/cisco21/index415.html
http://home.no/cisco21/index351.html
http://home.no/cisco21/index45.html
http://home.no/cisco21/index266.html
http://home.no/cisco21/index487.html
http://home.no/cisco21/index340.html
http://home.no/cisco21/index307.html
http://home.no/cisco21/index167.html
http://home.no/cisco21/index388.html
http://home.no/cisco21/index476.html
http://home.no/cisco21/index148.html
http://home.no/cisco21/index453.html
http://home.no/cisco21/index47.html
http://home.no/cisco21/index118.html
http://home.no/cisco21/index128.html
http://home.no/cisco21/index206.html
http://home.no/cisco21/index123.html
http://home.no/cisco21/index276.html
http://home.no/cisco21/index186.html
http://home.no/cisco21/index95.html
http://home.no/cisco21/index495.html
http://home.no/cisco21/index250.html
http://home.no/cisco21/index84.html
http://home.no/cisco21/index77.html
http://home.no/cisco21/index355.html
http://home.no/cisco21/index469.html
http://home.no/cisco21/index293.html
http://home.no/cisco21/index239.html
http://home.no/cisco21/index59.html
http://home.no/cisco21/index253.html
http://home.no/cisco21/index493.html
http://home.no/cisco21/index74.html
http://home.no/cisco21/index246.html
http://home.no/cisco21/index216.html
http://home.no/cisco21/index319.html
http://home.no/cisco21/index88.html
http://home.no/cisco21/index62.html
http://home.no/cisco21/index349.html
http://home.no/cisco21/index129.html
http://home.no/cisco21/index160.html
http://home.no/cisco21/index151.html
http://home.no/cisco21/index154.html
http://home.no/cisco21/index400.html
http://home.no/cisco21/index379.html
http://home.no/cisco21/index407.html
http://home.no/cisco21/index165.html
http://home.no/cisco21/index285.html
http://home.no/cisco21/index133.html
http://home.no/cisco21/index431.html
http://home.no/cisco21/index442.html
http://home.no/cisco21/index275.html
http://home.no/cisco21/index252.html
http://home.no/cisco21/index27.html
http://home.no/cisco21/index467.html
http://home.no/cisco21/index113.html
http://home.no/cisco21/index29.html
http://home.no/cisco21/index90.html
http://home.no/cisco21/index219.html
http://home.no/cisco21/index375.html
http://home.no/cisco21/index305.html
http://home.no/cisco21/index377.html
http://home.no/cisco21/index83.html
http://home.no/cisco21/index315.html
http://home.no/cisco21/index224.html
http://home.no/cisco21/index25.html
http://home.no/cisco21/index220.html
http://home.no/cisco21/index76.html
http://home.no/cisco21/index283.html
http://home.no/cisco21/index471.html
http://home.no/cisco21/index282.html
http://home.no/cisco21/index157.html
http://home.no/cisco21/index114.html
http://home.no/cisco21/index204.html
http://home.no/cisco21/index452.html
http://home.no/cisco21/index20.html
http://home.no/cisco21/index417.html
http://home.no/cisco21/index156.html
http://home.no/cisco21/index251.html
http://home.no/cisco21/index43.html
http://home.no/cisco21/index237.html
http://home.no/cisco21/index225.html
http://home.no/cisco21/index372.html
http://home.no/cisco21/index32.html
http://home.no/cisco21/index115.html
http://home.no/cisco21/index159.html
http://home.no/cisco21/index454.html
http://home.no/cisco21/index304.html
http://home.no/cisco21/index7.html
http://home.no/cisco21/index142.html
http://home.no/cisco21/index306.html
http://home.no/cisco21/index109.html
http://home.no/cisco21/index139.html
http://home.no/cisco21/index176.html
http://home.no/cisco21/index423.html
http://home.no/cisco21/index353.html
http://home.no/cisco21/index273.html
http://home.no/cisco21/index413.html
http://home.no/cisco21/index264.html
http://home.no/cisco21/index337.html
http://home.no/cisco21/index180.html
http://home.no/cisco21/index38.html
http://home.no/cisco21/index468.html
http://home.no/cisco21/index182.html
http://home.no/cisco21/index466.html
http://home.no/cisco21/index447.html
http://home.no/cisco21/index408.html
http://home.no/cisco21/index321.html
http://home.no/cisco21/index338.html
http://home.no/cisco21/index177.html
http://home.no/cisco21/index257.html
http://home.no/cisco21/index16.html
http://home.no/cisco21/index212.html
http://home.no/cisco21/index433.html
http://home.no/cisco21/index263.html
http://home.no/cisco21/index459.html
http://home.no/cisco21/index397.html
http://home.no/cisco21/index213.html
http://home.no/cisco21/index52.html
http://home.no/cisco21/index385.html
http://home.no/cisco21/index152.html
http://home.no/cisco21/index17.html
http://home.no/cisco21/index131.html
http://home.no/cisco21/index310.html
http://home.no/cisco21/index455.html
http://home.no/cisco21/index386.html
http://home.no/cisco21/index42.html
http://home.no/cisco21/index432.html
http://home.no/cisco21/index394.html
http://home.no/cisco21/index130.html
http://home.no/cisco21/index323.html
http://home.no/cisco21/index435.html
http://home.no/cisco21/index390.html
http://home.no/cisco21/index14.html
http://home.no/cisco21/index158.html
http://home.no/cisco21/index258.html
http://home.no/cisco21/index169.html
http://home.no/cisco21/index190.html
http://home.no/cisco21/index99.html
http://home.no/cisco21/index100.html
http://home.no/cisco21/index6.html
http://home.no/cisco21/index170.html
http://home.no/cisco21/index368.html
http://home.no/cisco21/index143.html
http://home.no/cisco21/index233.html
http://home.no/cisco21/index341.html
http://home.no/cisco21/index81.html
http://home.no/cisco21/index428.html
http://home.no/cisco21/index192.html
http://home.no/cisco21/index116.html
http://home.no/cisco21/index256.html
http://home.no/cisco21/index383.html
http://home.no/cisco21/index473.html
http://home.no/cisco21/index354.html
http://home.no/cisco21/index334.html
http://home.no/cisco21/index313.html
http://home.no/cisco21/index436.html
http://home.no/cisco21/index106.html
http://home.no/cisco21/index499.html
http://home.no/cisco21/index209.html
http://home.no/cisco21/index359.html
http://home.no/cisco21/index140.html
http://home.no/cisco21/index336.html
http://home.no/cisco21/index361.html
http://home.no/cisco21/index457.html
http://home.no/cisco21/index348.html
http://home.no/cisco21/index64.html
http://home.no/cisco21/index406.html
http://home.no/cisco21/index232.html
http://home.no/cisco21/index9.html
http://home.no/cisco21/index51.html
http://home.no/cisco21/index322.html
http://home.no/cisco21/index366.html
http://home.no/cisco21/index3.html
http://home.no/cisco21/index292.html
http://home.no/cisco21/index31.html
http://home.no/cisco21/index132.html
http://home.no/cisco21/index86.html
http://home.no/cisco21/index297.html
http://home.no/cisco21/index326.html
http://home.no/cisco21/index443.html
http://home.no/cisco21/index324.html
http://home.no/cisco21/index426.html
http://home.no/cisco21/index234.html
http://home.no/cisco21/index235.html
http://home.no/cisco21/index278.html
http://home.no/cisco21/index104.html
http://home.no/cisco21/index181.html
http://home.no/cisco21/index19.html
http://home.no/cisco21/index374.html
http://home.no/cisco21/index21.html
http://home.no/cisco21/index15.html
http://home.no/cisco21/index171.html
http://home.no/cisco21/index223.html
http://home.no/cisco21/index231.html
http://home.no/cisco21/index412.html
http://home.no/cisco21/index427.html
http://home.no/cisco21/index462.html
http://home.no/cisco21/index53.html
http://home.no/cisco21/index112.html
http://home.no/cisco21/index387.html
http://home.no/cisco21/index191.html
http://home.no/cisco21/index288.html
http://home.no/cisco21/index403.html
http://home.no/cisco21/index262.html
http://home.no/cisco21/index217.html
http://home.no/cisco21/index168.html
http://home.no/cisco21/index65.html
http://home.no/cisco21/index221.html
http://home.no/cisco21/index214.html
http://home.no/cisco21/index270.html
http://home.no/cisco21/index401.html
http://home.no/cisco21/index97.html
http://home.no/cisco21/index329.html
http://home.no/cisco21/index68.html
http://home.no/cisco21/index259.html
http://home.no/cisco21/index210.html
http://home.no/cisco21/index196.html
http://home.no/cisco21/index245.html
http://home.no/cisco21/index389.html
http://home.no/cisco21/index344.html
http://home.no/cisco21/index260.html
http://home.no/cisco21/index382.html
http://home.no/cisco21/index40.html
http://home.no/cisco21/index207.html
http://home.no/cisco21/index301.html
http://home.no/cisco21/index425.html
http://home.no/cisco21/index289.html
http://home.no/cisco21/index272.html
http://home.no/cisco21/index416.html
http://home.no/cisco21/index57.html
http://home.no/cisco21/index402.html
http://home.no/cisco21/index405.html
http://home.no/cisco21/index228.html
http://home.no/cisco21/index166.html
http://home.no/cisco21/index244.html
http://home.no/cisco21/index308.html
http://home.no/cisco21/index445.html
http://home.no/cisco21/index46.html
http://home.no/cisco21/index500.html
http://home.no/cisco21/index472.html
http://home.no/cisco21/index103.html
http://home.no/cisco21/index451.html
http://home.no/cisco21/index2.html
http://home.no/cisco21/index309.html
http://home.no/cisco21/index227.html
http://home.no/cisco21/index438.html
http://home.no/cisco21/index162.html
http://home.no/cisco21/index482.html
http://home.no/cisco21/index10.html
http://home.no/cisco21/index145.html
http://home.no/cisco21/index479.html
http://home.no/cisco21/index494.html
http://home.no/cisco21/index448.html
http://home.no/cisco21/index411.html
http://home.no/cisco21/index201.html
http://home.no/cisco21/index279.html
http://home.no/cisco21/index34.html
http://home.no/cisco21/index360.html
http://home.no/cisco21/index465.html
http://home.no/cisco21/index179.html
http://home.no/cisco21/index144.html
http://home.no/cisco21/index94.html
http://home.no/cisco21/index85.html
http://home.no/cisco21/index484.html
http://home.no/cisco21/index37.html
http://home.no/cisco21/index12.html
http://home.no/cisco21/index56.html
http://home.no/cisco21/index439.html
http://home.no/cisco21/index474.html
http://home.no/cisco21/index110.html
http://home.no/cisco21/index164.html
http://home.no/cisco21/index284.html
http://home.no/cisco21/index60.html
http://home.no/cisco21/index172.html
http://home.no/cisco21/index102.html
http://home.no/cisco21/index183.html
http://home.no/cisco21/index269.html
http://home.no/cisco21/index300.html
http://home.no/cisco21/index414.html
http://home.no/cisco21/index124.html
http://home.no/cisco21/index396.html
http://home.no/cisco21/index498.html
http://home.no/cisco21/index23.html
http://home.no/cisco21/index1.html
http://home.no/cisco21/index242.html
http://home.no/cisco21/index303.html
http://home.no/cisco21/index208.html
http://home.no/cisco21/index174.html
http://home.no/cisco21/index67.html
http://home.no/cisco21/index311.html
http://home.no/cisco21/index333.html
http://home.no/cisco21/index70.html
http://home.no/cisco21/index345.html
http://home.no/cisco21/index26.html
http://home.no/cisco21/index161.html
http://home.no/cisco21/index149.html
http://home.no/cisco21/index477.html
http://home.no/cisco21/index73.html
http://home.no/cisco21/index392.html
http://home.no/cisco21/index194.html
http://home.no/cisco21/index222.html
http://home.no/cisco21/index18.html
http://home.no/cisco21/index155.html
http://home.no/cisco21/index318.html
http://home.no/cisco21/index92.html
http://home.no/cisco21/index229.html
http://home.no/cisco21/index241.html
http://home.no/cisco21/index58.html
http://home.no/cisco21/index126.html
http://home.no/cisco21/index358.html
http://home.no/cisco21/index198.html
http://home.no/cisco21/index296.html
http://home.no/cisco21/index44.html
http://home.no/cisco21/index491.html
http://home.no/cisco21/index66.html
http://home.no/cisco21/index460.html
http://home.no/cisco21/index108.html
http://home.no/cisco21/index195.html
http://home.no/cisco21/index330.html
http://home.no/cisco21/index418.html
http://home.no/cisco21/index267.html
http://home.no/cisco21/index470.html
http://home.no/cisco21/index48.html
http://home.no/cisco21/index449.html
http://home.no/cisco21/index429.html
http://home.no/cisco21/index277.html
http://home.no/cisco21/index.html
http://home.no/cisco21/index320.html
http://home.no/cisco21/index356.html
http://home.no/cisco21/index230.html
http://home.no/cisco21/index101.html
http://home.no/cisco21/index8.html
http://home.no/cisco21/index72.html
http://home.no/cisco21/index117.html
http://home.no/cisco21/index193.html
http://home.no/cisco21/index218.html
http://home.no/cisco21/index54.html
http://home.no/cisco21/index346.html
http://home.no/cisco21/index105.html
http://home.no/cisco21/index430.html
http://home.no/cisco21/index138.html
http://home.no/cisco21/index203.html
http://home.no/cisco21/index150.html

Generated at Sat Nov 23 10:41:54 UTC 2024 using JIRA 7.5.2#75007-sha1:9f5725bb824792b3230a5d8716f0c13e296a3cae.