|
Marie, can we have a test case for this? It's important to know what exactly it means that some messages are missing from the file (which should never happen). Are exceptions silently swallowed? Are the contents mixed up? Also it's important to see that the order of the file contents is not guaranteed with async flows since threads can overtake each other.
Marie, see if you can get this fixed for the 1.4.3 release
I can't create a proper test case for this since it only occurs on very large amounts of data being processed concurrently. However, there are no exceptions being thrown, just one batch of messages overwriting others. The only indication of the messages being lost is in the number of messages being written to the output file which is less than that expected.
Marie, Dirk & I will write a test case and research the broader problem of file locking in general. Once we can rreliably reproduce the problem we'll provide at least a short-term fix.
Ah, sorry. I will retest to see whether the behaviour has been fixed with the changes implemented in
It is unlikely that the problem is fixed - I can easily think of a case where sending data to a file concurrently (via dispatch()) is mangled, especially in append mode. When there was only a single dispatcher for an endpoint, it would have been enough to make the doDispatch() synchronized; now there are multiple (pooled) and they are not coordinated. That is why i suggsted to limit their number to 1 as quick fix.
I guess in a way moving to Dirk (and me)
I have limited maxDispatchersActive to 1 in:
http://fisheye.codehaus.org/changelog/mule/?cs=10483 The default value (set in the constructor) and overridden setter can be removed once some kind of LockManager exists. This will of course not change the behaviour - it's just not possible to append to the same file with multiple writers without some kind of locking. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
In 1.4 all dispatcher actions are declared as unsynchronized for the dispatcher instance and always guaranteed to be performed by a single thread only, so only multiple concurrent requests to the same endpoint can conflict since they are done by multiple instances. The file transport is one of the modules that so far were not yet reviewed for the new transport architecture (see
MULE-1436).In the meantime check out AbstractConnector.get/setMaxDispatchersActive which was made for exactly the purpose of limiting the number of concurrent dispatchers. The "real" solution is to have a mutex (lock manager) by endpoint (directory path/file).
So as short-term fix configuring or hard-coding the file connector to return a constant 1 for getMaxDispatchersActive should have exactly the same effect as your mutex.