This is due to the current design of the device. The DataPower SOA Appliance is optimized to not waste resources on bad transactions. If the full attachment can be buffered before DataPower notices the error (i.e. the attachment exceeding the size limit imposed by the attachment-byte-count setting) an error message will be logged and a SOAPFault will be sent to the client. However, if DataPower is still reading the attachment when the configured limit is reached, it will close/reset the connection rather than wasting time, CPU and memory reading it all in.
If the 'Attachment Byte Count Limit' is set to 100kb, a messages of ~105kb will generate a SOAPFault whereas a message of ~200kb will cause a connection reset.
In either case the DataPower appliance will log an error message with error code below:
"0x00630004 - An attachment exceeded the size limit imposed by attachment-byte-count."