Actually in opentext there is a BDC for MIRO , when user choses to post invoice , this BDC is called after which depending on the success of the BDC , the open text tables are updated with various things like the invoice document number....
what really happened in this scenario of suspected duplicate is that the acounting document for that PO is present already when user tried to post the PO invoice....I checked the accounting document its creation date + time is same as in the workflow log of the first time user executed the step post PO invoice... which gives us the suggestion that the first time user clicked on post po invoice it was successful, however it did not updated the open text tables...
I tried the same case , by successfully creating a invoice document after the BDC and skipping the updation of opentext tables, however i still did not get the suspected duplicate exception when i again tried to post po invoice the second time , it just posted it with a different number and updated it on the second time....
so wanted to know if someone actually knows how this suspected duplicate exception might occur or just have an other opinion what could have occured ... i hope it was clear this time... 
BDC logs are saved in a table , however open text always keep the latest log and deletes the previous log , the user executed 3-4 times so the BDC log for the first time is already deleted which would have been helpful.....
one more thing you said that it depends on the vendor master customizing settings , would this setting be for all vendors or specific vendors ? , if only specific vendors are checked then may be i tried with a vendor for which this was unchecked ...
thanks for your help..