[OTR-dev] Interop problems with libotr - DSA signature

[Werner Dittmann]

Ian Goldberg schrieb:
> On Sat, Jul 18, 2009 at 05:00:18PM +0200, Werner Dittmann wrote:
>> Dear all,
>>
>> for the SIP Communicator project one of the GSOC'09 students (George)
>> develops a Java version of the OTR library (otr4j). During this
>> development he found an interoperability  problem with the libotr
>> DSA signature. George contacted me (I'm one of his mentors) to help
>> him to clarify this issue.
>>
>> George digged into the problem and tracked it down: otr4j uses the DSA
>> implementation of the BouncyCastle (BC) library which in turn
>> implements DSA according to FIPS-186-3. The libotr library uses the
>> DSA signature implementation of libgcrypt that is a generic DSA
>> implementation.
>>
>> Thus libotr uses a 160 bit q to sign 256 bit data. This is possible
>> because libgcrypt implements a generic DSA signature, thus the caller
>> of libgcrypt must truncate the data if necessary. I don't know if
>> using a 160 bit q to sign 256 bit of data causes any security problems
>> or if it is allowed according the the DSA algorithm.
>>
>> Therefore libotr uses DSA in a way that is not in accordance to
>> FIPS-186-1, 186-2, and 186-3. The two older standard versions (1 and
>> 2) allow 160 bit data only, the new standard allows different length
>> but in that case the data has to be truncated to the bit length of
>> q. Because BC implements this truncation an otr4j DSA signature does
>> not verify with the libotr DSA verification and vice versa. To
>> overcome this problem George uses a modified BC DSA implementation.
>>
>> However the main problem of the incorrect usage of DSA signature
>> remains.
>>
>> Any ideas or comments?
> 
> In fact, we came across the same problem while implementing java-otr
> just a couple of weeks ago.  It's definitely a problem in the spec, and
> we'll fix it in the next version of the wire protocol.  That of course
> doesn't help you interoperate with current implementations, which
> conform to the current spec.
> 
> Note that if you can get the standard DSA implementation you're using to
> not hash its input, 

The current BC DSA signature implemenation just takes the input and does not
hash it (only the JCE variant performs a hash before it signs, the
BC lightweight library provides a "raw" DSA implemenation acc. to FIPS-186-3).
Thus it is possible to hand over a (input mod 256) to the DSA (lightweight).
I'll ask George to check this and to modify the code accordingly.

you should be able to pass it ((256-bit value) mod q),
> (rather than truncating the 256-bit value) and all should be well.

Well, the wonders of mod mathematics remain a mystery to me ;-)

Regards,
Werner

> Otherwise, a slightly modified DSA implementation is probably the best
> you'll be able to do.
> 
> Totally my bad, sorry.  :-(
> 
>    - Ian
> _______________________________________________
> OTR-dev mailing list
> OTR-dev at lists.cypherpunks.ca
> http://lists.cypherpunks.ca/mailman/listinfo/otr-dev
>