[OTR-dev] Interop problems with libotr - DSA signature

[Ian Goldberg]

On Sat, Jul 18, 2009 at 05:00:18PM +0200, Werner Dittmann wrote:
> Dear all,
> 
> for the SIP Communicator project one of the GSOC'09 students (George)
> develops a Java version of the OTR library (otr4j). During this
> development he found an interoperability  problem with the libotr
> DSA signature. George contacted me (I'm one of his mentors) to help
> him to clarify this issue.
> 
> George digged into the problem and tracked it down: otr4j uses the DSA
> implementation of the BouncyCastle (BC) library which in turn
> implements DSA according to FIPS-186-3. The libotr library uses the
> DSA signature implementation of libgcrypt that is a generic DSA
> implementation.
> 
> Thus libotr uses a 160 bit q to sign 256 bit data. This is possible
> because libgcrypt implements a generic DSA signature, thus the caller
> of libgcrypt must truncate the data if necessary. I don't know if
> using a 160 bit q to sign 256 bit of data causes any security problems
> or if it is allowed according the the DSA algorithm.
> 
> Therefore libotr uses DSA in a way that is not in accordance to
> FIPS-186-1, 186-2, and 186-3. The two older standard versions (1 and
> 2) allow 160 bit data only, the new standard allows different length
> but in that case the data has to be truncated to the bit length of
> q. Because BC implements this truncation an otr4j DSA signature does
> not verify with the libotr DSA verification and vice versa. To
> overcome this problem George uses a modified BC DSA implementation.
> 
> However the main problem of the incorrect usage of DSA signature
> remains.
> 
> Any ideas or comments?

In fact, we came across the same problem while implementing java-otr
just a couple of weeks ago.  It's definitely a problem in the spec, and
we'll fix it in the next version of the wire protocol.  That of course
doesn't help you interoperate with current implementations, which
conform to the current spec.

Note that if you can get the standard DSA implementation you're using to
not hash its input, you should be able to pass it ((256-bit value) mod q),
(rather than truncating the 256-bit value) and all should be well.
Otherwise, a slightly modified DSA implementation is probably the best
you'll be able to do.

Totally my bad, sorry.  :-(

   - Ian