[OTR-dev] Interop problems with libotr - DSA signature

[Werner Dittmann]

Dear all,

for the SIP Communicator project one of the GSOC'09 students (George)
develops a Java version of the OTR library (otr4j). During this
development he found an interoperability  problem with the libotr
DSA signature. George contacted me (I'm one of his mentors) to help
him to clarify this issue.

George digged into the problem and tracked it down: otr4j uses the DSA
implementation of the BouncyCastle (BC) library which in turn
implements DSA according to FIPS-186-3. The libotr library uses the
DSA signature implementation of libgcrypt that is a generic DSA
implementation.

The interoperability problem is due to the following part of the OTR
protocol specification (reveal signature message, Signature message):

# Compute the 32-byte value MB to be the SHA256-HMAC of the
  following data, using the key m1:

gx (MPI)
gy (MPI)
pubB (PUBKEY)
keyidB (INT)

# Let XB be the following structure:

pubB (PUBKEY)
keyidB (INT)
sigB(MB) (SIG)
    This is the signature, using the private part of the key pubB,
    of the 32-byte MB (which does not need to be hashed again to
    produce the signature).

libotr implments this as follows (simlified for brevity)

- generates the SHA256-HMAC ot get the 32 byte MB
- hands over this 32 byte data to libgcrypt to sign this data
- the DSA keypair that libotr uses (and generated and stored) some
  time before has the following parameter attributes:
  p bit length (L) is 1024, q bit length (N) is 160.

Thus libotr uses a 160 bit q to sign 256 bit data. This is possible
because libgcrypt implements a generic DSA signature, thus the caller
of libgcrypt must truncate the data if necessary. I don't know if
using a 160 bit q to sign 256 bit of data causes any security problems
or if it is allowed according the the DSA algorithm.

Therefore libotr uses DSA in a way that is not in accordance to
FIPS-186-1, 186-2, and 186-3. The two older standard versions (1 and
2) allow 160 bit data only, the new standard allows different length
but in that case the data has to be truncated to the bit length of
q. Because BC implements this truncation an otr4j DSA signature does
not verify with the libotr DSA verification and vice versa. To
overcome this problem George uses a modified BC DSA implementation.

However the main problem of the incorrect usage of DSA signature
remains.

Any ideas or comments?

Regards,
Werner