I've downloaded and compiled openssl-1.1.0.

I can encrypt and decrypt using the same exe of openssl (as is here)

[email protected]:~/openssl-1.1.0$ LD_LIBRARY_PATH=. ./apps/openssl aes-256-cbc -a -salt -in file.txt -out file.txt.enc
enter aes-256-cbc encryption password: 123
Verifying - enter aes-256-cbc encryption password:
[email protected]:~/openssl-1.1.0$ LD_LIBRARY_PATH=. apps/openssl aes-256-cbc -a -d -in file.txt.enc -out file.txt.dec
enter aes-256-cbc decryption password: 123

This openssl uses: libcrypto.so.1.1, libssl.so.1.1

When I try to decrypt with the openssl installed on my ubuntu, which uses: /lib/x86_64-linux-gnu/libssl.so.1.0.0, /lib/x86_64-linux-gnu/libcrypto.so.1.0.0

I get an error:

[email protected]:~/openssl-1.1.0$ openssl aes-256-cbc -a -d -in file.txt.enc -out file.txt.dec2
enter aes-256-cbc decryption password: 123
bad decrypt
140456117421728:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:539:

What may cause this? Thanks

Solution 1

The default digest was changed from MD5 to SHA256 in Openssl 1.1

Try using -md md5

[email protected]:~$ echo "it-works!" > file.txt
[email protected]:~$ LD_LIBRARY_PATH=~/openssl-1.1.0/ openssl-1.1.0/apps/openssl aes-256-cbc -a -salt -in ~/file.txt -out ~/file.txt.enc -md md5
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
[email protected]:~$ LD_LIBRARY_PATH=~/openssl-1.0.1f/ openssl-1.0.1f/apps/openssl aes-256-cbc -a -in ~/file.txt.enc -d
enter aes-256-cbc decryption password:

The ugly details:

The entered password is not used as is by aes (or other encryption) but the command implicitly derives a key from it. The key derivation uses message digest that was changed in openssl 1.1 Use SHA256 not MD5 as default digest.

In case you want to keep it simple password, and not start messing with the keying martial (-K,-iv) just force the same digest with -md

Solution 2

I tested the AES encryption and decryption with version 1.1.0a (downloaded from openssl.org) and the version 1.0.2g-fips (from my ubuntu 16.04)

When using the -p option on with 2 different versions of openssl, the IV and key are different:

$ LD_LIBRARY_PATH=~/openssl-1.1.0a/ ~/openssl-1.1.0a/apps/openssl aes-256-cbc -a -p -salt -in file -out file.enc
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
iv =6AC7CE5C9AADC6C46C633BF5124DAFBF

$ openssl aes-256-cbc -a -d -p -in file.enc -out file.dec
enter aes-256-cbc decryption password:
iv =2DC04EF29AA57478EBE606DF87277EA6
bad decrypt
140557073118872:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:592:

I suspect a different derivation of key and IV based on the salt with the 2 versions.

If you want to get rid of this decryption error, you may remove the -salt option and use the options -K for the key and -iv in your openssl command.

Solution 3

This issue can also occur between OpenSSL 1.1 and LibreSSL. In this case, and in other cases where more secure message digests are available, you should avoid using -md md5 to encrypt new files since the MD5 algorithm has extensive vulnerabilities.

You should instead use -md sha256 or some other more secure message digest supported by all versions. -md md5 should only be used for decrypting old files, and they should ideally be re-encrypted using sha256. This is also mentioned in the OpenSSL FAQ:

A message digest is used to create the encrypt/decrypt key from a human-entered passphrase. In OpenSSL 1.1.0 we changed from MD5 to SHA-256. We did this as part of an overall change to move away from the now-insecure and broken MD5 algorithm. If you have old files, use the "-md md5" flag to decrypt them.

To check which message digests are supported by the different versions you have in play, run openssl help:

LibreSSL 2.2.7 (included with macOS 10.13 High Sierra):

$ openssl help

Message Digest commands (see the `dgst' command for more details)
gost-mac          md4               md5               md_gost94
ripemd160         sha               sha1              sha224
sha256            sha384            sha512            streebog256
streebog512       whirlpool

OpenSSL 1.1f:

$ openssl help

Message Digest commands (see the `dgst' command for more details)
blake2b512        blake2s256        gost              md4
md5               rmd160            sha1              sha224
sha256            sha384            sha512

Solution 4

There are various error strings that are thrown from openssl, depending on respective versions, and scenarios. Below is the checklist I use in case of openssl related issues:

  1. Ideally, openssl is able to encrypt/decrypt using same key (+ salt) & enc algo only.
  2. Ensure that openssl versions (used to encrypt/decrypt), are compatible. For eg. the hash used in openssl changed at version 1.1.0 from MD5 to SHA256. This produces a different key from the same password. Fix: add "-md md5" in 1.1.0 to decrypt data from lower versions, and add "-md sha256 in lower versions to decrypt data from 1.1.0

  3. Ensure that there is a single openssl version installed in your machine. In case there are multiple versions installed simultaneously (in my machine, these were installed :- 'LibreSSL 2.6.5' and 'openssl 1.1.1d'), make the sure that only the desired one appears in your PATH variable.