[Bug 233907] Memory leakage in opencrypto for gcm algorithm

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233907

Bug ID: 233907
Summary: Memory leakage in opencrypto for gcm algorithm
Product: Base System
Version: 11.2-RELEASE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: kern
Assignee: ***@FreeBSD.org
Reporter: ***@gmail.com

Opencrypto subsystem has memory leakage when gcm algorithm is used. To make
sure the bug exist it's enough to monitor memory usage through "vmstat -m |
grep crypto" while opening and freeing new sessions for gcm (the auth part of
gcm: CRYPTO_AES_256_NIST_GMAC).

Memory is allocated at swcr_newsession function:

929 case CRYPTO_AES_256_NIST_GMAC:
930 axf = &auth_hash_nist_gmac_aes_256;
931 auth4common:
932 len = cri->cri_klen / 8;
933 if (len != 16 && len != 24 && len != 32) {
934 swcr_freesession_locked(dev, i);
935 rw_runlock(&swcr_sessions_lock);
936 return EINVAL;
937 }
938
939 (*swd)->sw_ictx = malloc(axf->ctxsize,
M_CRYPTO_DATA,
940 M_NOWAIT);

and not released in swcr_freesession_locked function. I think it's enough to
make something like this in swcr_freesession_locked:

1055 case CRYPTO_MD5:
1056 case CRYPTO_SHA1:
case CRYPTO_AES_128_NIST_GMAC:
case CRYPTO_AES_192_NIST_GMAC:
case CRYPTO_AES_256_NIST_GMAC:

1057 axf = swd->sw_axf;
1058
1059 if (swd->sw_ictx)
1060 free(swd->sw_ictx, M_CRYPTO_DATA);
1061 break;