nettle-bugs

nettle-bugs@lists.lysator.liu.se

1 participants
2200 discussions

Re: [PATCH] (revision 3.2) Added bcrypt() support.
by Stephen R. van den Berg 23 Jun '20

23 Jun '20

Changes since 3.1: Took out the tablesize optimisations. --- Makefile.in | 2 +- blowfish-bcrypt.c | 519 ++++++++++++++++++++++++++++++ blowfish.h => blowfish-internal.h | 51 +-- blowfish.c | 23 +- blowfish.h | 15 + nettle.texinfo | 67 ++++ 6 files changed, 621 insertions(+), 56 deletions(-) create mode 100644 blowfish-bcrypt.c copy blowfish.h => blowfish-internal.h (50%) diff --git a/Makefile.in b/Makefile.in index e5ccfc76..4ddbb32e 100644 --- a/Makefile.in +++ b/Makefile.in @@ -73,7 +73,7 @@ nettle_SOURCES = aes-decrypt-internal.c aes-decrypt.c \ aes256-set-encrypt-key.c aes256-set-decrypt-key.c \ aes256-meta.c \ arcfour.c arcfour-crypt.c \ - arctwo.c arctwo-meta.c blowfish.c \ + arctwo.c arctwo-meta.c blowfish.c blowfish-bcrypt.c \ base16-encode.c base16-decode.c base16-meta.c \ base64-encode.c base64-decode.c base64-meta.c \ base64url-encode.c base64url-decode.c base64url-meta.c \ diff --git a/blowfish-bcrypt.c b/blowfish-bcrypt.c new file mode 100644 index 00000000..0354dbe6 --- /dev/null +++ b/blowfish-bcrypt.c @@ -0,0 +1,519 @@ +/* blowfish-bcrypt.c + + The blowfish bcrypt implementation. + + Copyright (c) 2020 Stephen R. van den Berg + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +*/ + +#if HAVE_CONFIG_H +#include "config.h" +#endif + +#include <assert.h> +#include <string.h> +#include <stdlib.h> + +#include "blowfish.h" +#include "blowfish-internal.h" +#include "base64.h" + +#include "macros.h" + +#define CRYPTPLEN 7 +#define SALTLEN ((BLOWFISH_BCRYPT_BINSALT_SIZE*8+5) / 6) + +#define HASHOFFSET (CRYPTPLEN + SALTLEN) + +static const signed char radix64_decode_table[0x100] = { + /* White space is HT, VT, FF, CR, LF and SPC */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -2, -2, -2, -2, -2, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -2, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0, 1, + 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, -1, -1, -1, -3, -1, -1, + -1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, + 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, -1, -1, -1, -1, -1, + -1, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, + 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, +}; + +static const char radix64_encode_table[64] = + "./ABCDEFGHIJKLMNOPQRSTUVWXYZ" + "abcdefghijklmnopqrstuvwxyz" + "0123456789"; + +int +blowfish_bcrypt_verify(const char *key, + const char *hashed) +{ + char newhash[BLOWFISH_BCRYPT_HASH_SIZE]; + + return blowfish_bcrypt_hash(sizeof newhash, + newhash, key, hashed, -1, (void*)0) + && !strcmp(newhash, hashed); +} + +static char *encode_radix64(char *dst, size_t len, const uint8_t *src) +{ + struct base64_encode_ctx ctx; + base64_encode_init(&ctx); + ctx.alphabet = radix64_encode_table; + dst += base64_encode_update(&ctx, dst, len, src); + dst += base64_encode_final(&ctx, dst); + *--dst = '\0'; /* Strip the trailing = */ + return dst; +} + +/* + * Large parts of the code below are based on public domain sources. + * The comments and copyright notices have been preserved. + * Any code added or modified by me is licensed under the + * licenses listed above. -- Stephen R. van den Berg + */ + +/* + * This code comes from John the Ripper password cracker, with reentrant + * and crypt(3) interfaces added, but optimizations specific to password + * cracking removed. + * + * Written by Solar Designer <solar at openwall.com> in 1998-2015. + * No copyright is claimed, and the software is hereby placed in the public + * domain. In case this attempt to disclaim copyright and place the software + * in the public domain is deemed null and void, then the software is + * Copyright (c) 1998-2015 Solar Designer and it is hereby released to the + * general public under the following terms: + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted. + * + * There's ABSOLUTELY NO WARRANTY, express or implied. + * + * It is my intent that you should be able to use this on your system, + * as part of a software package, or anywhere else to improve security, + * ensure compatibility, or for any other purpose. I would appreciate + * it if you give credit where it is due and keep your modifications in + * the public domain as well, but I don't require that in order to let + * you place this code and any modifications you make under a license + * of your choice. + * + * This implementation is fully compatible with OpenBSD's bcrypt.c for prefix + * "$2b$", originally by Niels Provos <provos at citi.umich.edu>, and it uses + * some of his ideas. The password hashing algorithm was designed by David + * Mazieres <dm at lcs.mit.edu>. For information on the level of + * compatibility for bcrypt hash prefixes other than "$2b$", please refer to + * the comments in set_key() below and to the included crypt(3) man page. + */ + +typedef uint32_t bf_key[_BLOWFISH_ROUNDS + 2]; + +/* + * Magic IV for 64 Blowfish encryptions that we do at the end. + * The string is "OrpheanBeholderScryDoubt" on big-endian. + */ +static uint32_t magic_w[6] = { + 0x4F727068, 0x65616E42, 0x65686F6C, + 0x64657253, 0x63727944, 0x6F756274 +}; + +static void swap32(uint32_t *x, int count) +{ +#if !WORDS_BIGENDIAN + do { + uint32_t tmp = *x; + tmp = (tmp << 16) | (tmp >> 16); + *x++ = ((tmp & 0x00FF00FF) << 8) | ((tmp >> 8) & 0x00FF00FF); + } while (--count); +#endif +} + +static void set_xkey(const char *key, bf_key expanded, bf_key initial, + unsigned bug, uint32_t safety) +{ + const char *ptr = key; + unsigned i, j; + uint32_t sign, diff, tmp[2]; + +/* + * There was a sign extension bug in older revisions of this function. While + * we would have liked to simply fix the bug and move on, we have to provide + * a backwards compatibility feature (essentially the bug) for some systems and + * a safety measure for some others. The latter is needed because for certain + * multiple inputs to the buggy algorithm there exist easily found inputs to + * the correct algorithm that produce the same hash. Thus, we optionally + * deviate from the correct algorithm just enough to avoid such collisions. + * While the bug itself affected the majority of passwords containing + * characters with the 8th bit set (although only a percentage of those in a + * collision-producing way), the anti-collision safety measure affects + * only a subset of passwords containing the '\xff' character (not even all of + * those passwords, just some of them). This character is not found in valid + * UTF-8 sequences and is rarely used in popular 8-bit character encodings. + * Thus, the safety measure is unlikely to cause much annoyance, and is a + * reasonable tradeoff to use when authenticating against existing hashes that + * are not reliably known to have been computed with the correct algorithm. + * + * We use an approach that tries to minimize side-channel leaks of password + * information - that is, we mostly use fixed-cost bitwise operations instead + * of branches or table lookups. (One conditional branch based on password + * length remains. It is not part of the bug aftermath, though, and is + * difficult and possibly unreasonable to avoid given the use of C strings by + * the caller, which results in similar timing leaks anyway.) + * + * For actual implementation, we set an array index in the variable "bug" + * (0 means no bug, 1 means sign extension bug emulation) and a flag in the + * variable "safety" (bit 16 is set when the safety measure is requested). + * Valid combinations of settings are: + * + * Prefix "$2a$": bug = 0, safety = 0x10000 + * Prefix "$2b$": bug = 0, safety = 0 + * Prefix "$2x$": bug = 1, safety = 0 + * Prefix "$2y$": bug = 0, safety = 0 + */ + + sign = diff = 0; + + for (i = 0; i < _BLOWFISH_ROUNDS + 2; i++) { + tmp[0] = tmp[1] = 0; + for (j = 0; j < 4; j++) { + tmp[0] <<= 8; + tmp[0] |= (unsigned char)*ptr; /* correct */ + tmp[1] <<= 8; + tmp[1] |= (signed char)*ptr; /* bug */ +/* + * Sign extension in the first char has no effect - nothing to overwrite yet, + * and those extra 24 bits will be fully shifted out of the 32-bit word. For + * chars 2, 3, 4 in each four-char block, we set bit 7 of "sign" if sign + * extension in tmp[1] occurs. Once this flag is set, it remains set. + */ + if (j) + sign |= tmp[1] & 0x80; + if (!*ptr) + ptr = key; + else + ptr++; + } + diff |= tmp[0] ^ tmp[1]; /* Non-zero on any differences */ + + expanded[i] = tmp[bug]; + initial[i] = _nettle_blowfish_initial_ctx.p[i] ^ tmp[bug]; + } + +/* + * At this point, "diff" is zero if the correct and buggy algorithms produced + * exactly the same result. If so and if "sign" is non-zero, which indicates + * that there was a non-benign sign extension, this means that we have a + * collision between the correctly computed hash for this password and a set of + * passwords that could be supplied to the buggy algorithm. Our safety measure + * is meant to protect from such many-buggy to one-correct collisions, by + * deviating from the correct algorithm in such cases. Let's check for this. + */ + diff |= diff >> 16; /* still zero if exact match */ + diff &= 0xffff; /* ditto */ + diff += 0xffff; /* bit 16 set if "diff" was non-zero (on non-match) */ + sign <<= 9; /* move the non-benign sign extension flag to bit 16 */ + sign &= ~diff & safety; /* action needed? */ + +/* + * If we have determined that we need to deviate from the correct algorithm, + * flip bit 16 in initial expanded key. (The choice of 16 is arbitrary, but + * let's stick to it now. It came out of the approach we used above, and it's + * not any worse than any other choice we could make.) + * + * It is crucial that we don't do the same to the expanded key used in the main + * Eksblowfish loop. By doing it to only one of these two, we deviate from a + * state that could be directly specified by a password to the buggy algorithm + * (and to the fully correct one as well, but that's a side-effect). + */ + initial[0] ^= sign; +} + +static int ibcrypt(size_t length, char *dst, + const char *key, const char *scheme, + int minlog2rounds, + int log2rounds, const uint8_t *salt) +{ + struct { + struct blowfish_ctx ctx; + bf_key expanded_key; + union { + uint32_t salt[4]; + uint32_t output[6]; + } binary; + } data; + uint8_t psalt[BLOWFISH_BCRYPT_BINSALT_SIZE]; + uint32_t L, R; + uint32_t *ptr; + uint32_t count; + int i; + size_t schemelen = strlen(scheme); + unsigned cscheme; + unsigned bug = 0; + uint32_t safety = 0; + if (length < BLOWFISH_BCRYPT_HASH_SIZE || + schemelen < 2) + return 0; + + if (schemelen >= 3 && *scheme++ != '$') + return 0; + if (*scheme++ != '2') + return 0; + + switch (cscheme = *scheme++) { + default: + return 0; + case 'a': safety = 0x10000; + break; + case 'x': bug = 1; + break; + case 'b': case 'y': + break; + } + + if (schemelen >= 4) { + if (*scheme++ != '$') + return 0; + if (schemelen >= 6) { + if (log2rounds < 0) + log2rounds = atoi(scheme); + scheme += 2; + if (schemelen >= CRYPTPLEN && *scheme++ != '$') + return 0; + if (schemelen >= HASHOFFSET && !salt) { + struct base64_decode_ctx ctx; + size_t saltlen = BLOWFISH_BCRYPT_BINSALT_SIZE; + + base64_decode_init(&ctx); + ctx.table = radix64_decode_table; + + if (!base64_decode_update(&ctx, &saltlen, (uint8_t *) data.binary.salt, + SALTLEN, scheme) + || saltlen != BLOWFISH_BCRYPT_BINSALT_SIZE) + return 0; + } + } + } + + if (salt) + memcpy(data.binary.salt, salt, BLOWFISH_BCRYPT_BINSALT_SIZE); + else if (schemelen < HASHOFFSET) + return 0; + memcpy(psalt, data.binary.salt, BLOWFISH_BCRYPT_BINSALT_SIZE); + swap32(data.binary.salt, 4); + + if (log2rounds < minlog2rounds || log2rounds > 31) + return 0; + count = (uint32_t)1 << log2rounds; + + set_xkey(key, data.expanded_key, data.ctx.p, bug, safety); + memcpy(data.ctx.s, _nettle_blowfish_initial_ctx.s, sizeof(data.ctx.s)); + + L = R = 0; + for (i = 0; i < _BLOWFISH_ROUNDS + 2; i += 2) { + L ^= data.binary.salt[i & 2]; + R ^= data.binary.salt[(i & 2) + 1]; + _nettle_blowfish_encround(&data.ctx, &L, &R); + data.ctx.p[i] = L; + data.ctx.p[i + 1] = R; + } + + ptr = data.ctx.s[0]; + do { + ptr += 4; + L ^= data.binary.salt[(_BLOWFISH_ROUNDS + 2) & 3]; + R ^= data.binary.salt[(_BLOWFISH_ROUNDS + 3) & 3]; + _nettle_blowfish_encround(&data.ctx, &L, &R); + *(ptr - 4) = L; + *(ptr - 3) = R; + + L ^= data.binary.salt[(_BLOWFISH_ROUNDS + 4) & 3]; + R ^= data.binary.salt[(_BLOWFISH_ROUNDS + 5) & 3]; + _nettle_blowfish_encround(&data.ctx, &L, &R); + *(ptr - 2) = L; + *(ptr - 1) = R; + } while (ptr < &data.ctx.s[3][0xFF]); + + do { + int done; + + for (i = 0; i < _BLOWFISH_ROUNDS + 2; i += 2) { + data.ctx.p[i] ^= data.expanded_key[i]; + data.ctx.p[i + 1] ^= data.expanded_key[i + 1]; + } + + done = 0; + do { + uint32_t tmp1, tmp2, tmp3, tmp4; + + L = R = 0; + ptr = data.ctx.p; + do { + ptr += 2; + _nettle_blowfish_encround(&data.ctx, &L, &R); + *(ptr - 2) = L; + *(ptr - 1) = R; + } while (ptr < &data.ctx.p[_BLOWFISH_ROUNDS + 2]); + + ptr = data.ctx.s[0]; + do { + ptr += 2; + _nettle_blowfish_encround(&data.ctx, &L, &R); + *(ptr - 2) = L; + *(ptr - 1) = R; + } while (ptr < &data.ctx.s[3][0xFF]); + + if (done) + break; + done = 1; + + tmp1 = data.binary.salt[0]; + tmp2 = data.binary.salt[1]; + tmp3 = data.binary.salt[2]; + tmp4 = data.binary.salt[3]; + for (i = 0; i < _BLOWFISH_ROUNDS; i += 4) { + data.ctx.p[i] ^= tmp1; + data.ctx.p[i + 1] ^= tmp2; + data.ctx.p[i + 2] ^= tmp3; + data.ctx.p[i + 3] ^= tmp4; + } + data.ctx.p[16] ^= tmp1; + data.ctx.p[17] ^= tmp2; + } while (1); + } while (--count); + + for (i = 0; i < 6; i += 2) { + L = magic_w[i]; + R = magic_w[i + 1]; + + count = 64; + do + _nettle_blowfish_encround(&data.ctx, &L, &R); + while (--count); + + data.binary.output[i] = L; + data.binary.output[i + 1] = R; + } + + *dst++ = '$'; + *dst++ = '2'; + *dst++ = cscheme; + *dst++ = '$'; + *dst++ = '0' + log2rounds / 10; + *dst++ = '0' + log2rounds % 10; + *dst++ = '$'; + dst = encode_radix64(dst, BLOWFISH_BCRYPT_BINSALT_SIZE, psalt) - 1; + + swap32(data.binary.output, 6); +/* This has to be bug-compatible with the original implementation, so + only encode 23 of the 24 bytes. */ + encode_radix64(dst, 23, (uint8_t *) data.binary.output); + return cscheme; +} + +/* + * Please preserve the runtime self-test. It serves two purposes at once: + * + * 1. We really can't afford the risk of producing incompatible hashes e.g. + * when there's something like gcc bug 26587 again, whereas an application or + * library integrating this code might not also integrate our external tests or + * it might not run them after every build. Even if it does, the miscompile + * might only occur on the production build, but not on a testing build (such + * as because of different optimization settings). It is painful to recover + * from incorrectly-computed hashes - merely fixing whatever broke is not + * enough. Thus, a proactive measure like this self-test is needed. + * + * 2. We don't want to leave sensitive data from our actual password hash + * computation on the stack or in registers. Previous revisions of the code + * would do explicit cleanups, but simply running the self-test after hash + * computation is more reliable. + * + * The performance cost of this quick self-test is around 0.6% at the "$2a$08" + * setting. + */ +int blowfish_bcrypt_hash(size_t length, char *dst, + const char *key, const char *scheme, + int log2rounds, const uint8_t *salt) +{ + const char *test_pw = "8b \xd0\xc1\xd2\xcf\xcc\xd8"; + const char *test_scheme = "$2a$00$abcdefghijklmnopqrstuu"; + static const char * const test_hashes[2] = + {"i1D709vfamulimlGcq0qq3UvuUasvEa\0\x55", /* 'a', 'b', 'y' */ + "VUrPmXD6q/nVSSp7pNDhCR9071IfIRe\0\x55"}; /* 'x' */ + const char *test_hash = test_hashes[0]; + int cscheme; + int ok; + struct { + char s[HASHOFFSET + 1]; + char o[HASHOFFSET + 31 + 1 + 1 + 1]; + } buf; + + if (length) + *dst = '\0'; +/* Hash the supplied password */ + cscheme = ibcrypt(length, dst, key, scheme, 4, log2rounds, salt); + +/* + * Do a quick self-test. It is important that we make both calls to ibcrypt() + * from the same scope such that they likely use the same stack locations, + * which makes the second call overwrite the first call's sensitive data on the + * stack and makes it more likely that any alignment related issues would be + * detected by the self-test. + */ + memcpy(buf.s, test_scheme, sizeof(buf.s)); + + if (cscheme) + test_hash = test_hashes[(buf.s[2] = cscheme) == 'x']; + + memset(buf.o, 0x55, sizeof(buf.o)); + buf.o[sizeof(buf.o) - 1] = 0; + ok = ibcrypt(sizeof(buf.o) - (1 + 1), buf.o, test_pw, + buf.s, 0, -1, (void*)0); + + ok = (ok && + !memcmp(buf.o, buf.s, HASHOFFSET) && + !memcmp(buf.o + HASHOFFSET, test_hash, 31 + 1 + 1 + 1)); + + { + const char *k = "\xff\xa3" "34" "\xff\xff\xff\xa3" "345"; + bf_key ae, ai, ye, yi; + set_xkey(k, ae, ai, 0, 0x10000); /* $2a$ */ + set_xkey(k, ye, yi, 0, 0); /* $2y$ */ + ai[0] ^= 0x10000; /* undo the safety (for comparison) */ + ok = ok && ai[0] == 0xdb9c59bc && ye[17] == 0x33343500 && + !memcmp(ae, ye, sizeof(ae)) && + !memcmp(ai, yi, sizeof(ai)); + } + + return ok && !!cscheme; +} diff --git a/blowfish.h b/blowfish-internal.h similarity index 50% copy from blowfish.h copy to blowfish-internal.h index bcdc7cb6..da28fa34 100644 --- a/blowfish.h +++ b/blowfish-internal.h @@ -1,4 +1,4 @@ -/* blowfish.h +/* blowfish-internal.h Blowfish block cipher. @@ -32,8 +32,8 @@ not, see http://www.gnu.org/licenses/. */ -#ifndef NETTLE_BLOWFISH_H_INCLUDED -#define NETTLE_BLOWFISH_H_INCLUDED +#ifndef NETTLE_BLOWFISH_INTERNAL_H_INCLUDED +#define NETTLE_BLOWFISH_INTERNAL_H_INCLUDED #include "nettle-types.h" @@ -41,49 +41,12 @@ extern "C" { #endif -/* Name mangling */ -#define blowfish_set_key nettle_blowfish_set_key -#define blowfish128_set_key nettle_blowfish128_set_key -#define blowfish_encrypt nettle_blowfish_encrypt -#define blowfish_decrypt nettle_blowfish_decrypt - -#define BLOWFISH_BLOCK_SIZE 8 - -/* Variable key size between 64 and 448 bits. */ -#define BLOWFISH_MIN_KEY_SIZE 8 -#define BLOWFISH_MAX_KEY_SIZE 56 - -/* Default to 128 bits */ -#define BLOWFISH_KEY_SIZE 16 - -#define BLOWFISH128_KEY_SIZE 16 - -#define _BLOWFISH_ROUNDS 16 - -struct blowfish_ctx -{ - uint32_t s[4][256]; - uint32_t p[_BLOWFISH_ROUNDS+2]; -}; - -/* Returns 0 for weak keys, otherwise 1. */ -int -blowfish_set_key(struct blowfish_ctx *ctx, - size_t length, const uint8_t *key); -int -blowfish128_set_key(struct blowfish_ctx *ctx, const uint8_t *key); - -void -blowfish_encrypt(const struct blowfish_ctx *ctx, - size_t length, uint8_t *dst, - const uint8_t *src); -void -blowfish_decrypt(const struct blowfish_ctx *ctx, - size_t length, uint8_t *dst, - const uint8_t *src); +extern const struct blowfish_ctx _nettle_blowfish_initial_ctx; +extern void _nettle_blowfish_encround (const struct blowfish_ctx *ctx, + uint32_t * ret_xl, uint32_t * ret_xr); #ifdef __cplusplus } #endif -#endif /* NETTLE_BLOWFISH_H_INCLUDED */ +#endif /* NETTLE_BLOWFISH_INTERNAL_H_INCLUDED */ diff --git a/blowfish.c b/blowfish.c index 52040f13..e73caffe 100644 --- a/blowfish.c +++ b/blowfish.c @@ -54,12 +54,13 @@ #include <assert.h> #include "blowfish.h" +#include "blowfish-internal.h" #include "macros.h" /* precomputed S boxes */ -static const struct blowfish_ctx -initial_ctx = { +const struct blowfish_ctx +_nettle_blowfish_initial_ctx = { { { /* ks0 */ 0xD1310BA6, 0x98DFB5AC, 0x2FFD72DB, 0xD01ADFB7, 0xB8E1AFED, 0x6A267E96, @@ -261,8 +262,8 @@ initial_ctx = { #define R(c, l,r,i) do { l ^= c->p[i]; r ^= F(c,l); } while(0) -static void -encrypt (const struct blowfish_ctx *ctx, uint32_t * ret_xl, +void +_nettle_blowfish_encround (const struct blowfish_ctx *ctx, uint32_t * ret_xl, uint32_t * ret_xr) { uint32_t xl, xr; @@ -295,7 +296,7 @@ encrypt (const struct blowfish_ctx *ctx, uint32_t * ret_xl, } static void -decrypt (const struct blowfish_ctx *ctx, uint32_t * ret_xl, uint32_t * ret_xr) +decround (const struct blowfish_ctx *ctx, uint32_t * ret_xl, uint32_t * ret_xr) { uint32_t xl, xr; @@ -339,7 +340,7 @@ blowfish_encrypt (const struct blowfish_ctx *ctx, d1 = READ_UINT32(src); d2 = READ_UINT32(src+4); - encrypt (ctx, &d1, &d2); + _nettle_blowfish_encround (ctx, &d1, &d2); dst[0] = (d1 >> 24) & 0xff; dst[1] = (d1 >> 16) & 0xff; dst[2] = (d1 >> 8) & 0xff; @@ -361,7 +362,7 @@ blowfish_decrypt (const struct blowfish_ctx *ctx, d1 = READ_UINT32(src); d2 = READ_UINT32(src+4); - decrypt (ctx, &d1, &d2); + decround (ctx, &d1, &d2); dst[0] = (d1 >> 24) & 0xff; dst[1] = (d1 >> 16) & 0xff; dst[2] = (d1 >> 8) & 0xff; @@ -380,7 +381,7 @@ blowfish_set_key (struct blowfish_ctx *ctx, int i, j; uint32_t data, datal, datar; - *ctx = initial_ctx; + *ctx = _nettle_blowfish_initial_ctx; for (i = j = 0; i < _BLOWFISH_ROUNDS + 2; i++) { @@ -393,15 +394,15 @@ blowfish_set_key (struct blowfish_ctx *ctx, datal = datar = 0; for (i = 0; i < _BLOWFISH_ROUNDS + 2; i += 2) { - encrypt (ctx, &datal, &datar); + _nettle_blowfish_encround (ctx, &datal, &datar); ctx->p[i] = datal; ctx->p[i + 1] = datar; } - + for (j = 0; j < 4; j++) for (i = 0; i < 256; i += 2) { - encrypt (ctx, &datal, &datar); + _nettle_blowfish_encround (ctx, &datal, &datar); ctx->s[j][i] = datal; ctx->s[j][i + 1] = datar; } diff --git a/blowfish.h b/blowfish.h index bcdc7cb6..af48e20f 100644 --- a/blowfish.h +++ b/blowfish.h @@ -46,6 +46,8 @@ extern "C" { #define blowfish128_set_key nettle_blowfish128_set_key #define blowfish_encrypt nettle_blowfish_encrypt #define blowfish_decrypt nettle_blowfish_decrypt +#define blowfish_bcrypt_hash nettle_blowfish_bcrypt_hash +#define blowfish_bcrypt_verify nettle_blowfish_bcrypt_verify #define BLOWFISH_BLOCK_SIZE 8 @@ -60,6 +62,9 @@ extern "C" { #define _BLOWFISH_ROUNDS 16 +#define BLOWFISH_BCRYPT_HASH_SIZE (60 + 1) /* Including null-terminator */ +#define BLOWFISH_BCRYPT_BINSALT_SIZE 16 /* Binary string size */ + struct blowfish_ctx { uint32_t s[4][256]; @@ -81,6 +86,16 @@ void blowfish_decrypt(const struct blowfish_ctx *ctx, size_t length, uint8_t *dst, const uint8_t *src); +int +blowfish_bcrypt_hash(size_t length, + char *dst, + const char *key, + const char *scheme, + int log2rounds, + const uint8_t *salt); +int +blowfish_bcrypt_verify(const char *key, + const char *hashed); #ifdef __cplusplus } diff --git a/nettle.texinfo b/nettle.texinfo index 995d5de8..75e18b58 100644 --- a/nettle.texinfo +++ b/nettle.texinfo @@ -1513,6 +1513,73 @@ in any other way. Analogous to @code{blowfish_encrypt} @end deftypefun +@deftypefun int blowfish_bcrypt_hash (size_t @var{length}, char *@var{dst}, const char *@var{key}, const char *@var{scheme}, int @var{log2rounds}, const uint8_t *@var{salt}) +Compute the bcrypt password hash. +The function will return @code{0} if the hash cannot be computed +due to invalid input. +The function will return @code{1} and store the computed hash +in the array pointed to by @var{dst}. The hash is computed based +on the chosen @var{scheme}, number of rounds @var{log2rounds} and +specified @var{salt}. + +@var{length} must be at least @code{BLOWFISH_BCRYPT_HASH_SIZE}. + +@var{dst} must point to a character array of the specified @var{length}. + +@var{key} contains the zero terminated plaintext password string. + +@var{scheme} contains either just the chosen scheme (valid schemes +are: @code{2a}, @code{2b}, @code{2x} or @code{2y}), or +(the prefix of) an existing hashed password (typically @code{$2b$10$...}). + +@var{log2rounds} contains the log2 of the number of encryption rounds +that must be used to compute the hash. If it is @code{-1} the value +will be extracted from @var{scheme}. + +@var{salt} should point to an array of @code{BLOWFISH_BCRYPT_BINSALT_SIZE} +random bytes to be used to perturb the hash computation. If it is @code{NULL} +the salt will be extracted from @var{scheme}. + +Sample code to generate a bcrypt hash: +@example +char cleartxtpassword[] = "ExamplePassword"; +uint8_t salt[BLOWFISH_BCRYPT_BINSALT_SIZE]; +@dots{} +/* Make sure that salt is filled with random bytes */ +@dots{} +char hashedresult[BLOWFISH_BCRYPT_HASH_SIZE]; +int result = blowfish_bcrypt(sizeof(hashedresult), hashedresult, + cleartxtpassword, "2b", 10, salt); +if (result) + printf("%s\n", hashedresult); +@end example +@end deftypefun + +@deftypefun int blowfish_bcrypt_verify (const char *@var{key}, const char *@var{hashed}) +Verifies the bcrypt password hash against the supplied plaintext password. +The function will return @code{0} if the password does not match. +The function will return @code{1} if the password matches. + +@var{key} contains the zero terminated plaintext password string. + +@var{hashed} contains the zero terminated hashed string to compare with. + +Sample code to verify a bcrypt hash: +@example +char cleartxtpassword[] = "ExamplePassword"; +char existinghashed[] = + "$2y$" /* Hash algorithm version */ + "10" /* 2^10 hash rounds (strength) */ + "$" /* separator */ + "1b2lPgo4XumibnJGN3r3sO" /* base64 encoded 16-byte salt */ + "u7wE7xNfYDKlAxZffJDCJdVfFTAyevu"; /* Hashedpart */ +if (blowfish_bcrypt_verify(cleartxtpassword, existinghashed)) + printf("Password is correct."); +else + printf("Password is incorrect."); +@end example +@end deftypefun + @subsection Camellia @cindex Camellia -- 2.20.1

2 7

Fwd: PPC64LE optimizing AES and GHASH
by Maamoun TK 22 Jun '20

22 Jun '20

On Sat, Jun 20, 2020 at 11:54 AM Niels Möller <nisse(a)lysator.liu.se> wrote: > Have you measured speedup when going from 4 to 8 blocks? We shouldn't > add larger loops than needed. > The 8x loop has x~1.15 performance boost over 4x loop, if you think it's not worth it, I can add only 4x loop to make the code simpler. > Do you measure a speedup from this? Karatsuba usually pays off only for > a bit larger sizes (but I guess overhead is a little less here than for > standard multiplication). > Actually, I considered the Karatsuba algorithm not only for performance but to reduce the number of registers used. However, I believe that using the Karatsuba algorithm in my case performs better or similar to classical multiplication. > > - Since the functionality of gcm_set_key() is replaced with > > gcm_init_key() for PowerPC64LE, two warnings will pop up: > [‘gcm_gf_shift’ > > defined but not used] and [‘gcm_gf_add’ defined but not used] > When I applied the patch to the last upstream, these warnings did not appear, some changes have occurred to gcm.c, I will look at it. > To test PPC code, I wonder if it's easy to add a PPC build to > .gitlab-ci, in the same way as arm and mips tests. These are based on > Debian packaged cross compilers and qemu-user. I'm also not that familiar > with the variants within the Power and PowerPC family of processors. > I will see what I can do.

1 0

PPC64LE optimizing AES and GHASH
by Maamoun TK 20 Jun '20

20 Jun '20

I added a PowerPC64LE optimized version of AES and GHASH to nettle. Patch summary: GHASH Algorithm I took the advantage of several references and researches to achieve the high-speed implementation of this algorithm. These references include several techniques that have been used to improve the performance of the algorithm, I will summarize the important techniques used as follows: - The main equation: The main equation for 4 block (128-bit each) can be seen in reference [1] Digest = (((((((Digest⊕C0)*H)⊕C1)*H)⊕C2)*H)⊕C3)*H = ((Digest⊕C0)*H4)⊕(C1*H3)⊕(C2*H2)⊕(C3*H) to achieve more parallelism, this equation can be modified to address 8 blocks per one loop. It looks like as follows Digest = ((Digest⊕C0)*H8)⊕(C1*H7)⊕(C2*H6)⊕(C3*H5)⊕(C4*H4)⊕(C5*H3)⊕(C6*H2)⊕(C7*H) - Handling Bit-reflection of the multiplication product [1]: This technique moves part of the workload inside the loop to the init function so it is executed only once. - Karatsuba Algorithm: This algorithm allows to perform three multiplication instructions instead of four, in exchange for two additional Xor. This technique is well explained with figures in reference [1] - Deferred Recombination of partial products This technique is well explained with figures in reference [1] - Multiplication-based reduction: I tested both classical shift-based reduction and multiplication-based reduction, the multiplication-based reduction achieved better performance and less instructions. Example of both reductions can be seen in reference [2] AES Power ISA makes it easy to optimize AES by offering built-in AES instructions. AES-GCM performance (Tested on POWER9): - GCM_AES Encrypt ~x13.5 of nettle C implementation - GCM_AES Decrypt ~x13.5 of nettle C implementation - GCM_AES Update (Only GHASH is called) ~x26 of nettle C implementation Notes: - Test 128 bytes is added to gcm-test in testsuite to test 8x loop in GHASH optimized function. - Since the functionality of gcm_set_key() is replaced with gcm_init_key() for PowerPC64LE, two warnings will pop up: [‘gcm_gf_shift’ defined but not used] and [‘gcm_gf_add’ defined but not used] References: [1] https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/c… [2] https://www.intel.com/content/dam/www/public/us/en/documents/software-suppo… [3] https://software.intel.com/file/24918 [4] https://github.com/dot-asm/cryptogams

2 3

Post patch on nettle-bugs list
by Maamoun TK 20 Jun '20

20 Jun '20

Is there a limit to the number of lines of a message posted on the list or special option that should be passed to git diff? I created a patch with command "git diff --stat --summary --patch HEAD~1 > patch.patch", when I posted this patch on the list nothing happened.

2 1

[PATCH] optimizing AES and GHASH for PPC64LE
by Maamoun TK 19 Jun '20

19 Jun '20

Makefile.in | 2 +- configure.ac | 5 + gcm.c | 19 +- powerpc64le/aes-decrypt-internal.asm | 573 ++++++++++++++++++++ powerpc64le/aes-encrypt-internal.asm | 534 +++++++++++++++++++ powerpc64le/gcm-hash8.asm | 992 +++++++++++++++++++++++++++++++++++ powerpc64le/machine.m4 | 0 testsuite/gcm-test.c | 23 + 8 files changed, 2146 insertions(+), 2 deletions(-) create mode 100644 powerpc64le/aes-decrypt-internal.asm create mode 100644 powerpc64le/aes-encrypt-internal.asm create mode 100644 powerpc64le/gcm-hash8.asm create mode 100644 powerpc64le/machine.m4 diff --git a/Makefile.in b/Makefile.in index 64ff1001..5bbc0f79 100644 --- a/Makefile.in +++ b/Makefile.in @@ -603,7 +603,7 @@ distdir: $(DISTFILES) done set -e; for d in sparc32 sparc64 x86 \ x86_64 x86_64/aesni x86_64/sha_ni x86_64/fat \ - arm arm/neon arm/v6 arm/fat ; do \ + arm arm/neon arm/v6 arm/fat powerpc64le ; do \ mkdir "$(distdir)/$$d" ; \ find "$(srcdir)/$$d" -maxdepth 1 '(' -name '*.asm' -o -name '*.m4' ')' \ -exec cp '{}' "$(distdir)/$$d" ';' ; \ diff --git a/configure.ac b/configure.ac index 90ea1ea8..1ea54ce8 100644 --- a/configure.ac +++ b/configure.ac @@ -435,6 +435,9 @@ if test "x$enable_assembler" = xyes ; then esac fi ;; + *powerpc64le*) + asm_path=powerpc64le + ;; *) enable_assembler=no ;; @@ -572,7 +575,9 @@ AH_VERBATIM([HAVE_NATIVE], #undef HAVE_NATIVE_ecc_secp384r1_redc #undef HAVE_NATIVE_ecc_secp521r1_modp #undef HAVE_NATIVE_ecc_secp521r1_redc +#undef HAVE_NATIVE_gcm_init_key8 #undef HAVE_NATIVE_gcm_hash8 +#undef HAVE_NATIVE_gcm_fill #undef HAVE_NATIVE_salsa20_core #undef HAVE_NATIVE_sha1_compress #undef HAVE_NATIVE_sha256_compress diff --git a/gcm.c b/gcm.c index cf615daf..809c03bc 100644 --- a/gcm.c +++ b/gcm.c @@ -140,6 +140,12 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) memcpy (x->b, Z.b, sizeof(Z)); } # elif GCM_TABLE_BITS == 8 +# if HAVE_NATIVE_gcm_init_key8 + +#define gcm_init_key _nettle_gcm_init_key8 +void +_nettle_gcm_init_key8 (union nettle_block16 *table); +# endif /* HAVE_NATIVE_gcm_init_key8 */ # if HAVE_NATIVE_gcm_hash8 #define gcm_hash _nettle_gcm_hash8 @@ -225,6 +231,13 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) #endif /* GCM_TABLE_BITS */ +#if HAVE_NATIVE_gcm_fill + +#define gcm_fill _nettle_gcm_fill +void +_nettle_gcm_fill (uint8_t *ctr, size_t blocks, union nettle_block16 *buffer); +#endif /* HAVE_NATIVE_gcm_fill */ + /* Increment the rightmost 32 bits. */ #define INC32(block) INCREMENT(4, (block.b) + GCM_BLOCK_SIZE - 4) @@ -245,7 +258,9 @@ gcm_set_key(struct gcm_key *key, memset(key->h[0].b, 0, GCM_BLOCK_SIZE); f (cipher, GCM_BLOCK_SIZE, key->h[i].b, key->h[0].b); -#if GCM_TABLE_BITS +#ifdef gcm_init_key + gcm_init_key(key->h); +#elif GCM_TABLE_BITS /* Algorithm 3 from the gcm paper. First do powers of two, then do the rest by adding. */ while (i /= 2) @@ -333,6 +348,7 @@ gcm_update(struct gcm_ctx *ctx, const struct gcm_key *key, ctx->auth_size += length; } +#ifndef gcm_fill static nettle_fill16_func gcm_fill; static void gcm_fill(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) @@ -349,6 +365,7 @@ gcm_fill(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) WRITE_UINT32(ctr + GCM_BLOCK_SIZE - 4, c); } +#endif /* !gcm_fill */ void gcm_encrypt (struct gcm_ctx *ctx, const struct gcm_key *key, diff --git a/powerpc64le/aes-decrypt-internal.asm b/powerpc64le/aes-decrypt-internal.asm new file mode 100644 index 00000000..bde34779 --- /dev/null +++ b/powerpc64le/aes-decrypt-internal.asm @@ -0,0 +1,573 @@ +C powerpc64le/aes-decrypt-internal.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<ROUNDS>, <3>) +define(<KEYS>, <4>) +define(<LENGTH>, <6>) +define(<DST>, <7>) +define(<SRC>, <8>) + +define(<swap_mask>, <0>) + +define(<K>, <1>) +define(<S0>, <2>) +define(<S1>, <3>) +define(<S2>, <4>) +define(<S3>, <5>) +define(<S4>, <6>) +define(<S5>, <7>) +define(<S6>, <8>) +define(<S7>, <9>) +define(<S8>, <10>) +define(<S9>, <11>) +define(<S10>, <12>) +define(<S11>, <13>) +define(<S12>, <14>) +define(<S13>, <15>) +define(<S14>, <16>) +define(<S15>, <17>) + +define(<KX>, <33>) +define(<S0X>, <34>) +define(<S1X>, <35>) +define(<S2X>, <36>) +define(<S3X>, <37>) +define(<S4X>, <38>) +define(<S5X>, <39>) +define(<S6X>, <40>) +define(<S7X>, <41>) +define(<S8X>, <42>) +define(<S9X>, <43>) +define(<S10X>, <44>) +define(<S11X>, <45>) +define(<S12X>, <46>) +define(<S13X>, <47>) +define(<S14X>, <48>) +define(<S15X>, <49>) + +C ZERO vector register is used in place of RoundKey +C for vncipher instruction because the order of InvMixColumns +C and Xor processes are flipped in that instruction. +C The Xor process with RoundKey is executed afterward. +define(<ZERO>, <18>) + + .file "aes-decrypt-internal.asm" + + C _aes_decrypt(unsigned rounds, const uint32_t *keys, + C const struct aes_table *T, + C size_t length, uint8_t *dst, + C uint8_t *src) + + .text +.align 5 +PROLOGUE(_nettle_aes_decrypt) + vxor ZERO,ZERO,ZERO + + ld 5,.swap_mask@got(TOCP) + lvx swap_mask,0,5 + + subi ROUNDS,ROUNDS,1 + srdi LENGTH,LENGTH,4 + + srdi 5,LENGTH,4 # 16x loop count + cmpldi 5,0 + beq L8x + + std 17,-120(SP); + std 18,-112(SP); + std 19,-104(SP); + std 20,-96(SP); + std 21,-88(SP); + std 22,-80(SP); + std 23,-72(SP); + std 24,-64(SP); + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 17,0x10 + li 18,0x20 + li 19,0x30 + li 20,0x40 + li 21,0x50 + li 22,0x60 + li 23,0x70 + li 24,0x80 + li 25,0x90 + li 26,0xA0 + li 27,0xB0 + li 28,0xC0 + li 29,0xD0 + li 30,0xE0 + li 31,0xF0 + +.align 5 +Lx16_loop: + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + lxvd2x S1X,17,SRC + lxvd2x S2X,18,SRC + lxvd2x S3X,19,SRC + lxvd2x S4X,20,SRC + lxvd2x S5X,21,SRC + lxvd2x S6X,22,SRC + lxvd2x S7X,23,SRC + lxvd2x S8X,24,SRC + lxvd2x S9X,25,SRC + lxvd2x S10X,26,SRC + lxvd2x S11X,27,SRC + lxvd2x S12X,28,SRC + lxvd2x S13X,29,SRC + lxvd2x S14X,30,SRC + lxvd2x S15X,31,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L16x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vncipher S4,S4,ZERO + vncipher S5,S5,ZERO + vncipher S6,S6,ZERO + vncipher S7,S7,ZERO + vncipher S8,S8,ZERO + vncipher S9,S9,ZERO + vncipher S10,S10,ZERO + vncipher S11,S11,ZERO + vncipher S12,S12,ZERO + vncipher S13,S13,ZERO + vncipher S14,S14,ZERO + vncipher S15,S15,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + addi 10,10,0x10 + bdnz L16x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + vncipherlast S4,S4,K + vncipherlast S5,S5,K + vncipherlast S6,S6,K + vncipherlast S7,S7,K + vncipherlast S8,S8,K + vncipherlast S9,S9,K + vncipherlast S10,S10,K + vncipherlast S11,S11,K + vncipherlast S12,S12,K + vncipherlast S13,S13,K + vncipherlast S14,S14,K + vncipherlast S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + stxvd2x S0X,0,DST + stxvd2x S1X,17,DST + stxvd2x S2X,18,DST + stxvd2x S3X,19,DST + stxvd2x S4X,20,DST + stxvd2x S5X,21,DST + stxvd2x S6X,22,DST + stxvd2x S7X,23,DST + stxvd2x S8X,24,DST + stxvd2x S9X,25,DST + stxvd2x S10X,26,DST + stxvd2x S11X,27,DST + stxvd2x S12X,28,DST + stxvd2x S13X,29,DST + stxvd2x S14X,30,DST + stxvd2x S15X,31,DST + + addi SRC,SRC,0x100 + addi DST,DST,0x100 + subic. 5,5,1 + bne Lx16_loop + + ld 17,-120(SP); + ld 18,-112(SP); + ld 19,-104(SP); + ld 20,-96(SP); + ld 21,-88(SP); + ld 22,-80(SP); + ld 23,-72(SP); + ld 24,-64(SP); + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi LENGTH,LENGTH,60 + +L8x: + srdi 5,LENGTH,3 + cmpldi 5,0 + beq L4x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + addi 9,9,0x10 + lxvd2x S4X,9,SRC + addi 9,9,0x10 + lxvd2x S5X,9,SRC + addi 9,9,0x10 + lxvd2x S6X,9,SRC + addi 9,9,0x10 + lxvd2x S7X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L8x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vncipher S4,S4,ZERO + vncipher S5,S5,ZERO + vncipher S6,S6,ZERO + vncipher S7,S7,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + addi 10,10,0x10 + bdnz L8x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + vncipherlast S4,S4,K + vncipherlast S5,S5,K + vncipherlast S6,S6,K + vncipherlast S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + addi 9,9,0x10 + stxvd2x S4X,9,DST + addi 9,9,0x10 + stxvd2x S5X,9,DST + addi 9,9,0x10 + stxvd2x S6X,9,DST + addi 9,9,0x10 + stxvd2x S7X,9,DST + + addi SRC,SRC,0x80 + addi DST,DST,0x80 + + clrldi LENGTH,LENGTH,61 + +L4x: + srdi 5,LENGTH,2 + cmpldi 5,0 + beq L2x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L4x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + addi 10,10,0x10 + bdnz L4x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + + addi SRC,SRC,0x40 + addi DST,DST,0x40 + + clrldi LENGTH,LENGTH,62 + +L2x: + srdi 5,LENGTH,1 + cmpldi 5,0 + beq L1x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L2x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vxor S0,S0,K + vxor S1,S1,K + addi 10,10,0x10 + bdnz L2x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + + addi SRC,SRC,0x20 + addi DST,DST,0x20 + + clrldi LENGTH,LENGTH,63 + +L1x: + cmpldi LENGTH,0 + beq Ldone + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + + vxor S0,S0,K + + vperm S0,S0,S0,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L1x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vxor S0,S0,K + addi 10,10,0x10 + bdnz L1x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + + vperm S0,S0,S0,swap_mask + + stxvd2x S0X,0,DST + +Ldone: + blr +EPILOGUE(_nettle_aes_decrypt) + + .data + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 diff --git a/powerpc64le/aes-encrypt-internal.asm b/powerpc64le/aes-encrypt-internal.asm new file mode 100644 index 00000000..1bbd86a8 --- /dev/null +++ b/powerpc64le/aes-encrypt-internal.asm @@ -0,0 +1,534 @@ +C powerpc64le/aes-encrypt-internal.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<ROUNDS>, <3>) +define(<KEYS>, <4>) +define(<LENGTH>, <6>) +define(<DST>, <7>) +define(<SRC>, <8>) + +define(<swap_mask>, <0>) + +define(<K>, <1>) +define(<S0>, <2>) +define(<S1>, <3>) +define(<S2>, <4>) +define(<S3>, <5>) +define(<S4>, <6>) +define(<S5>, <7>) +define(<S6>, <8>) +define(<S7>, <9>) +define(<S8>, <10>) +define(<S9>, <11>) +define(<S10>, <12>) +define(<S11>, <13>) +define(<S12>, <14>) +define(<S13>, <15>) +define(<S14>, <16>) +define(<S15>, <17>) + +define(<KX>, <33>) +define(<S0X>, <34>) +define(<S1X>, <35>) +define(<S2X>, <36>) +define(<S3X>, <37>) +define(<S4X>, <38>) +define(<S5X>, <39>) +define(<S6X>, <40>) +define(<S7X>, <41>) +define(<S8X>, <42>) +define(<S9X>, <43>) +define(<S10X>, <44>) +define(<S11X>, <45>) +define(<S12X>, <46>) +define(<S13X>, <47>) +define(<S14X>, <48>) +define(<S15X>, <49>) + + .file "aes-encrypt-internal.asm" + + C _aes_encrypt(unsigned rounds, const uint32_t *keys, + C const struct aes_table *T, + C size_t length, uint8_t *dst, + C uint8_t *src) + + .text +.align 5 +PROLOGUE(_nettle_aes_encrypt) + ld 5,.swap_mask@got(TOCP) + lvx swap_mask,0,5 + + subi ROUNDS,ROUNDS,1 + srdi LENGTH,LENGTH,4 + + srdi 5,LENGTH,4 # 16x loop count + cmpldi 5,0 + beq L8x + + std 17,-120(SP); + std 18,-112(SP); + std 19,-104(SP); + std 20,-96(SP); + std 21,-88(SP); + std 22,-80(SP); + std 23,-72(SP); + std 24,-64(SP); + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 17,0x10 + li 18,0x20 + li 19,0x30 + li 20,0x40 + li 21,0x50 + li 22,0x60 + li 23,0x70 + li 24,0x80 + li 25,0x90 + li 26,0xA0 + li 27,0xB0 + li 28,0xC0 + li 29,0xD0 + li 30,0xE0 + li 31,0xF0 + +.align 5 +Lx16_loop: + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + lxvd2x S1X,17,SRC + lxvd2x S2X,18,SRC + lxvd2x S3X,19,SRC + lxvd2x S4X,20,SRC + lxvd2x S5X,21,SRC + lxvd2x S6X,22,SRC + lxvd2x S7X,23,SRC + lxvd2x S8X,24,SRC + lxvd2x S9X,25,SRC + lxvd2x S10X,26,SRC + lxvd2x S11X,27,SRC + lxvd2x S12X,28,SRC + lxvd2x S13X,29,SRC + lxvd2x S14X,30,SRC + lxvd2x S15X,31,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L16x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + vcipher S4,S4,K + vcipher S5,S5,K + vcipher S6,S6,K + vcipher S7,S7,K + vcipher S8,S8,K + vcipher S9,S9,K + vcipher S10,S10,K + vcipher S11,S11,K + vcipher S12,S12,K + vcipher S13,S13,K + vcipher S14,S14,K + vcipher S15,S15,K + addi 10,10,0x10 + bdnz L16x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + vcipherlast S4,S4,K + vcipherlast S5,S5,K + vcipherlast S6,S6,K + vcipherlast S7,S7,K + vcipherlast S8,S8,K + vcipherlast S9,S9,K + vcipherlast S10,S10,K + vcipherlast S11,S11,K + vcipherlast S12,S12,K + vcipherlast S13,S13,K + vcipherlast S14,S14,K + vcipherlast S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + stxvd2x S0X,0,DST + stxvd2x S1X,17,DST + stxvd2x S2X,18,DST + stxvd2x S3X,19,DST + stxvd2x S4X,20,DST + stxvd2x S5X,21,DST + stxvd2x S6X,22,DST + stxvd2x S7X,23,DST + stxvd2x S8X,24,DST + stxvd2x S9X,25,DST + stxvd2x S10X,26,DST + stxvd2x S11X,27,DST + stxvd2x S12X,28,DST + stxvd2x S13X,29,DST + stxvd2x S14X,30,DST + stxvd2x S15X,31,DST + + addi SRC,SRC,0x100 + addi DST,DST,0x100 + subic. 5,5,1 + bne Lx16_loop + + ld 17,-120(SP); + ld 18,-112(SP); + ld 19,-104(SP); + ld 20,-96(SP); + ld 21,-88(SP); + ld 22,-80(SP); + ld 23,-72(SP); + ld 24,-64(SP); + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi LENGTH,LENGTH,60 + +L8x: + srdi 5,LENGTH,3 + cmpldi 5,0 + beq L4x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + addi 9,9,0x10 + lxvd2x S4X,9,SRC + addi 9,9,0x10 + lxvd2x S5X,9,SRC + addi 9,9,0x10 + lxvd2x S6X,9,SRC + addi 9,9,0x10 + lxvd2x S7X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L8x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + vcipher S4,S4,K + vcipher S5,S5,K + vcipher S6,S6,K + vcipher S7,S7,K + addi 10,10,0x10 + bdnz L8x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + vcipherlast S4,S4,K + vcipherlast S5,S5,K + vcipherlast S6,S6,K + vcipherlast S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + addi 9,9,0x10 + stxvd2x S4X,9,DST + addi 9,9,0x10 + stxvd2x S5X,9,DST + addi 9,9,0x10 + stxvd2x S6X,9,DST + addi 9,9,0x10 + stxvd2x S7X,9,DST + + addi SRC,SRC,0x80 + addi DST,DST,0x80 + + clrldi LENGTH,LENGTH,61 + +L4x: + srdi 5,LENGTH,2 + cmpldi 5,0 + beq L2x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L4x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + addi 10,10,0x10 + bdnz L4x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + + addi SRC,SRC,0x40 + addi DST,DST,0x40 + + clrldi LENGTH,LENGTH,62 + +L2x: + srdi 5,LENGTH,1 + cmpldi 5,0 + beq L1x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L2x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + addi 10,10,0x10 + bdnz L2x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + + addi SRC,SRC,0x20 + addi DST,DST,0x20 + + clrldi LENGTH,LENGTH,63 + +L1x: + cmpldi LENGTH,0 + beq Ldone + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + + vxor S0,S0,K + + vperm S0,S0,S0,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L1x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + addi 10,10,0x10 + bdnz L1x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + + vperm S0,S0,S0,swap_mask + + stxvd2x S0X,0,DST + +Ldone: + blr +EPILOGUE(_nettle_aes_encrypt) + + .data + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 diff --git a/powerpc64le/gcm-hash8.asm b/powerpc64le/gcm-hash8.asm new file mode 100644 index 00000000..a809f6ef --- /dev/null +++ b/powerpc64le/gcm-hash8.asm @@ -0,0 +1,992 @@ +C powerpc64le/gcm-hash8.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: +C VSX instructions is used to load and store data to memory "lxvd2x, stxvd2x" +C instead of VR instructions "lvx, stvx" as a workaround to access unaligned data +C VSX registers are defined with "X" suffix + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<TABLE>, <3>) +define(<X>, <4>) +define(<LENGTH>, <5>) +define(<DATA>, <6>) + +define(<zero>, <0>) +define(<swap_mask>, <1>) +define(<hidw_mask>, <2>) +define(<lodw_mask>, <3>) +define(<poly>, <4>) +define(<poly_h>, <4>) +define(<poly_l>, <5>) +define(<RP>, <6>) +define(<Mh>, <7>) +define(<Ml>, <8>) +define(<H>, <9>) +define(<Hh>, <10>) +define(<Hl>, <11>) +define(<RP2>, <9>) +define(<M2h>, <10>) +define(<M2l>, <11>) + +define(<HX>, <41>) +define(<HhX>, <42>) +define(<HlX>, <43>) +define(<H_HhX>, <44>) +define(<H_HX>, <45>) +define(<H_HlX>, <46>) + +define(<sl1>, <1>) +define(<msb>, <5>) +define(<H2>, <6>) +define(<H2h>, <7>) +define(<H2l>, <8>) +define(<H_h>, <12>) +define(<H_m>, <13>) +define(<H_l>, <14>) +define(<H_Hh>, <12>) +define(<H_H>, <13>) +define(<H_Hl>, <14>) +define(<H_t>, <15>) +define(<H2_h>, <16>) +define(<H2_m>, <17>) +define(<H2_l>, <18>) +define(<H2_t>, <19>) + +define(<C0X>, <38>) +define(<C1X>, <39>) +define(<C2X>, <40>) +define(<C3X>, <44>) +define(<C4X>, <38>) +define(<C5X>, <39>) +define(<C6X>, <40>) +define(<C7X>, <44>) + +define(<CX>, <45>) + +define(<C0>, <6>) +define(<C1>, <7>) +define(<C2>, <8>) +define(<C3>, <12>) +define(<C4>, <6>) +define(<C5>, <7>) +define(<C6>, <8>) +define(<C7>, <12>) + +define(<C>, <13>) + +define(<Ch>, <14>) +define(<Cl>, <15>) +define(<Cm>, <16>) + +define(<C01h>, <14>) +define(<C01l>, <15>) +define(<C01>, <16>) +define(<C23h>, <17>) +define(<C23l>, <18>) +define(<C23>, <19>) +define(<C45h>, <20>) +define(<C45l>, <21>) +define(<C45>, <22>) +define(<C67h>, <6>) +define(<C67l>, <7>) +define(<C67>, <8>) + +define(<H21>, <9>) +define(<H21h>, <10>) +define(<H21l>, <11>) +define(<H43>, <23>) +define(<H43h>, <24>) +define(<H43l>, <25>) +define(<H65>, <26>) +define(<H65h>, <27>) +define(<H65l>, <28>) +define(<H87>, <29>) +define(<H87h>, <30>) +define(<H87l>, <31>) + +define(<H21X>, <41>) +define(<H21hX>, <42>) +define(<H21lX>, <43>) +define(<H43X>, <55>) +define(<H43hX>, <56>) +define(<H43lX>, <57>) +define(<H65X>, <58>) +define(<H65hX>, <59>) +define(<H65lX>, <60>) +define(<H87X>, <61>) +define(<H87hX>, <62>) +define(<H87lX>, <63>) + +# gcm_fill registers: + +define(<CTR>, <3>) +define(<BLOCKS>, <4>) +define(<BUFFER>, <5>) + +define(<CTR0>, <2>) +define(<CTR0S>, <3>) +define(<CTR1>, <4>) +define(<CTR2>, <5>) +define(<CTR3>, <6>) +define(<CTR4>, <7>) +define(<CTR5>, <8>) +define(<CTR6>, <9>) +define(<CTR7>, <10>) + +define(<CTR0X>, <34>) +define(<CTR0SX>, <35>) +define(<CTR1X>, <36>) +define(<CTR2X>, <37>) +define(<CTR3X>, <38>) +define(<CTR4X>, <39>) +define(<CTR5X>, <40>) +define(<CTR6X>, <41>) +define(<CTR7X>, <42>) + +define(<I1>, <11>) +define(<I2>, <12>) +define(<I3>, <13>) +define(<I4>, <14>) +define(<I5>, <15>) +define(<I6>, <16>) +define(<I7>, <17>) +define(<I8>, <18>) + + .file "gcm-hash8.asm" + + # void gcm_init_key (union gcm_block *table) + + .text +.align 5 +PROLOGUE(_nettle_gcm_init_key8) + ld 7,.polynomial@got(TOCP) + lvx poly,0,7 + ld 7,.swap_mask@got(TOCP) + lvx swap_mask,0,7 + ld 7,.hidw_mask@got(TOCP) + lvx hidw_mask,0,7 + ld 7,.lodw_mask@got(TOCP) + lvx lodw_mask,0,7 + + li 10,0x800 + lxvd2x HX,10,TABLE # load H + vperm H,H,H,swap_mask + + # --- calculate H = H shift left 1 modulo polynomial --- + + vupkhsw msb,H # most significant bit word-extend + vspltisb sl1,1 # splat 1 for shift left + vspltw msb,msb,0 # most significant bit extend + vsl H,H,sl1 # H shift left 1 + vand msb,msb,poly + vxor zero,zero,zero + vxor H_t,H,msb + + vsldoi H,H_t,H_t,8 # doubleword swap + vsldoi Hh,H,zero,8 + vsldoi Hl,zero,H,8 + + # --- calculate H^2 = H*H --- + + # reduction pre-processing + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + # polynomial multiplication "classical" + vpmsumd H_h,H_t,Hh # H^1h*H^1h + vpmsumd H_l,H_t,Hl # H^1l*H^1l + vpmsumd H_m,H_t,H # H^1h*H^1l⊕H^1l*H^1h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] + vsldoi Ml,H_m,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor H_h,H_h,Mh # [2] + vxor H_l,H_l,Ml # [2] + vxor H_l,H_l,RP # [1] + + # reduction second phase + vpmsumd RP,H_l,poly_l + vxor H_h,H_l,H_h + vxor H2_t,H_h,RP + + vsldoi H2,H2_t,H2_t,8 + vsldoi H2h,H2,zero,8 + vsldoi H2l,zero,H2,8 + + # --- calculate [H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store H,[H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + li 8,0x00 + li 9,0x100 + li 10,0x200 + stxvd2x HlX,8,TABLE + stxvd2x HX,9,TABLE + stxvd2x HhX,10,TABLE + + li 8,0x300 + li 9,0x400 + li 10,0x500 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^3,H^4 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^2l + vpmsumd H_m,H_t,H2 # H^1h*H^2l⊕H^1l*H^2h + vpmsumd H_h,H_t,H2h # H^1h*H^2h + vpmsumd H2_l,H2_t,H2l # H^2l*H^2l + vpmsumd H2_m,H2_t,H2 # H^2h*H^2l⊕H^2l*H^2h + vpmsumd H2_h,H2_t,H2h # H^2h*H^2h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^3 + vpmsumd RP2,H2_l,poly_h # [1] H^4 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^3 + vsldoi M2h,zero,H2_m,8 # [2] H^4 + vsldoi Ml,H_m,zero,8 # [2] H^3 + vsldoi M2l,H2_m,zero,8 # [2] H^4 + vsldoi RP,RP,RP,8 # [1] H^3 + vsldoi RP2,RP2,RP2,8 # [1] H^4 + vxor H_h,H_h,Mh # [2] H^3 + vxor H2_h,H2_h,M2h # [2] H^4 + vxor H_l,H_l,Ml # [2] H^3 + vxor H2_l,H2_l,M2l # [2] H^4 + vxor H_l,H_l,RP # [1] H^3 + vxor H2_l,H2_l,RP2 # [1] H^4 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^3 + vpmsumd RP2,H2_l,poly_l # H^4 + vxor H_h,H_l,H_h # H^3 + vxor H2_h,H2_l,H2_h # H^4 + vxor H_h,H_h,RP # H^3 + vxor H2_h,H2_h,RP2 # H^4 + + vsldoi H2,H2_h,H2_h,8 # H^4 + vsldoi H,H_h,H_h,8 # H^3 + vsldoi H2l,zero,H2,8 # H^4 + vsldoi H2h,H2,zero,8 # H^4 + + # --- calculate [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + li 8,0x600 + li 9,0x700 + li 10,0x800 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^5,H^6 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^4l + vpmsumd H_m,H_t,H2 # H^1h*H^4l⊕H^1l*H^4h + vpmsumd H_h,H_t,H2h # H^1h*H^4h + vpmsumd H2_l,H2_t,H2l # H^2l*H^4l + vpmsumd H2_m,H2_t,H2 # H^2h*H^4l⊕H^2l*H^4h + vpmsumd H2_h,H2_t,H2h # H^2h*H^4h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^5 + vpmsumd RP2,H2_l,poly_h # [1] H^6 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^5 + vsldoi M2h,zero,H2_m,8 # [2] H^6 + vsldoi Ml,H_m,zero,8 # [2] H^5 + vsldoi M2l,H2_m,zero,8 # [2] H^6 + vsldoi RP,RP,RP,8 # [1] H^5 + vsldoi RP2,RP2,RP2,8 # [1] H^6 + vxor H_h,H_h,Mh # [2] H^5 + vxor H2_h,H2_h,M2h # [2] H^6 + vxor H_l,H_l,Ml # [2] H^5 + vxor H2_l,H2_l,M2l # [2] H^6 + vxor H_l,H_l,RP # [1] H^5 + vxor H2_l,H2_l,RP2 # [1] H^6 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^5 + vpmsumd RP2,H2_l,poly_l # H^6 + vxor H_h,H_l,H_h # H^5 + vxor H2_h,H2_l,H2_h # H^6 + vxor H_h,H_h,RP # H^5 + vxor H2_h,H2_h,RP2 # H^6 + + vsldoi H2,H2_h,H2_h,8 # H^6 + vsldoi H,H_h,H_h,8 # H^5 + vsldoi H2l,zero,H2,8 # H^6 + vsldoi H2h,H2,zero,8 # H^6 + + # --- calculate [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + li 8,0x900 + li 9,0xA00 + li 10,0xB00 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^7,H^8 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^6l + vpmsumd H_m,H_t,H2 # H^1h*H^6l⊕H^1l*H^6h + vpmsumd H_h,H_t,H2h # H^1h*H^6h + vpmsumd H2_l,H2_t,H2l # H^2l*H^6l + vpmsumd H2_m,H2_t,H2 # H^2h*H^6l⊕H^2l*H^6h + vpmsumd H2_h,H2_t,H2h # H^2h*H^6h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^7 + vpmsumd RP2,H2_l,poly_h # [1] H^8 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^7 + vsldoi M2h,zero,H2_m,8 # [2] H^8 + vsldoi Ml,H_m,zero,8 # [2] H^7 + vsldoi M2l,H2_m,zero,8 # [2] H^8 + vsldoi RP,RP,RP,8 # [1] H^7 + vsldoi RP2,RP2,RP2,8 # [1] H^8 + vxor H_h,H_h,Mh # [2] H^7 + vxor H2_h,H2_h,M2h # [2] H^8 + vxor H_l,H_l,Ml # [2] H^7 + vxor H2_l,H2_l,M2l # [2] H^8 + vxor H_l,H_l,RP # [1] H^7 + vxor H2_l,H2_l,RP2 # [1] H^8 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^7 + vpmsumd RP2,H2_l,poly_l # H^8 + vxor H_h,H_l,H_h # H^7 + vxor H2_h,H2_l,H2_h # H^8 + vxor H_h,H_h,RP # H^7 + vxor H2_h,H2_h,RP2 # H^8 + + vsldoi H,H_h,H_h,8 # H^7 + vsldoi H2,H2_h,H2_h,8 # H^8 + + # --- calculate [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + li 8,0xC00 + li 9,0xD00 + li 10,0xE00 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + blr +EPILOGUE(_nettle_gcm_init_key8) + + # void gcm_hash (const struct gcm_key *key, union gcm_block *x, + # size_t length, const uint8_t *data) + +.align 5 +PROLOGUE(_nettle_gcm_hash8) + vxor zero,zero,zero + + ld 7,.polynomial@got(TOCP) + lvx poly,0,7 + ld 7,.swap_mask@got(TOCP) + lvx swap_mask,0,7 + ld 7,.hidw_mask@got(TOCP) + lvx hidw_mask,0,7 + ld 7,.lodw_mask@got(TOCP) + lvx lodw_mask,0,7 + + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + lxvd2x CX,0,X # load X + vperm C,C,C,swap_mask + + srdi 7,LENGTH,7 # 8x loop count + cmpldi 7,0 + beq L2x + + # backup registers + stdu SP,-224(SP) + std 28,216(SP) + std 29,208(SP) + std 30,200(SP) + std 31,192(SP) + li 8,176 + stvx 20,8,SP + subi 8,8,16 + stvx 21,8,SP + subi 8,8,16 + stvx 22,8,SP + subi 8,8,16 + stvx 23,8,SP + subi 8,8,16 + stvx 24,8,SP + subi 8,8,16 + stvx 25,8,SP + subi 8,8,16 + stvx 26,8,SP + subi 8,8,16 + stvx 27,8,SP + subi 8,8,16 + stvx 28,8,SP + subi 8,8,16 + stvx 29,8,SP + subi 8,8,16 + stvx 30,8,SP + subi 8,8,16 + stvx 31,8,SP + + # table loading + li 8,0x300 + li 9,0x400 + li 10,0x500 + lxvd2x H21hX,8,TABLE + lxvd2x H21X,9,TABLE + lxvd2x H21lX,10,TABLE + li 8,0x600 + li 9,0x700 + li 10,0x800 + lxvd2x H43hX,8,TABLE + lxvd2x H43X,9,TABLE + lxvd2x H43lX,10,TABLE + li 8,0x900 + li 9,0xA00 + li 10,0xB00 + lxvd2x H65hX,8,TABLE + lxvd2x H65X,9,TABLE + lxvd2x H65lX,10,TABLE + li 8,0xC00 + li 9,0xD00 + li 10,0xE00 + lxvd2x H87hX,8,TABLE + lxvd2x H87X,9,TABLE + lxvd2x H87lX,10,TABLE + + li 8,0x10 + li 9,0x20 + li 10,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + + mtctr 7 +.align 5 +L8x_loop: + # input loading + lxvd2x C0X,0,DATA # load C0 + lxvd2x C1X,8,DATA # load C1 + lxvd2x C2X,9,DATA # load C2 + lxvd2x C3X,10,DATA # load C3 + + # swap permuting + vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask + vperm C2,C2,C2,swap_mask + vperm C3,C3,C3,swap_mask + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C23h,C2,C3,hidw_mask + vperm C23l,C2,C3,lodw_mask + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + + # input loading + lxvd2x C4X,28,DATA # load C4 + lxvd2x C5X,29,DATA # load C5 + lxvd2x C6X,30,DATA # load C6 + lxvd2x C7X,31,DATA # load C7 + + # swap permuting + vperm C4,C4,C4,swap_mask + vperm C5,C5,C5,swap_mask + vperm C6,C6,C6,swap_mask + vperm C7,C7,C7,swap_mask + + # polynomial multiplication "karatsuba" pre-processing + vperm C45h,C4,C5,hidw_mask + vperm C45l,C4,C5,lodw_mask + vperm C67h,C6,C7,hidw_mask + vperm C67l,C6,C7,lodw_mask + vxor C23,C23h,C23l + vxor C01,C01h,C01l + vxor C45,C45h,C45l + vxor C67,C67h,C67l + + # polynomial multiplication "karatsuba" + vpmsumd C23h,C23h,H65h # H23 = H^6h*C2h⊕H^5h*C3h + vpmsumd C23l,C23l,H65l # L23 = H^6l*C2l⊕H^5l*C3l + vpmsumd C01h,C01h,H87h # H01 = H^8h*C0h⊕H^7h*C1h + vpmsumd C01l,C01l,H87l # L01 = H^8l*C0l⊕H^7l*C1l + vpmsumd C67h,C67h,H21h # H67 = H^2h*C6h⊕H^1h*C7h + vpmsumd C67l,C67l,H21l # L67 = H^2l*C6l⊕H^1l*C7l + vpmsumd C45h,C45h,H43h # H45 = H^4h*C4h⊕H^3h*C5h + vpmsumd C45l,C45l,H43l # L45 = H^4l*C4l⊕H^3l*C5l + vpmsumd C23,C23,H65 # M23 = (H^6h⊕H^5h)*(C2h⊕C3h)⊕(H^6l⊕H^5l)*(C2l⊕C3l) + vpmsumd C01,C01,H87 # M01 = (H^8h⊕H^7h)*(C0h⊕C1h)⊕(H^8l⊕H^7l)*(C0l⊕C1l) + vpmsumd C45,C45,H43 # M45 = (H^4h⊕H^3h)*(C4h⊕C5h)⊕(H^4l⊕H^3l)*(C4l⊕C5l) + vpmsumd C67,C67,H21 # M67 = (H^2h⊕H^1h)*(C6h⊕C7h)⊕(H^2l⊕H^1l)*(C6l⊕C7l) + + # polynomial multiplication "karatsuba" post-processing + vxor C23,C23,C23h + vxor C01,C01,C01h + vxor C45,C45,C45h + vxor C67,C67,C67h + vxor C23,C23,C23l + vxor C01,C01,C01l + vxor C45,C45,C45l + vxor C67,C67,C67l + + # deferred recombination of partial products + vxor C01h,C01h,C23h # H0 = H01⊕H23 + vxor C45h,C45h,C67h # H1 = H45⊕H67 + vxor C01l,C01l,C23l # L0 = L01⊕L23 + vxor C45l,C45l,C67l # L1 = L45⊕L45 + vxor C01,C01,C23 # M0 = M01⊕M23 + vxor C45,C45,C67 # M1 = M45⊕M45 + vxor C01h,C01h,C45h # H = H0⊕H1 + vxor C01l,C01l,C45l # L = L0⊕L1 + vxor C01,C01,C45 # M = M0⊕M1 + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x80 + bdnz L8x_loop + + # restore registers + li 8,0 + lvx 31,8,SP + addi 8,8,16 + lvx 30,8,SP + addi 8,8,16 + lvx 29,8,SP + addi 8,8,16 + lvx 28,8,SP + addi 8,8,16 + lvx 27,8,SP + addi 8,8,16 + lvx 26,8,SP + addi 8,8,16 + lvx 25,8,SP + addi 8,8,16 + lvx 24,8,SP + addi 8,8,16 + lvx 23,8,SP + addi 8,8,16 + lvx 22,8,SP + addi 8,8,16 + lvx 21,8,SP + addi 8,8,16 + lvx 20,8,SP + ld 31,192(SP) + ld 30,200(SP) + ld 29,208(SP) + ld 28,216(SP) + addi SP,SP,224 + + clrldi LENGTH,LENGTH,57 +L2x: + srdi 7,LENGTH,5 + cmpldi 7,0 + beq L1x + + # table loading + li 8,0x300 + li 9,0x400 + li 10,0x500 + lxvd2x H21hX,8,TABLE + lxvd2x H21X,9,TABLE + lxvd2x H21lX,10,TABLE + + li 10,0x10 + + mtctr 7 +.align 5 +L2x_loop: + # input loading + lxvd2x C0X,0,DATA # load C0 + lxvd2x C1X,10,DATA # load C1 + + # swap permuting + vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + vxor C01,C01h,C01l + + # polynomial multiplication "karatsuba" + vpmsumd C01h,C01h,H21h # H01 = H^2h*C0h⊕H^1h*C1h + vpmsumd C01l,C01l,H21l # L01 = H^2l*C0l⊕H^1l*C1l + vpmsumd C01,C01,H21 # M01 = (H^2h⊕H^1h)*(C0h⊕C1h)⊕(H^2l⊕H^1l)*(C0l⊕C1l) + + # polynomial multiplication "karatsuba" post-processing + vxor C01,C01,C01h + vxor C01,C01,C01l + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x20 + bdnz L2x_loop + + clrldi LENGTH,LENGTH,59 +L1x: + srdi 7,LENGTH,4 + cmpldi 7,0 + beq Lrem + + # table loading + li 9,0x100 + li 10,0x200 + lxvd2x HlX,0,TABLE + lxvd2x HX, 9,TABLE + lxvd2x HhX,10,TABLE + + # input loading + lxvd2x C0X,0,DATA # load C0 + + # swap permuting + vperm C0,C0,C0,swap_mask + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + + addi DATA,DATA,0x10 + clrldi LENGTH,LENGTH,60 +Lrem: + cmpldi LENGTH,0 + beq Ldone + + # table loading + li 9,0x100 + li 10,0x200 + lxvd2x HlX,0,TABLE + lxvd2x HX, 9,TABLE + lxvd2x HhX,10,TABLE + + # input loading + stdu SP,-16(SP) + stvx zero,0,SP +Lst_loop: + subic. LENGTH,LENGTH,1 + lbzx 7,LENGTH,DATA + stbx 7,LENGTH,SP + bne Lst_loop + lxvd2x C0X,0,SP + addi SP,SP,16 + + # swap permuting + vperm C0,C0,C0,swap_mask + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + +Ldone: + vperm C,C,C,swap_mask + stxvd2x CX,0,X # store C + blr +EPILOGUE(_nettle_gcm_hash8) + + # gcm_fill (uint8_t *ctr, size_t blocks, union gcm_block *buffer) + +.align 5 +PROLOGUE(_nettle_gcm_fill) + ld 6,.swap_mask@got(TOCP) + lvx swap_mask,0,6 + + vxor zero,zero,zero + vspltisb I1,1 + vspltisb I2,2 + vspltisb I3,3 + vspltisb I4,4 + vspltisb I5,5 + vspltisb I6,6 + vspltisb I7,7 + vspltisb I8,8 + vsldoi I1,zero,I1,1 + vsldoi I2,zero,I2,1 + vsldoi I3,zero,I3,1 + vsldoi I4,zero,I4,1 + vsldoi I5,zero,I5,1 + vsldoi I6,zero,I6,1 + vsldoi I7,zero,I7,1 + vsldoi I8,zero,I8,1 + + lxvd2x CTR0X,0,CTR + vperm CTR0,CTR0,CTR0,swap_mask + + srdi 6,BLOCKS,3 # 8x loop count + cmpldi 6,0 + beq Lfill_4x + + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 25,0x10 + li 26,0x20 + li 27,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + + mtctr 6 +L8x_fill_loop: + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + vadduwm CTR4,CTR0,I4 + vadduwm CTR5,CTR0,I5 + vadduwm CTR6,CTR0,I6 + vadduwm CTR7,CTR0,I7 + + vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask + vperm CTR4,CTR4,CTR4,swap_mask + vperm CTR5,CTR5,CTR5,swap_mask + vperm CTR6,CTR6,CTR6,swap_mask + vperm CTR7,CTR7,CTR7,swap_mask + + stxvd2x CTR0SX,0,BUFFER + stxvd2x CTR1X,25,BUFFER + stxvd2x CTR2X,26,BUFFER + stxvd2x CTR3X,27,BUFFER + stxvd2x CTR4X,28,BUFFER + stxvd2x CTR5X,29,BUFFER + stxvd2x CTR6X,30,BUFFER + stxvd2x CTR7X,31,BUFFER + + vadduwm CTR0,CTR0,I8 + addi BUFFER,BUFFER,0x80 + bdnz L8x_fill_loop + + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi BLOCKS,BLOCKS,61 + +Lfill_4x: + srdi 6,BLOCKS,2 + cmpldi 6,0 + beq Lfill_2x + + li 8,0x10 + li 9,0x20 + li 10,0x30 + + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + + vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask + + stxvd2x CTR0SX,0,BUFFER + stxvd2x CTR1X,8,BUFFER + stxvd2x CTR2X,9,BUFFER + stxvd2x CTR3X,10,BUFFER + + vadduwm CTR0,CTR0,I4 + addi BUFFER,BUFFER,0x40 + + clrldi BLOCKS,BLOCKS,62 + +Lfill_2x: + srdi 6,BLOCKS,1 + cmpldi 6,0 + beq Lfill_1x + + li 10,0x10 + + vadduwm CTR1,CTR0,I1 + + vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + + stxvd2x CTR0SX,0,BUFFER + stxvd2x CTR1X,10,BUFFER + + vadduwm CTR0,CTR0,I2 + addi BUFFER,BUFFER,0x20 + + clrldi BLOCKS,BLOCKS,63 + +Lfill_1x: + cmpldi BLOCKS,0 + beq Lfill_done + + vperm CTR0S,CTR0,CTR0,swap_mask + + stxvd2x CTR0SX,0,BUFFER + + vadduwm CTR0,CTR0,I1 + +Lfill_done: + vperm CTR0,CTR0,CTR0,swap_mask + stxvd2x CTR0X,0,CTR + + blr +EPILOGUE(_nettle_gcm_fill) + + .data + .align 4 +.polynomial: + .byte 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0xc2 + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 + .align 4 +.hidw_mask: + .byte 23,22,21,20,19,18,17,16,7,6,5,4,3,2,1,0 + .align 4 +.lodw_mask: + .byte 31,30,29,28,27,26,25,24,15,14,13,12,11,10,9,8 diff --git a/powerpc64le/machine.m4 b/powerpc64le/machine.m4 new file mode 100644 index 00000000..e69de29b diff --git a/testsuite/gcm-test.c b/testsuite/gcm-test.c index c8174019..df1fc94a 100644 --- a/testsuite/gcm-test.c +++ b/testsuite/gcm-test.c @@ -170,6 +170,29 @@ test_main(void) "16aedbf5a0de6a57a637b39b"), SHEX("619cc5aefffe0bfa462af43c1699d050")); + /* Test 128 bytes */ + test_aead(&nettle_gcm_aes128, NULL, + SHEX("feffe9928665731c6d6a8f9467308308"), + SHEX(""), + SHEX("d9313225f88406e5a55909c5aff5269a" + "86a7a9531534f7da2e4c303d8a318a72" + "1c3c0c95956809532fcf0e2449a6b525" + "b16aedf5aa0de657ba637b391aafd255" + "5ae376bc5e9f6a1b08e34db7a6ee0736" + "9ba662ea12f6f197e6bc3ed69d2480f3" + "ea5691347f2ba69113eb37910ebc18c8" + "0f697234582016fa956ca8f63ae6b473"), + SHEX("42831ec2217774244b7221b784d0d49c" + "e3aa212f2c02a4e035c17e2329aca12e" + "21d514b25466931c7d8f6a5aac84aa05" + "1ba30b396a0aac973d58e091473f5985" + "874b1178906ddbeab04ab2fe6cce8c57" + "8d7e961bd13fd6a8c56b66ca5e576492" + "1a48cd8bda04e66343e73055118b69b9" + "ced486813846958a11e602c03cfc232b"), + SHEX("cafebabefacedbaddecaf888"), + SHEX("796836f1246c9d735c5e1be0a715ccc3")); + /* Test case 7 */ test_aead(&nettle_gcm_aes192, NULL, SHEX("00000000000000000000000000000000"

1 0

[PATCH] optimizing AES and GHASH for PPC64LE
by Maamoun TK 19 Jun '20

19 Jun '20

--- Makefile.in | 2 +- configure.ac | 5 + gcm.c | 19 +- powerpc64le/aes-decrypt-internal.asm | 573 +++++++++++++++ powerpc64le/aes-encrypt-internal.asm | 534 ++++++++++++++ powerpc64le/gcm-hash8.asm | 992 ++++++++++++++++++++++++++ powerpc64le/machine.m4 | 0 testsuite/gcm-test.c | 23 + 8 files changed, 2146 insertions(+), 2 deletions(-) create mode 100644 powerpc64le/aes-decrypt-internal.asm create mode 100644 powerpc64le/aes-encrypt-internal.asm create mode 100644 powerpc64le/gcm-hash8.asm create mode 100644 powerpc64le/machine.m4 diff --git a/Makefile.in b/Makefile.in index 64ff1001..5bbc0f79 100644 --- a/Makefile.in +++ b/Makefile.in @@ -603,7 +603,7 @@ distdir: $(DISTFILES) done set -e; for d in sparc32 sparc64 x86 \ x86_64 x86_64/aesni x86_64/sha_ni x86_64/fat \ - arm arm/neon arm/v6 arm/fat ; do \ + arm arm/neon arm/v6 arm/fat powerpc64le ; do \ mkdir "$(distdir)/$$d" ; \ find "$(srcdir)/$$d" -maxdepth 1 '(' -name '*.asm' -o -name '*.m4' ')' \ -exec cp '{}' "$(distdir)/$$d" ';' ; \ diff --git a/configure.ac b/configure.ac index 90ea1ea8..1ea54ce8 100644 --- a/configure.ac +++ b/configure.ac @@ -435,6 +435,9 @@ if test "x$enable_assembler" = xyes ; then esac fi ;; + *powerpc64le*) + asm_path=powerpc64le + ;; *) enable_assembler=no ;; @@ -572,7 +575,9 @@ AH_VERBATIM([HAVE_NATIVE], #undef HAVE_NATIVE_ecc_secp384r1_redc #undef HAVE_NATIVE_ecc_secp521r1_modp #undef HAVE_NATIVE_ecc_secp521r1_redc +#undef HAVE_NATIVE_gcm_init_key8 #undef HAVE_NATIVE_gcm_hash8 +#undef HAVE_NATIVE_gcm_fill #undef HAVE_NATIVE_salsa20_core #undef HAVE_NATIVE_sha1_compress #undef HAVE_NATIVE_sha256_compress diff --git a/gcm.c b/gcm.c index cf615daf..809c03bc 100644 --- a/gcm.c +++ b/gcm.c @@ -140,6 +140,12 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) memcpy (x->b, Z.b, sizeof(Z)); } # elif GCM_TABLE_BITS == 8 +# if HAVE_NATIVE_gcm_init_key8 + +#define gcm_init_key _nettle_gcm_init_key8 +void +_nettle_gcm_init_key8 (union nettle_block16 *table); +# endif /* HAVE_NATIVE_gcm_init_key8 */ # if HAVE_NATIVE_gcm_hash8 #define gcm_hash _nettle_gcm_hash8 @@ -225,6 +231,13 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) #endif /* GCM_TABLE_BITS */ +#if HAVE_NATIVE_gcm_fill + +#define gcm_fill _nettle_gcm_fill +void +_nettle_gcm_fill (uint8_t *ctr, size_t blocks, union nettle_block16 *buffer); +#endif /* HAVE_NATIVE_gcm_fill */ + /* Increment the rightmost 32 bits. */ #define INC32(block) INCREMENT(4, (block.b) + GCM_BLOCK_SIZE - 4) @@ -245,7 +258,9 @@ gcm_set_key(struct gcm_key *key, memset(key->h[0].b, 0, GCM_BLOCK_SIZE); f (cipher, GCM_BLOCK_SIZE, key->h[i].b, key->h[0].b); -#if GCM_TABLE_BITS +#ifdef gcm_init_key + gcm_init_key(key->h); +#elif GCM_TABLE_BITS /* Algorithm 3 from the gcm paper. First do powers of two, then do the rest by adding. */ while (i /= 2) @@ -333,6 +348,7 @@ gcm_update(struct gcm_ctx *ctx, const struct gcm_key *key, ctx->auth_size += length; } +#ifndef gcm_fill static nettle_fill16_func gcm_fill; static void gcm_fill(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) @@ -349,6 +365,7 @@ gcm_fill(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) WRITE_UINT32(ctr + GCM_BLOCK_SIZE - 4, c); } +#endif /* !gcm_fill */ void gcm_encrypt (struct gcm_ctx *ctx, const struct gcm_key *key, diff --git a/powerpc64le/aes-decrypt-internal.asm b/powerpc64le/aes-decrypt-internal.asm new file mode 100644 index 00000000..bde34779 --- /dev/null +++ b/powerpc64le/aes-decrypt-internal.asm @@ -0,0 +1,573 @@ +C powerpc64le/aes-decrypt-internal.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<ROUNDS>, <3>) +define(<KEYS>, <4>) +define(<LENGTH>, <6>) +define(<DST>, <7>) +define(<SRC>, <8>) + +define(<swap_mask>, <0>) + +define(<K>, <1>) +define(<S0>, <2>) +define(<S1>, <3>) +define(<S2>, <4>) +define(<S3>, <5>) +define(<S4>, <6>) +define(<S5>, <7>) +define(<S6>, <8>) +define(<S7>, <9>) +define(<S8>, <10>) +define(<S9>, <11>) +define(<S10>, <12>) +define(<S11>, <13>) +define(<S12>, <14>) +define(<S13>, <15>) +define(<S14>, <16>) +define(<S15>, <17>) + +define(<KX>, <33>) +define(<S0X>, <34>) +define(<S1X>, <35>) +define(<S2X>, <36>) +define(<S3X>, <37>) +define(<S4X>, <38>) +define(<S5X>, <39>) +define(<S6X>, <40>) +define(<S7X>, <41>) +define(<S8X>, <42>) +define(<S9X>, <43>) +define(<S10X>, <44>) +define(<S11X>, <45>) +define(<S12X>, <46>) +define(<S13X>, <47>) +define(<S14X>, <48>) +define(<S15X>, <49>) + +C ZERO vector register is used in place of RoundKey +C for vncipher instruction because the order of InvMixColumns +C and Xor processes are flipped in that instruction. +C The Xor process with RoundKey is executed afterward. +define(<ZERO>, <18>) + + .file "aes-decrypt-internal.asm" + + C _aes_decrypt(unsigned rounds, const uint32_t *keys, + C const struct aes_table *T, + C size_t length, uint8_t *dst, + C uint8_t *src) + + .text +.align 5 +PROLOGUE(_nettle_aes_decrypt) + vxor ZERO,ZERO,ZERO + + ld 5,.swap_mask@got(TOCP) + lvx swap_mask,0,5 + + subi ROUNDS,ROUNDS,1 + srdi LENGTH,LENGTH,4 + + srdi 5,LENGTH,4 # 16x loop count + cmpldi 5,0 + beq L8x + + std 17,-120(SP); + std 18,-112(SP); + std 19,-104(SP); + std 20,-96(SP); + std 21,-88(SP); + std 22,-80(SP); + std 23,-72(SP); + std 24,-64(SP); + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 17,0x10 + li 18,0x20 + li 19,0x30 + li 20,0x40 + li 21,0x50 + li 22,0x60 + li 23,0x70 + li 24,0x80 + li 25,0x90 + li 26,0xA0 + li 27,0xB0 + li 28,0xC0 + li 29,0xD0 + li 30,0xE0 + li 31,0xF0 + +.align 5 +Lx16_loop: + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + lxvd2x S1X,17,SRC + lxvd2x S2X,18,SRC + lxvd2x S3X,19,SRC + lxvd2x S4X,20,SRC + lxvd2x S5X,21,SRC + lxvd2x S6X,22,SRC + lxvd2x S7X,23,SRC + lxvd2x S8X,24,SRC + lxvd2x S9X,25,SRC + lxvd2x S10X,26,SRC + lxvd2x S11X,27,SRC + lxvd2x S12X,28,SRC + lxvd2x S13X,29,SRC + lxvd2x S14X,30,SRC + lxvd2x S15X,31,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L16x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vncipher S4,S4,ZERO + vncipher S5,S5,ZERO + vncipher S6,S6,ZERO + vncipher S7,S7,ZERO + vncipher S8,S8,ZERO + vncipher S9,S9,ZERO + vncipher S10,S10,ZERO + vncipher S11,S11,ZERO + vncipher S12,S12,ZERO + vncipher S13,S13,ZERO + vncipher S14,S14,ZERO + vncipher S15,S15,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + addi 10,10,0x10 + bdnz L16x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + vncipherlast S4,S4,K + vncipherlast S5,S5,K + vncipherlast S6,S6,K + vncipherlast S7,S7,K + vncipherlast S8,S8,K + vncipherlast S9,S9,K + vncipherlast S10,S10,K + vncipherlast S11,S11,K + vncipherlast S12,S12,K + vncipherlast S13,S13,K + vncipherlast S14,S14,K + vncipherlast S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + stxvd2x S0X,0,DST + stxvd2x S1X,17,DST + stxvd2x S2X,18,DST + stxvd2x S3X,19,DST + stxvd2x S4X,20,DST + stxvd2x S5X,21,DST + stxvd2x S6X,22,DST + stxvd2x S7X,23,DST + stxvd2x S8X,24,DST + stxvd2x S9X,25,DST + stxvd2x S10X,26,DST + stxvd2x S11X,27,DST + stxvd2x S12X,28,DST + stxvd2x S13X,29,DST + stxvd2x S14X,30,DST + stxvd2x S15X,31,DST + + addi SRC,SRC,0x100 + addi DST,DST,0x100 + subic. 5,5,1 + bne Lx16_loop + + ld 17,-120(SP); + ld 18,-112(SP); + ld 19,-104(SP); + ld 20,-96(SP); + ld 21,-88(SP); + ld 22,-80(SP); + ld 23,-72(SP); + ld 24,-64(SP); + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi LENGTH,LENGTH,60 + +L8x: + srdi 5,LENGTH,3 + cmpldi 5,0 + beq L4x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + addi 9,9,0x10 + lxvd2x S4X,9,SRC + addi 9,9,0x10 + lxvd2x S5X,9,SRC + addi 9,9,0x10 + lxvd2x S6X,9,SRC + addi 9,9,0x10 + lxvd2x S7X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L8x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vncipher S4,S4,ZERO + vncipher S5,S5,ZERO + vncipher S6,S6,ZERO + vncipher S7,S7,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + addi 10,10,0x10 + bdnz L8x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + vncipherlast S4,S4,K + vncipherlast S5,S5,K + vncipherlast S6,S6,K + vncipherlast S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + addi 9,9,0x10 + stxvd2x S4X,9,DST + addi 9,9,0x10 + stxvd2x S5X,9,DST + addi 9,9,0x10 + stxvd2x S6X,9,DST + addi 9,9,0x10 + stxvd2x S7X,9,DST + + addi SRC,SRC,0x80 + addi DST,DST,0x80 + + clrldi LENGTH,LENGTH,61 + +L4x: + srdi 5,LENGTH,2 + cmpldi 5,0 + beq L2x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L4x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + addi 10,10,0x10 + bdnz L4x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + + addi SRC,SRC,0x40 + addi DST,DST,0x40 + + clrldi LENGTH,LENGTH,62 + +L2x: + srdi 5,LENGTH,1 + cmpldi 5,0 + beq L1x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L2x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vxor S0,S0,K + vxor S1,S1,K + addi 10,10,0x10 + bdnz L2x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + + addi SRC,SRC,0x20 + addi DST,DST,0x20 + + clrldi LENGTH,LENGTH,63 + +L1x: + cmpldi LENGTH,0 + beq Ldone + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + + vxor S0,S0,K + + vperm S0,S0,S0,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L1x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vxor S0,S0,K + addi 10,10,0x10 + bdnz L1x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + + vperm S0,S0,S0,swap_mask + + stxvd2x S0X,0,DST + +Ldone: + blr +EPILOGUE(_nettle_aes_decrypt) + + .data + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 diff --git a/powerpc64le/aes-encrypt-internal.asm b/powerpc64le/aes-encrypt-internal.asm new file mode 100644 index 00000000..1bbd86a8 --- /dev/null +++ b/powerpc64le/aes-encrypt-internal.asm @@ -0,0 +1,534 @@ +C powerpc64le/aes-encrypt-internal.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<ROUNDS>, <3>) +define(<KEYS>, <4>) +define(<LENGTH>, <6>) +define(<DST>, <7>) +define(<SRC>, <8>) + +define(<swap_mask>, <0>) + +define(<K>, <1>) +define(<S0>, <2>) +define(<S1>, <3>) +define(<S2>, <4>) +define(<S3>, <5>) +define(<S4>, <6>) +define(<S5>, <7>) +define(<S6>, <8>) +define(<S7>, <9>) +define(<S8>, <10>) +define(<S9>, <11>) +define(<S10>, <12>) +define(<S11>, <13>) +define(<S12>, <14>) +define(<S13>, <15>) +define(<S14>, <16>) +define(<S15>, <17>) + +define(<KX>, <33>) +define(<S0X>, <34>) +define(<S1X>, <35>) +define(<S2X>, <36>) +define(<S3X>, <37>) +define(<S4X>, <38>) +define(<S5X>, <39>) +define(<S6X>, <40>) +define(<S7X>, <41>) +define(<S8X>, <42>) +define(<S9X>, <43>) +define(<S10X>, <44>) +define(<S11X>, <45>) +define(<S12X>, <46>) +define(<S13X>, <47>) +define(<S14X>, <48>) +define(<S15X>, <49>) + + .file "aes-encrypt-internal.asm" + + C _aes_encrypt(unsigned rounds, const uint32_t *keys, + C const struct aes_table *T, + C size_t length, uint8_t *dst, + C uint8_t *src) + + .text +.align 5 +PROLOGUE(_nettle_aes_encrypt) + ld 5,.swap_mask@got(TOCP) + lvx swap_mask,0,5 + + subi ROUNDS,ROUNDS,1 + srdi LENGTH,LENGTH,4 + + srdi 5,LENGTH,4 # 16x loop count + cmpldi 5,0 + beq L8x + + std 17,-120(SP); + std 18,-112(SP); + std 19,-104(SP); + std 20,-96(SP); + std 21,-88(SP); + std 22,-80(SP); + std 23,-72(SP); + std 24,-64(SP); + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 17,0x10 + li 18,0x20 + li 19,0x30 + li 20,0x40 + li 21,0x50 + li 22,0x60 + li 23,0x70 + li 24,0x80 + li 25,0x90 + li 26,0xA0 + li 27,0xB0 + li 28,0xC0 + li 29,0xD0 + li 30,0xE0 + li 31,0xF0 + +.align 5 +Lx16_loop: + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + lxvd2x S1X,17,SRC + lxvd2x S2X,18,SRC + lxvd2x S3X,19,SRC + lxvd2x S4X,20,SRC + lxvd2x S5X,21,SRC + lxvd2x S6X,22,SRC + lxvd2x S7X,23,SRC + lxvd2x S8X,24,SRC + lxvd2x S9X,25,SRC + lxvd2x S10X,26,SRC + lxvd2x S11X,27,SRC + lxvd2x S12X,28,SRC + lxvd2x S13X,29,SRC + lxvd2x S14X,30,SRC + lxvd2x S15X,31,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L16x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + vcipher S4,S4,K + vcipher S5,S5,K + vcipher S6,S6,K + vcipher S7,S7,K + vcipher S8,S8,K + vcipher S9,S9,K + vcipher S10,S10,K + vcipher S11,S11,K + vcipher S12,S12,K + vcipher S13,S13,K + vcipher S14,S14,K + vcipher S15,S15,K + addi 10,10,0x10 + bdnz L16x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + vcipherlast S4,S4,K + vcipherlast S5,S5,K + vcipherlast S6,S6,K + vcipherlast S7,S7,K + vcipherlast S8,S8,K + vcipherlast S9,S9,K + vcipherlast S10,S10,K + vcipherlast S11,S11,K + vcipherlast S12,S12,K + vcipherlast S13,S13,K + vcipherlast S14,S14,K + vcipherlast S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + stxvd2x S0X,0,DST + stxvd2x S1X,17,DST + stxvd2x S2X,18,DST + stxvd2x S3X,19,DST + stxvd2x S4X,20,DST + stxvd2x S5X,21,DST + stxvd2x S6X,22,DST + stxvd2x S7X,23,DST + stxvd2x S8X,24,DST + stxvd2x S9X,25,DST + stxvd2x S10X,26,DST + stxvd2x S11X,27,DST + stxvd2x S12X,28,DST + stxvd2x S13X,29,DST + stxvd2x S14X,30,DST + stxvd2x S15X,31,DST + + addi SRC,SRC,0x100 + addi DST,DST,0x100 + subic. 5,5,1 + bne Lx16_loop + + ld 17,-120(SP); + ld 18,-112(SP); + ld 19,-104(SP); + ld 20,-96(SP); + ld 21,-88(SP); + ld 22,-80(SP); + ld 23,-72(SP); + ld 24,-64(SP); + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi LENGTH,LENGTH,60 + +L8x: + srdi 5,LENGTH,3 + cmpldi 5,0 + beq L4x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + addi 9,9,0x10 + lxvd2x S4X,9,SRC + addi 9,9,0x10 + lxvd2x S5X,9,SRC + addi 9,9,0x10 + lxvd2x S6X,9,SRC + addi 9,9,0x10 + lxvd2x S7X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L8x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + vcipher S4,S4,K + vcipher S5,S5,K + vcipher S6,S6,K + vcipher S7,S7,K + addi 10,10,0x10 + bdnz L8x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + vcipherlast S4,S4,K + vcipherlast S5,S5,K + vcipherlast S6,S6,K + vcipherlast S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + addi 9,9,0x10 + stxvd2x S4X,9,DST + addi 9,9,0x10 + stxvd2x S5X,9,DST + addi 9,9,0x10 + stxvd2x S6X,9,DST + addi 9,9,0x10 + stxvd2x S7X,9,DST + + addi SRC,SRC,0x80 + addi DST,DST,0x80 + + clrldi LENGTH,LENGTH,61 + +L4x: + srdi 5,LENGTH,2 + cmpldi 5,0 + beq L2x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L4x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + addi 10,10,0x10 + bdnz L4x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + + addi SRC,SRC,0x40 + addi DST,DST,0x40 + + clrldi LENGTH,LENGTH,62 + +L2x: + srdi 5,LENGTH,1 + cmpldi 5,0 + beq L1x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L2x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + addi 10,10,0x10 + bdnz L2x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + + addi SRC,SRC,0x20 + addi DST,DST,0x20 + + clrldi LENGTH,LENGTH,63 + +L1x: + cmpldi LENGTH,0 + beq Ldone + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + + vxor S0,S0,K + + vperm S0,S0,S0,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L1x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + addi 10,10,0x10 + bdnz L1x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + + vperm S0,S0,S0,swap_mask + + stxvd2x S0X,0,DST + +Ldone: + blr +EPILOGUE(_nettle_aes_encrypt) + + .data + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 diff --git a/powerpc64le/gcm-hash8.asm b/powerpc64le/gcm-hash8.asm new file mode 100644 index 00000000..a809f6ef --- /dev/null +++ b/powerpc64le/gcm-hash8.asm @@ -0,0 +1,992 @@ +C powerpc64le/gcm-hash8.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: +C VSX instructions is used to load and store data to memory "lxvd2x, stxvd2x" +C instead of VR instructions "lvx, stvx" as a workaround to access unaligned data +C VSX registers are defined with "X" suffix + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<TABLE>, <3>) +define(<X>, <4>) +define(<LENGTH>, <5>) +define(<DATA>, <6>) + +define(<zero>, <0>) +define(<swap_mask>, <1>) +define(<hidw_mask>, <2>) +define(<lodw_mask>, <3>) +define(<poly>, <4>) +define(<poly_h>, <4>) +define(<poly_l>, <5>) +define(<RP>, <6>) +define(<Mh>, <7>) +define(<Ml>, <8>) +define(<H>, <9>) +define(<Hh>, <10>) +define(<Hl>, <11>) +define(<RP2>, <9>) +define(<M2h>, <10>) +define(<M2l>, <11>) + +define(<HX>, <41>) +define(<HhX>, <42>) +define(<HlX>, <43>) +define(<H_HhX>, <44>) +define(<H_HX>, <45>) +define(<H_HlX>, <46>) + +define(<sl1>, <1>) +define(<msb>, <5>) +define(<H2>, <6>) +define(<H2h>, <7>) +define(<H2l>, <8>) +define(<H_h>, <12>) +define(<H_m>, <13>) +define(<H_l>, <14>) +define(<H_Hh>, <12>) +define(<H_H>, <13>) +define(<H_Hl>, <14>) +define(<H_t>, <15>) +define(<H2_h>, <16>) +define(<H2_m>, <17>) +define(<H2_l>, <18>) +define(<H2_t>, <19>) + +define(<C0X>, <38>) +define(<C1X>, <39>) +define(<C2X>, <40>) +define(<C3X>, <44>) +define(<C4X>, <38>) +define(<C5X>, <39>) +define(<C6X>, <40>) +define(<C7X>, <44>) + +define(<CX>, <45>) + +define(<C0>, <6>) +define(<C1>, <7>) +define(<C2>, <8>) +define(<C3>, <12>) +define(<C4>, <6>) +define(<C5>, <7>) +define(<C6>, <8>) +define(<C7>, <12>) + +define(<C>, <13>) + +define(<Ch>, <14>) +define(<Cl>, <15>) +define(<Cm>, <16>) + +define(<C01h>, <14>) +define(<C01l>, <15>) +define(<C01>, <16>) +define(<C23h>, <17>) +define(<C23l>, <18>) +define(<C23>, <19>) +define(<C45h>, <20>) +define(<C45l>, <21>) +define(<C45>, <22>) +define(<C67h>, <6>) +define(<C67l>, <7>) +define(<C67>, <8>) + +define(<H21>, <9>) +define(<H21h>, <10>) +define(<H21l>, <11>) +define(<H43>, <23>) +define(<H43h>, <24>) +define(<H43l>, <25>) +define(<H65>, <26>) +define(<H65h>, <27>) +define(<H65l>, <28>) +define(<H87>, <29>) +define(<H87h>, <30>) +define(<H87l>, <31>) + +define(<H21X>, <41>) +define(<H21hX>, <42>) +define(<H21lX>, <43>) +define(<H43X>, <55>) +define(<H43hX>, <56>) +define(<H43lX>, <57>) +define(<H65X>, <58>) +define(<H65hX>, <59>) +define(<H65lX>, <60>) +define(<H87X>, <61>) +define(<H87hX>, <62>) +define(<H87lX>, <63>) + +# gcm_fill registers: + +define(<CTR>, <3>) +define(<BLOCKS>, <4>) +define(<BUFFER>, <5>) + +define(<CTR0>, <2>) +define(<CTR0S>, <3>) +define(<CTR1>, <4>) +define(<CTR2>, <5>) +define(<CTR3>, <6>) +define(<CTR4>, <7>) +define(<CTR5>, <8>) +define(<CTR6>, <9>) +define(<CTR7>, <10>) + +define(<CTR0X>, <34>) +define(<CTR0SX>, <35>) +define(<CTR1X>, <36>) +define(<CTR2X>, <37>) +define(<CTR3X>, <38>) +define(<CTR4X>, <39>) +define(<CTR5X>, <40>) +define(<CTR6X>, <41>) +define(<CTR7X>, <42>) + +define(<I1>, <11>) +define(<I2>, <12>) +define(<I3>, <13>) +define(<I4>, <14>) +define(<I5>, <15>) +define(<I6>, <16>) +define(<I7>, <17>) +define(<I8>, <18>) + + .file "gcm-hash8.asm" + + # void gcm_init_key (union gcm_block *table) + + .text +.align 5 +PROLOGUE(_nettle_gcm_init_key8) + ld 7,.polynomial@got(TOCP) + lvx poly,0,7 + ld 7,.swap_mask@got(TOCP) + lvx swap_mask,0,7 + ld 7,.hidw_mask@got(TOCP) + lvx hidw_mask,0,7 + ld 7,.lodw_mask@got(TOCP) + lvx lodw_mask,0,7 + + li 10,0x800 + lxvd2x HX,10,TABLE # load H + vperm H,H,H,swap_mask + + # --- calculate H = H shift left 1 modulo polynomial --- + + vupkhsw msb,H # most significant bit word-extend + vspltisb sl1,1 # splat 1 for shift left + vspltw msb,msb,0 # most significant bit extend + vsl H,H,sl1 # H shift left 1 + vand msb,msb,poly + vxor zero,zero,zero + vxor H_t,H,msb + + vsldoi H,H_t,H_t,8 # doubleword swap + vsldoi Hh,H,zero,8 + vsldoi Hl,zero,H,8 + + # --- calculate H^2 = H*H --- + + # reduction pre-processing + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + # polynomial multiplication "classical" + vpmsumd H_h,H_t,Hh # H^1h*H^1h + vpmsumd H_l,H_t,Hl # H^1l*H^1l + vpmsumd H_m,H_t,H # H^1h*H^1l⊕H^1l*H^1h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] + vsldoi Ml,H_m,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor H_h,H_h,Mh # [2] + vxor H_l,H_l,Ml # [2] + vxor H_l,H_l,RP # [1] + + # reduction second phase + vpmsumd RP,H_l,poly_l + vxor H_h,H_l,H_h + vxor H2_t,H_h,RP + + vsldoi H2,H2_t,H2_t,8 + vsldoi H2h,H2,zero,8 + vsldoi H2l,zero,H2,8 + + # --- calculate [H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store H,[H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + li 8,0x00 + li 9,0x100 + li 10,0x200 + stxvd2x HlX,8,TABLE + stxvd2x HX,9,TABLE + stxvd2x HhX,10,TABLE + + li 8,0x300 + li 9,0x400 + li 10,0x500 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^3,H^4 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^2l + vpmsumd H_m,H_t,H2 # H^1h*H^2l⊕H^1l*H^2h + vpmsumd H_h,H_t,H2h # H^1h*H^2h + vpmsumd H2_l,H2_t,H2l # H^2l*H^2l + vpmsumd H2_m,H2_t,H2 # H^2h*H^2l⊕H^2l*H^2h + vpmsumd H2_h,H2_t,H2h # H^2h*H^2h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^3 + vpmsumd RP2,H2_l,poly_h # [1] H^4 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^3 + vsldoi M2h,zero,H2_m,8 # [2] H^4 + vsldoi Ml,H_m,zero,8 # [2] H^3 + vsldoi M2l,H2_m,zero,8 # [2] H^4 + vsldoi RP,RP,RP,8 # [1] H^3 + vsldoi RP2,RP2,RP2,8 # [1] H^4 + vxor H_h,H_h,Mh # [2] H^3 + vxor H2_h,H2_h,M2h # [2] H^4 + vxor H_l,H_l,Ml # [2] H^3 + vxor H2_l,H2_l,M2l # [2] H^4 + vxor H_l,H_l,RP # [1] H^3 + vxor H2_l,H2_l,RP2 # [1] H^4 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^3 + vpmsumd RP2,H2_l,poly_l # H^4 + vxor H_h,H_l,H_h # H^3 + vxor H2_h,H2_l,H2_h # H^4 + vxor H_h,H_h,RP # H^3 + vxor H2_h,H2_h,RP2 # H^4 + + vsldoi H2,H2_h,H2_h,8 # H^4 + vsldoi H,H_h,H_h,8 # H^3 + vsldoi H2l,zero,H2,8 # H^4 + vsldoi H2h,H2,zero,8 # H^4 + + # --- calculate [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + li 8,0x600 + li 9,0x700 + li 10,0x800 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^5,H^6 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^4l + vpmsumd H_m,H_t,H2 # H^1h*H^4l⊕H^1l*H^4h + vpmsumd H_h,H_t,H2h # H^1h*H^4h + vpmsumd H2_l,H2_t,H2l # H^2l*H^4l + vpmsumd H2_m,H2_t,H2 # H^2h*H^4l⊕H^2l*H^4h + vpmsumd H2_h,H2_t,H2h # H^2h*H^4h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^5 + vpmsumd RP2,H2_l,poly_h # [1] H^6 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^5 + vsldoi M2h,zero,H2_m,8 # [2] H^6 + vsldoi Ml,H_m,zero,8 # [2] H^5 + vsldoi M2l,H2_m,zero,8 # [2] H^6 + vsldoi RP,RP,RP,8 # [1] H^5 + vsldoi RP2,RP2,RP2,8 # [1] H^6 + vxor H_h,H_h,Mh # [2] H^5 + vxor H2_h,H2_h,M2h # [2] H^6 + vxor H_l,H_l,Ml # [2] H^5 + vxor H2_l,H2_l,M2l # [2] H^6 + vxor H_l,H_l,RP # [1] H^5 + vxor H2_l,H2_l,RP2 # [1] H^6 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^5 + vpmsumd RP2,H2_l,poly_l # H^6 + vxor H_h,H_l,H_h # H^5 + vxor H2_h,H2_l,H2_h # H^6 + vxor H_h,H_h,RP # H^5 + vxor H2_h,H2_h,RP2 # H^6 + + vsldoi H2,H2_h,H2_h,8 # H^6 + vsldoi H,H_h,H_h,8 # H^5 + vsldoi H2l,zero,H2,8 # H^6 + vsldoi H2h,H2,zero,8 # H^6 + + # --- calculate [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + li 8,0x900 + li 9,0xA00 + li 10,0xB00 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^7,H^8 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^6l + vpmsumd H_m,H_t,H2 # H^1h*H^6l⊕H^1l*H^6h + vpmsumd H_h,H_t,H2h # H^1h*H^6h + vpmsumd H2_l,H2_t,H2l # H^2l*H^6l + vpmsumd H2_m,H2_t,H2 # H^2h*H^6l⊕H^2l*H^6h + vpmsumd H2_h,H2_t,H2h # H^2h*H^6h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^7 + vpmsumd RP2,H2_l,poly_h # [1] H^8 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^7 + vsldoi M2h,zero,H2_m,8 # [2] H^8 + vsldoi Ml,H_m,zero,8 # [2] H^7 + vsldoi M2l,H2_m,zero,8 # [2] H^8 + vsldoi RP,RP,RP,8 # [1] H^7 + vsldoi RP2,RP2,RP2,8 # [1] H^8 + vxor H_h,H_h,Mh # [2] H^7 + vxor H2_h,H2_h,M2h # [2] H^8 + vxor H_l,H_l,Ml # [2] H^7 + vxor H2_l,H2_l,M2l # [2] H^8 + vxor H_l,H_l,RP # [1] H^7 + vxor H2_l,H2_l,RP2 # [1] H^8 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^7 + vpmsumd RP2,H2_l,poly_l # H^8 + vxor H_h,H_l,H_h # H^7 + vxor H2_h,H2_l,H2_h # H^8 + vxor H_h,H_h,RP # H^7 + vxor H2_h,H2_h,RP2 # H^8 + + vsldoi H,H_h,H_h,8 # H^7 + vsldoi H2,H2_h,H2_h,8 # H^8 + + # --- calculate [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + li 8,0xC00 + li 9,0xD00 + li 10,0xE00 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + blr +EPILOGUE(_nettle_gcm_init_key8) + + # void gcm_hash (const struct gcm_key *key, union gcm_block *x, + # size_t length, const uint8_t *data) + +.align 5 +PROLOGUE(_nettle_gcm_hash8) + vxor zero,zero,zero + + ld 7,.polynomial@got(TOCP) + lvx poly,0,7 + ld 7,.swap_mask@got(TOCP) + lvx swap_mask,0,7 + ld 7,.hidw_mask@got(TOCP) + lvx hidw_mask,0,7 + ld 7,.lodw_mask@got(TOCP) + lvx lodw_mask,0,7 + + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + lxvd2x CX,0,X # load X + vperm C,C,C,swap_mask + + srdi 7,LENGTH,7 # 8x loop count + cmpldi 7,0 + beq L2x + + # backup registers + stdu SP,-224(SP) + std 28,216(SP) + std 29,208(SP) + std 30,200(SP) + std 31,192(SP) + li 8,176 + stvx 20,8,SP + subi 8,8,16 + stvx 21,8,SP + subi 8,8,16 + stvx 22,8,SP + subi 8,8,16 + stvx 23,8,SP + subi 8,8,16 + stvx 24,8,SP + subi 8,8,16 + stvx 25,8,SP + subi 8,8,16 + stvx 26,8,SP + subi 8,8,16 + stvx 27,8,SP + subi 8,8,16 + stvx 28,8,SP + subi 8,8,16 + stvx 29,8,SP + subi 8,8,16 + stvx 30,8,SP + subi 8,8,16 + stvx 31,8,SP + + # table loading + li 8,0x300 + li 9,0x400 + li 10,0x500 + lxvd2x H21hX,8,TABLE + lxvd2x H21X,9,TABLE + lxvd2x H21lX,10,TABLE + li 8,0x600 + li 9,0x700 + li 10,0x800 + lxvd2x H43hX,8,TABLE + lxvd2x H43X,9,TABLE + lxvd2x H43lX,10,TABLE + li 8,0x900 + li 9,0xA00 + li 10,0xB00 + lxvd2x H65hX,8,TABLE + lxvd2x H65X,9,TABLE + lxvd2x H65lX,10,TABLE + li 8,0xC00 + li 9,0xD00 + li 10,0xE00 + lxvd2x H87hX,8,TABLE + lxvd2x H87X,9,TABLE + lxvd2x H87lX,10,TABLE + + li 8,0x10 + li 9,0x20 + li 10,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + + mtctr 7 +.align 5 +L8x_loop: + # input loading + lxvd2x C0X,0,DATA # load C0 + lxvd2x C1X,8,DATA # load C1 + lxvd2x C2X,9,DATA # load C2 + lxvd2x C3X,10,DATA # load C3 + + # swap permuting + vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask + vperm C2,C2,C2,swap_mask + vperm C3,C3,C3,swap_mask + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C23h,C2,C3,hidw_mask + vperm C23l,C2,C3,lodw_mask + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + + # input loading + lxvd2x C4X,28,DATA # load C4 + lxvd2x C5X,29,DATA # load C5 + lxvd2x C6X,30,DATA # load C6 + lxvd2x C7X,31,DATA # load C7 + + # swap permuting + vperm C4,C4,C4,swap_mask + vperm C5,C5,C5,swap_mask + vperm C6,C6,C6,swap_mask + vperm C7,C7,C7,swap_mask + + # polynomial multiplication "karatsuba" pre-processing + vperm C45h,C4,C5,hidw_mask + vperm C45l,C4,C5,lodw_mask + vperm C67h,C6,C7,hidw_mask + vperm C67l,C6,C7,lodw_mask + vxor C23,C23h,C23l + vxor C01,C01h,C01l + vxor C45,C45h,C45l + vxor C67,C67h,C67l + + # polynomial multiplication "karatsuba" + vpmsumd C23h,C23h,H65h # H23 = H^6h*C2h⊕H^5h*C3h + vpmsumd C23l,C23l,H65l # L23 = H^6l*C2l⊕H^5l*C3l + vpmsumd C01h,C01h,H87h # H01 = H^8h*C0h⊕H^7h*C1h + vpmsumd C01l,C01l,H87l # L01 = H^8l*C0l⊕H^7l*C1l + vpmsumd C67h,C67h,H21h # H67 = H^2h*C6h⊕H^1h*C7h + vpmsumd C67l,C67l,H21l # L67 = H^2l*C6l⊕H^1l*C7l + vpmsumd C45h,C45h,H43h # H45 = H^4h*C4h⊕H^3h*C5h + vpmsumd C45l,C45l,H43l # L45 = H^4l*C4l⊕H^3l*C5l + vpmsumd C23,C23,H65 # M23 = (H^6h⊕H^5h)*(C2h⊕C3h)⊕(H^6l⊕H^5l)*(C2l⊕C3l) + vpmsumd C01,C01,H87 # M01 = (H^8h⊕H^7h)*(C0h⊕C1h)⊕(H^8l⊕H^7l)*(C0l⊕C1l) + vpmsumd C45,C45,H43 # M45 = (H^4h⊕H^3h)*(C4h⊕C5h)⊕(H^4l⊕H^3l)*(C4l⊕C5l) + vpmsumd C67,C67,H21 # M67 = (H^2h⊕H^1h)*(C6h⊕C7h)⊕(H^2l⊕H^1l)*(C6l⊕C7l) + + # polynomial multiplication "karatsuba" post-processing + vxor C23,C23,C23h + vxor C01,C01,C01h + vxor C45,C45,C45h + vxor C67,C67,C67h + vxor C23,C23,C23l + vxor C01,C01,C01l + vxor C45,C45,C45l + vxor C67,C67,C67l + + # deferred recombination of partial products + vxor C01h,C01h,C23h # H0 = H01⊕H23 + vxor C45h,C45h,C67h # H1 = H45⊕H67 + vxor C01l,C01l,C23l # L0 = L01⊕L23 + vxor C45l,C45l,C67l # L1 = L45⊕L45 + vxor C01,C01,C23 # M0 = M01⊕M23 + vxor C45,C45,C67 # M1 = M45⊕M45 + vxor C01h,C01h,C45h # H = H0⊕H1 + vxor C01l,C01l,C45l # L = L0⊕L1 + vxor C01,C01,C45 # M = M0⊕M1 + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x80 + bdnz L8x_loop + + # restore registers + li 8,0 + lvx 31,8,SP + addi 8,8,16 + lvx 30,8,SP + addi 8,8,16 + lvx 29,8,SP + addi 8,8,16 + lvx 28,8,SP + addi 8,8,16 + lvx 27,8,SP + addi 8,8,16 + lvx 26,8,SP + addi 8,8,16 + lvx 25,8,SP + addi 8,8,16 + lvx 24,8,SP + addi 8,8,16 + lvx 23,8,SP + addi 8,8,16 + lvx 22,8,SP + addi 8,8,16 + lvx 21,8,SP + addi 8,8,16 + lvx 20,8,SP + ld 31,192(SP) + ld 30,200(SP) + ld 29,208(SP) + ld 28,216(SP) + addi SP,SP,224 + + clrldi LENGTH,LENGTH,57 +L2x: + srdi 7,LENGTH,5 + cmpldi 7,0 + beq L1x + + # table loading + li 8,0x300 + li 9,0x400 + li 10,0x500 + lxvd2x H21hX,8,TABLE + lxvd2x H21X,9,TABLE + lxvd2x H21lX,10,TABLE + + li 10,0x10 + + mtctr 7 +.align 5 +L2x_loop: + # input loading + lxvd2x C0X,0,DATA # load C0 + lxvd2x C1X,10,DATA # load C1 + + # swap permuting + vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + vxor C01,C01h,C01l + + # polynomial multiplication "karatsuba" + vpmsumd C01h,C01h,H21h # H01 = H^2h*C0h⊕H^1h*C1h + vpmsumd C01l,C01l,H21l # L01 = H^2l*C0l⊕H^1l*C1l + vpmsumd C01,C01,H21 # M01 = (H^2h⊕H^1h)*(C0h⊕C1h)⊕(H^2l⊕H^1l)*(C0l⊕C1l) + + # polynomial multiplication "karatsuba" post-processing + vxor C01,C01,C01h + vxor C01,C01,C01l + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x20 + bdnz L2x_loop + + clrldi LENGTH,LENGTH,59 +L1x: + srdi 7,LENGTH,4 + cmpldi 7,0 + beq Lrem + + # table loading + li 9,0x100 + li 10,0x200 + lxvd2x HlX,0,TABLE + lxvd2x HX, 9,TABLE + lxvd2x HhX,10,TABLE + + # input loading + lxvd2x C0X,0,DATA # load C0 + + # swap permuting + vperm C0,C0,C0,swap_mask + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + + addi DATA,DATA,0x10 + clrldi LENGTH,LENGTH,60 +Lrem: + cmpldi LENGTH,0 + beq Ldone + + # table loading + li 9,0x100 + li 10,0x200 + lxvd2x HlX,0,TABLE + lxvd2x HX, 9,TABLE + lxvd2x HhX,10,TABLE + + # input loading + stdu SP,-16(SP) + stvx zero,0,SP +Lst_loop: + subic. LENGTH,LENGTH,1 + lbzx 7,LENGTH,DATA + stbx 7,LENGTH,SP + bne Lst_loop + lxvd2x C0X,0,SP + addi SP,SP,16 + + # swap permuting + vperm C0,C0,C0,swap_mask + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + +Ldone: + vperm C,C,C,swap_mask + stxvd2x CX,0,X # store C + blr +EPILOGUE(_nettle_gcm_hash8) + + # gcm_fill (uint8_t *ctr, size_t blocks, union gcm_block *buffer) + +.align 5 +PROLOGUE(_nettle_gcm_fill) + ld 6,.swap_mask@got(TOCP) + lvx swap_mask,0,6 + + vxor zero,zero,zero + vspltisb I1,1 + vspltisb I2,2 + vspltisb I3,3 + vspltisb I4,4 + vspltisb I5,5 + vspltisb I6,6 + vspltisb I7,7 + vspltisb I8,8 + vsldoi I1,zero,I1,1 + vsldoi I2,zero,I2,1 + vsldoi I3,zero,I3,1 + vsldoi I4,zero,I4,1 + vsldoi I5,zero,I5,1 + vsldoi I6,zero,I6,1 + vsldoi I7,zero,I7,1 + vsldoi I8,zero,I8,1 + + lxvd2x CTR0X,0,CTR + vperm CTR0,CTR0,CTR0,swap_mask + + srdi 6,BLOCKS,3 # 8x loop count + cmpldi 6,0 + beq Lfill_4x + + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 25,0x10 + li 26,0x20 + li 27,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + + mtctr 6 +L8x_fill_loop: + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + vadduwm CTR4,CTR0,I4 + vadduwm CTR5,CTR0,I5 + vadduwm CTR6,CTR0,I6 + vadduwm CTR7,CTR0,I7 + + vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask + vperm CTR4,CTR4,CTR4,swap_mask + vperm CTR5,CTR5,CTR5,swap_mask + vperm CTR6,CTR6,CTR6,swap_mask + vperm CTR7,CTR7,CTR7,swap_mask + + stxvd2x CTR0SX,0,BUFFER + stxvd2x CTR1X,25,BUFFER + stxvd2x CTR2X,26,BUFFER + stxvd2x CTR3X,27,BUFFER + stxvd2x CTR4X,28,BUFFER + stxvd2x CTR5X,29,BUFFER + stxvd2x CTR6X,30,BUFFER + stxvd2x CTR7X,31,BUFFER + + vadduwm CTR0,CTR0,I8 + addi BUFFER,BUFFER,0x80 + bdnz L8x_fill_loop + + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi BLOCKS,BLOCKS,61 + +Lfill_4x: + srdi 6,BLOCKS,2 + cmpldi 6,0 + beq Lfill_2x + + li 8,0x10 + li 9,0x20 + li 10,0x30 + + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + + vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask + + stxvd2x CTR0SX,0,BUFFER + stxvd2x CTR1X,8,BUFFER + stxvd2x CTR2X,9,BUFFER + stxvd2x CTR3X,10,BUFFER + + vadduwm CTR0,CTR0,I4 + addi BUFFER,BUFFER,0x40 + + clrldi BLOCKS,BLOCKS,62 + +Lfill_2x: + srdi 6,BLOCKS,1 + cmpldi 6,0 + beq Lfill_1x + + li 10,0x10 + + vadduwm CTR1,CTR0,I1 + + vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + + stxvd2x CTR0SX,0,BUFFER + stxvd2x CTR1X,10,BUFFER + + vadduwm CTR0,CTR0,I2 + addi BUFFER,BUFFER,0x20 + + clrldi BLOCKS,BLOCKS,63 + +Lfill_1x: + cmpldi BLOCKS,0 + beq Lfill_done + + vperm CTR0S,CTR0,CTR0,swap_mask + + stxvd2x CTR0SX,0,BUFFER + + vadduwm CTR0,CTR0,I1 + +Lfill_done: + vperm CTR0,CTR0,CTR0,swap_mask + stxvd2x CTR0X,0,CTR + + blr +EPILOGUE(_nettle_gcm_fill) + + .data + .align 4 +.polynomial: + .byte 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0xc2 + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 + .align 4 +.hidw_mask: + .byte 23,22,21,20,19,18,17,16,7,6,5,4,3,2,1,0 + .align 4 +.lodw_mask: + .byte 31,30,29,28,27,26,25,24,15,14,13,12,11,10,9,8 diff --git a/powerpc64le/machine.m4 b/powerpc64le/machine.m4 new file mode 100644 index 00000000..e69de29b diff --git a/testsuite/gcm-test.c b/testsuite/gcm-test.c index c8174019..df1fc94a 100644 --- a/testsuite/gcm-test.c +++ b/testsuite/gcm-test.c @@ -170,6 +170,29 @@ test_main(void) "16aedbf5a0de6a57a637b39b"), SHEX("619cc5aefffe0bfa462af43c1699d050")); + /* Test 128 bytes */ + test_aead(&nettle_gcm_aes128, NULL, + SHEX("feffe9928665731c6d6a8f9467308308"), + SHEX(""), + SHEX("d9313225f88406e5a55909c5aff5269a" + "86a7a9531534f7da2e4c303d8a318a72" + "1c3c0c95956809532fcf0e2449a6b525" + "b16aedf5aa0de657ba637b391aafd255" + "5ae376bc5e9f6a1b08e34db7a6ee0736" + "9ba662ea12f6f197e6bc3ed69d2480f3" + "ea5691347f2ba69113eb37910ebc18c8" + "0f697234582016fa956ca8f63ae6b473"), + SHEX("42831ec2217774244b7221b784d0d49c" + "e3aa212f2c02a4e035c17e2329aca12e" + "21d514b25466931c7d8f6a5aac84aa05" + "1ba30b396a0aac973d58e091473f5985" + "874b1178906ddbeab04ab2fe6cce8c57" + "8d7e961bd13fd6a8c56b66ca5e576492" + "1a48cd8bda04e66343e73055118b69b9" + "ced486813846958a11e602c03cfc232b"), + SHEX("cafebabefacedbaddecaf888"), + SHEX("796836f1246c9d735c5e1be0a715ccc3")); + /* Test case 7 */ test_aead(&nettle_gcm_aes192, NULL, SHEX("00000000000000000000000000000000" -- 2.17.1

1 0

[PATCH] optimized AES and GHASH for PPC64LE
by Maamoun TK 19 Jun '20

19 Jun '20

Makefile.in | 2 +- configure.ac | 5 + gcm.c | 19 +- powerpc64le/aes-decrypt-internal.asm (new +x) | 573 +++++++++++++++ powerpc64le/aes-encrypt-internal.asm (new +x) | 534 ++++++++++++++ powerpc64le/gcm-hash8.asm (new +x) | 992 ++++++++++++++++++++++++++ powerpc64le/machine.m4 (new +x) | 0 testsuite/gcm-test.c | 23 + 8 files changed, 2146 insertions(+), 2 deletions(-) create mode 100644 powerpc64le/aes-decrypt-internal.asm create mode 100644 powerpc64le/aes-encrypt-internal.asm create mode 100644 powerpc64le/gcm-hash8.asm create mode 100644 powerpc64le/machine.m4 diff --git a/Makefile.in b/Makefile.in index 64ff1001..5bbc0f79 100644 --- a/Makefile.in +++ b/Makefile.in @@ -603,7 +603,7 @@ distdir: $(DISTFILES) done set -e; for d in sparc32 sparc64 x86 \ x86_64 x86_64/aesni x86_64/sha_ni x86_64/fat \ - arm arm/neon arm/v6 arm/fat ; do \ + arm arm/neon arm/v6 arm/fat powerpc64le ; do \ mkdir "$(distdir)/$$d" ; \ find "$(srcdir)/$$d" -maxdepth 1 '(' -name '*.asm' -o -name '*.m4' ')' \ -exec cp '{}' "$(distdir)/$$d" ';' ; \ diff --git a/configure.ac b/configure.ac index 90ea1ea8..1ea54ce8 100644 --- a/configure.ac +++ b/configure.ac @@ -435,6 +435,9 @@ if test "x$enable_assembler" = xyes ; then esac fi ;; + *powerpc64le*) + asm_path=powerpc64le + ;; *) enable_assembler=no ;; @@ -572,7 +575,9 @@ AH_VERBATIM([HAVE_NATIVE], #undef HAVE_NATIVE_ecc_secp384r1_redc #undef HAVE_NATIVE_ecc_secp521r1_modp #undef HAVE_NATIVE_ecc_secp521r1_redc +#undef HAVE_NATIVE_gcm_init_key8 #undef HAVE_NATIVE_gcm_hash8 +#undef HAVE_NATIVE_gcm_fill #undef HAVE_NATIVE_salsa20_core #undef HAVE_NATIVE_sha1_compress #undef HAVE_NATIVE_sha256_compress diff --git a/gcm.c b/gcm.c index cf615daf..809c03bc 100644 --- a/gcm.c +++ b/gcm.c @@ -140,6 +140,12 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) memcpy (x->b, Z.b, sizeof(Z)); } # elif GCM_TABLE_BITS == 8 +# if HAVE_NATIVE_gcm_init_key8 + +#define gcm_init_key _nettle_gcm_init_key8 +void +_nettle_gcm_init_key8 (union nettle_block16 *table); +# endif /* HAVE_NATIVE_gcm_init_key8 */ # if HAVE_NATIVE_gcm_hash8 #define gcm_hash _nettle_gcm_hash8 @@ -225,6 +231,13 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) #endif /* GCM_TABLE_BITS */ +#if HAVE_NATIVE_gcm_fill + +#define gcm_fill _nettle_gcm_fill +void +_nettle_gcm_fill (uint8_t *ctr, size_t blocks, union nettle_block16 *buffer); +#endif /* HAVE_NATIVE_gcm_fill */ + /* Increment the rightmost 32 bits. */ #define INC32(block) INCREMENT(4, (block.b) + GCM_BLOCK_SIZE - 4) @@ -245,7 +258,9 @@ gcm_set_key(struct gcm_key *key, memset(key->h[0].b, 0, GCM_BLOCK_SIZE); f (cipher, GCM_BLOCK_SIZE, key->h[i].b, key->h[0].b); -#if GCM_TABLE_BITS +#ifdef gcm_init_key + gcm_init_key(key->h); +#elif GCM_TABLE_BITS /* Algorithm 3 from the gcm paper. First do powers of two, then do the rest by adding. */ while (i /= 2) @@ -333,6 +348,7 @@ gcm_update(struct gcm_ctx *ctx, const struct gcm_key *key, ctx->auth_size += length; } +#ifndef gcm_fill static nettle_fill16_func gcm_fill; static void gcm_fill(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) @@ -349,6 +365,7 @@ gcm_fill(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) WRITE_UINT32(ctr + GCM_BLOCK_SIZE - 4, c); } +#endif /* !gcm_fill */ void gcm_encrypt (struct gcm_ctx *ctx, const struct gcm_key *key, diff --git a/powerpc64le/aes-decrypt-internal.asm b/powerpc64le/aes-decrypt-internal.asm new file mode 100644 index 00000000..bde34779 --- /dev/null +++ b/powerpc64le/aes-decrypt-internal.asm @@ -0,0 +1,573 @@ +C powerpc64le/aes-decrypt-internal.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<ROUNDS>, <3>) +define(<KEYS>, <4>) +define(<LENGTH>, <6>) +define(<DST>, <7>) +define(<SRC>, <8>) + +define(<swap_mask>, <0>) + +define(<K>, <1>) +define(<S0>, <2>) +define(<S1>, <3>) +define(<S2>, <4>) +define(<S3>, <5>) +define(<S4>, <6>) +define(<S5>, <7>) +define(<S6>, <8>) +define(<S7>, <9>) +define(<S8>, <10>) +define(<S9>, <11>) +define(<S10>, <12>) +define(<S11>, <13>) +define(<S12>, <14>) +define(<S13>, <15>) +define(<S14>, <16>) +define(<S15>, <17>) + +define(<KX>, <33>) +define(<S0X>, <34>) +define(<S1X>, <35>) +define(<S2X>, <36>) +define(<S3X>, <37>) +define(<S4X>, <38>) +define(<S5X>, <39>) +define(<S6X>, <40>) +define(<S7X>, <41>) +define(<S8X>, <42>) +define(<S9X>, <43>) +define(<S10X>, <44>) +define(<S11X>, <45>) +define(<S12X>, <46>) +define(<S13X>, <47>) +define(<S14X>, <48>) +define(<S15X>, <49>) + +C ZERO vector register is used in place of RoundKey +C for vncipher instruction because the order of InvMixColumns +C and Xor processes are flipped in that instruction. +C The Xor process with RoundKey is executed afterward. +define(<ZERO>, <18>) + + .file "aes-decrypt-internal.asm" + + C _aes_decrypt(unsigned rounds, const uint32_t *keys, + C const struct aes_table *T, + C size_t length, uint8_t *dst, + C uint8_t *src) + + .text +.align 5 +PROLOGUE(_nettle_aes_decrypt) + vxor ZERO,ZERO,ZERO + + ld 5,.swap_mask@got(TOCP) + lvx swap_mask,0,5 + + subi ROUNDS,ROUNDS,1 + srdi LENGTH,LENGTH,4 + + srdi 5,LENGTH,4 # 16x loop count + cmpldi 5,0 + beq L8x + + std 17,-120(SP); + std 18,-112(SP); + std 19,-104(SP); + std 20,-96(SP); + std 21,-88(SP); + std 22,-80(SP); + std 23,-72(SP); + std 24,-64(SP); + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 17,0x10 + li 18,0x20 + li 19,0x30 + li 20,0x40 + li 21,0x50 + li 22,0x60 + li 23,0x70 + li 24,0x80 + li 25,0x90 + li 26,0xA0 + li 27,0xB0 + li 28,0xC0 + li 29,0xD0 + li 30,0xE0 + li 31,0xF0 + +.align 5 +Lx16_loop: + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + lxvd2x S1X,17,SRC + lxvd2x S2X,18,SRC + lxvd2x S3X,19,SRC + lxvd2x S4X,20,SRC + lxvd2x S5X,21,SRC + lxvd2x S6X,22,SRC + lxvd2x S7X,23,SRC + lxvd2x S8X,24,SRC + lxvd2x S9X,25,SRC + lxvd2x S10X,26,SRC + lxvd2x S11X,27,SRC + lxvd2x S12X,28,SRC + lxvd2x S13X,29,SRC + lxvd2x S14X,30,SRC + lxvd2x S15X,31,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L16x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vncipher S4,S4,ZERO + vncipher S5,S5,ZERO + vncipher S6,S6,ZERO + vncipher S7,S7,ZERO + vncipher S8,S8,ZERO + vncipher S9,S9,ZERO + vncipher S10,S10,ZERO + vncipher S11,S11,ZERO + vncipher S12,S12,ZERO + vncipher S13,S13,ZERO + vncipher S14,S14,ZERO + vncipher S15,S15,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + addi 10,10,0x10 + bdnz L16x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + vncipherlast S4,S4,K + vncipherlast S5,S5,K + vncipherlast S6,S6,K + vncipherlast S7,S7,K + vncipherlast S8,S8,K + vncipherlast S9,S9,K + vncipherlast S10,S10,K + vncipherlast S11,S11,K + vncipherlast S12,S12,K + vncipherlast S13,S13,K + vncipherlast S14,S14,K + vncipherlast S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + stxvd2x S0X,0,DST + stxvd2x S1X,17,DST + stxvd2x S2X,18,DST + stxvd2x S3X,19,DST + stxvd2x S4X,20,DST + stxvd2x S5X,21,DST + stxvd2x S6X,22,DST + stxvd2x S7X,23,DST + stxvd2x S8X,24,DST + stxvd2x S9X,25,DST + stxvd2x S10X,26,DST + stxvd2x S11X,27,DST + stxvd2x S12X,28,DST + stxvd2x S13X,29,DST + stxvd2x S14X,30,DST + stxvd2x S15X,31,DST + + addi SRC,SRC,0x100 + addi DST,DST,0x100 + subic. 5,5,1 + bne Lx16_loop + + ld 17,-120(SP); + ld 18,-112(SP); + ld 19,-104(SP); + ld 20,-96(SP); + ld 21,-88(SP); + ld 22,-80(SP); + ld 23,-72(SP); + ld 24,-64(SP); + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi LENGTH,LENGTH,60 + +L8x: + srdi 5,LENGTH,3 + cmpldi 5,0 + beq L4x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + addi 9,9,0x10 + lxvd2x S4X,9,SRC + addi 9,9,0x10 + lxvd2x S5X,9,SRC + addi 9,9,0x10 + lxvd2x S6X,9,SRC + addi 9,9,0x10 + lxvd2x S7X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L8x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vncipher S4,S4,ZERO + vncipher S5,S5,ZERO + vncipher S6,S6,ZERO + vncipher S7,S7,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + addi 10,10,0x10 + bdnz L8x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + vncipherlast S4,S4,K + vncipherlast S5,S5,K + vncipherlast S6,S6,K + vncipherlast S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + addi 9,9,0x10 + stxvd2x S4X,9,DST + addi 9,9,0x10 + stxvd2x S5X,9,DST + addi 9,9,0x10 + stxvd2x S6X,9,DST + addi 9,9,0x10 + stxvd2x S7X,9,DST + + addi SRC,SRC,0x80 + addi DST,DST,0x80 + + clrldi LENGTH,LENGTH,61 + +L4x: + srdi 5,LENGTH,2 + cmpldi 5,0 + beq L2x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L4x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + addi 10,10,0x10 + bdnz L4x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + + addi SRC,SRC,0x40 + addi DST,DST,0x40 + + clrldi LENGTH,LENGTH,62 + +L2x: + srdi 5,LENGTH,1 + cmpldi 5,0 + beq L1x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L2x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vxor S0,S0,K + vxor S1,S1,K + addi 10,10,0x10 + bdnz L2x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + + addi SRC,SRC,0x20 + addi DST,DST,0x20 + + clrldi LENGTH,LENGTH,63 + +L1x: + cmpldi LENGTH,0 + beq Ldone + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + + vxor S0,S0,K + + vperm S0,S0,S0,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L1x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vxor S0,S0,K + addi 10,10,0x10 + bdnz L1x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + + vperm S0,S0,S0,swap_mask + + stxvd2x S0X,0,DST + +Ldone: + blr +EPILOGUE(_nettle_aes_decrypt) + + .data + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 diff --git a/powerpc64le/aes-encrypt-internal.asm b/powerpc64le/aes-encrypt-internal.asm new file mode 100644 index 00000000..1bbd86a8 --- /dev/null +++ b/powerpc64le/aes-encrypt-internal.asm @@ -0,0 +1,534 @@ +C powerpc64le/aes-encrypt-internal.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<ROUNDS>, <3>) +define(<KEYS>, <4>) +define(<LENGTH>, <6>) +define(<DST>, <7>) +define(<SRC>, <8>) + +define(<swap_mask>, <0>) + +define(<K>, <1>) +define(<S0>, <2>) +define(<S1>, <3>) +define(<S2>, <4>) +define(<S3>, <5>) +define(<S4>, <6>) +define(<S5>, <7>) +define(<S6>, <8>) +define(<S7>, <9>) +define(<S8>, <10>) +define(<S9>, <11>) +define(<S10>, <12>) +define(<S11>, <13>) +define(<S12>, <14>) +define(<S13>, <15>) +define(<S14>, <16>) +define(<S15>, <17>) + +define(<KX>, <33>) +define(<S0X>, <34>) +define(<S1X>, <35>) +define(<S2X>, <36>) +define(<S3X>, <37>) +define(<S4X>, <38>) +define(<S5X>, <39>) +define(<S6X>, <40>) +define(<S7X>, <41>) +define(<S8X>, <42>) +define(<S9X>, <43>) +define(<S10X>, <44>) +define(<S11X>, <45>) +define(<S12X>, <46>) +define(<S13X>, <47>) +define(<S14X>, <48>) +define(<S15X>, <49>) + + .file "aes-encrypt-internal.asm" + + C _aes_encrypt(unsigned rounds, const uint32_t *keys, + C const struct aes_table *T, + C size_t length, uint8_t *dst, + C uint8_t *src) + + .text +.align 5 +PROLOGUE(_nettle_aes_encrypt) + ld 5,.swap_mask@got(TOCP) + lvx swap_mask,0,5 + + subi ROUNDS,ROUNDS,1 + srdi LENGTH,LENGTH,4 + + srdi 5,LENGTH,4 # 16x loop count + cmpldi 5,0 + beq L8x + + std 17,-120(SP); + std 18,-112(SP); + std 19,-104(SP); + std 20,-96(SP); + std 21,-88(SP); + std 22,-80(SP); + std 23,-72(SP); + std 24,-64(SP); + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 17,0x10 + li 18,0x20 + li 19,0x30 + li 20,0x40 + li 21,0x50 + li 22,0x60 + li 23,0x70 + li 24,0x80 + li 25,0x90 + li 26,0xA0 + li 27,0xB0 + li 28,0xC0 + li 29,0xD0 + li 30,0xE0 + li 31,0xF0 + +.align 5 +Lx16_loop: + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + lxvd2x S1X,17,SRC + lxvd2x S2X,18,SRC + lxvd2x S3X,19,SRC + lxvd2x S4X,20,SRC + lxvd2x S5X,21,SRC + lxvd2x S6X,22,SRC + lxvd2x S7X,23,SRC + lxvd2x S8X,24,SRC + lxvd2x S9X,25,SRC + lxvd2x S10X,26,SRC + lxvd2x S11X,27,SRC + lxvd2x S12X,28,SRC + lxvd2x S13X,29,SRC + lxvd2x S14X,30,SRC + lxvd2x S15X,31,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L16x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + vcipher S4,S4,K + vcipher S5,S5,K + vcipher S6,S6,K + vcipher S7,S7,K + vcipher S8,S8,K + vcipher S9,S9,K + vcipher S10,S10,K + vcipher S11,S11,K + vcipher S12,S12,K + vcipher S13,S13,K + vcipher S14,S14,K + vcipher S15,S15,K + addi 10,10,0x10 + bdnz L16x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + vcipherlast S4,S4,K + vcipherlast S5,S5,K + vcipherlast S6,S6,K + vcipherlast S7,S7,K + vcipherlast S8,S8,K + vcipherlast S9,S9,K + vcipherlast S10,S10,K + vcipherlast S11,S11,K + vcipherlast S12,S12,K + vcipherlast S13,S13,K + vcipherlast S14,S14,K + vcipherlast S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + stxvd2x S0X,0,DST + stxvd2x S1X,17,DST + stxvd2x S2X,18,DST + stxvd2x S3X,19,DST + stxvd2x S4X,20,DST + stxvd2x S5X,21,DST + stxvd2x S6X,22,DST + stxvd2x S7X,23,DST + stxvd2x S8X,24,DST + stxvd2x S9X,25,DST + stxvd2x S10X,26,DST + stxvd2x S11X,27,DST + stxvd2x S12X,28,DST + stxvd2x S13X,29,DST + stxvd2x S14X,30,DST + stxvd2x S15X,31,DST + + addi SRC,SRC,0x100 + addi DST,DST,0x100 + subic. 5,5,1 + bne Lx16_loop + + ld 17,-120(SP); + ld 18,-112(SP); + ld 19,-104(SP); + ld 20,-96(SP); + ld 21,-88(SP); + ld 22,-80(SP); + ld 23,-72(SP); + ld 24,-64(SP); + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi LENGTH,LENGTH,60 + +L8x: + srdi 5,LENGTH,3 + cmpldi 5,0 + beq L4x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + addi 9,9,0x10 + lxvd2x S4X,9,SRC + addi 9,9,0x10 + lxvd2x S5X,9,SRC + addi 9,9,0x10 + lxvd2x S6X,9,SRC + addi 9,9,0x10 + lxvd2x S7X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L8x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + vcipher S4,S4,K + vcipher S5,S5,K + vcipher S6,S6,K + vcipher S7,S7,K + addi 10,10,0x10 + bdnz L8x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + vcipherlast S4,S4,K + vcipherlast S5,S5,K + vcipherlast S6,S6,K + vcipherlast S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + addi 9,9,0x10 + stxvd2x S4X,9,DST + addi 9,9,0x10 + stxvd2x S5X,9,DST + addi 9,9,0x10 + stxvd2x S6X,9,DST + addi 9,9,0x10 + stxvd2x S7X,9,DST + + addi SRC,SRC,0x80 + addi DST,DST,0x80 + + clrldi LENGTH,LENGTH,61 + +L4x: + srdi 5,LENGTH,2 + cmpldi 5,0 + beq L2x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L4x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + addi 10,10,0x10 + bdnz L4x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + + addi SRC,SRC,0x40 + addi DST,DST,0x40 + + clrldi LENGTH,LENGTH,62 + +L2x: + srdi 5,LENGTH,1 + cmpldi 5,0 + beq L1x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L2x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + addi 10,10,0x10 + bdnz L2x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + + addi SRC,SRC,0x20 + addi DST,DST,0x20 + + clrldi LENGTH,LENGTH,63 + +L1x: + cmpldi LENGTH,0 + beq Ldone + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + + vxor S0,S0,K + + vperm S0,S0,S0,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L1x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + addi 10,10,0x10 + bdnz L1x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + + vperm S0,S0,S0,swap_mask + + stxvd2x S0X,0,DST + +Ldone: + blr +EPILOGUE(_nettle_aes_encrypt) + + .data + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 diff --git a/powerpc64le/gcm-hash8.asm b/powerpc64le/gcm-hash8.asm new file mode 100644 index 00000000..a809f6ef --- /dev/null +++ b/powerpc64le/gcm-hash8.asm @@ -0,0 +1,992 @@ +C powerpc64le/gcm-hash8.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: +C VSX instructions is used to load and store data to memory "lxvd2x, stxvd2x" +C instead of VR instructions "lvx, stvx" as a workaround to access unaligned data +C VSX registers are defined with "X" suffix + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<TABLE>, <3>) +define(<X>, <4>) +define(<LENGTH>, <5>) +define(<DATA>, <6>) + +define(<zero>, <0>) +define(<swap_mask>, <1>) +define(<hidw_mask>, <2>) +define(<lodw_mask>, <3>) +define(<poly>, <4>) +define(<poly_h>, <4>) +define(<poly_l>, <5>) +define(<RP>, <6>) +define(<Mh>, <7>) +define(<Ml>, <8>) +define(<H>, <9>) +define(<Hh>, <10>) +define(<Hl>, <11>) +define(<RP2>, <9>) +define(<M2h>, <10>) +define(<M2l>, <11>) + +define(<HX>, <41>) +define(<HhX>, <42>) +define(<HlX>, <43>) +define(<H_HhX>, <44>) +define(<H_HX>, <45>) +define(<H_HlX>, <46>) + +define(<sl1>, <1>) +define(<msb>, <5>) +define(<H2>, <6>) +define(<H2h>, <7>) +define(<H2l>, <8>) +define(<H_h>, <12>) +define(<H_m>, <13>) +define(<H_l>, <14>) +define(<H_Hh>, <12>) +define(<H_H>, <13>) +define(<H_Hl>, <14>) +define(<H_t>, <15>) +define(<H2_h>, <16>) +define(<H2_m>, <17>) +define(<H2_l>, <18>) +define(<H2_t>, <19>) + +define(<C0X>, <38>) +define(<C1X>, <39>) +define(<C2X>, <40>) +define(<C3X>, <44>) +define(<C4X>, <38>) +define(<C5X>, <39>) +define(<C6X>, <40>) +define(<C7X>, <44>) + +define(<CX>, <45>) + +define(<C0>, <6>) +define(<C1>, <7>) +define(<C2>, <8>) +define(<C3>, <12>) +define(<C4>, <6>) +define(<C5>, <7>) +define(<C6>, <8>) +define(<C7>, <12>) + +define(<C>, <13>) + +define(<Ch>, <14>) +define(<Cl>, <15>) +define(<Cm>, <16>) + +define(<C01h>, <14>) +define(<C01l>, <15>) +define(<C01>, <16>) +define(<C23h>, <17>) +define(<C23l>, <18>) +define(<C23>, <19>) +define(<C45h>, <20>) +define(<C45l>, <21>) +define(<C45>, <22>) +define(<C67h>, <6>) +define(<C67l>, <7>) +define(<C67>, <8>) + +define(<H21>, <9>) +define(<H21h>, <10>) +define(<H21l>, <11>) +define(<H43>, <23>) +define(<H43h>, <24>) +define(<H43l>, <25>) +define(<H65>, <26>) +define(<H65h>, <27>) +define(<H65l>, <28>) +define(<H87>, <29>) +define(<H87h>, <30>) +define(<H87l>, <31>) + +define(<H21X>, <41>) +define(<H21hX>, <42>) +define(<H21lX>, <43>) +define(<H43X>, <55>) +define(<H43hX>, <56>) +define(<H43lX>, <57>) +define(<H65X>, <58>) +define(<H65hX>, <59>) +define(<H65lX>, <60>) +define(<H87X>, <61>) +define(<H87hX>, <62>) +define(<H87lX>, <63>) + +# gcm_fill registers: + +define(<CTR>, <3>) +define(<BLOCKS>, <4>) +define(<BUFFER>, <5>) + +define(<CTR0>, <2>) +define(<CTR0S>, <3>) +define(<CTR1>, <4>) +define(<CTR2>, <5>) +define(<CTR3>, <6>) +define(<CTR4>, <7>) +define(<CTR5>, <8>) +define(<CTR6>, <9>) +define(<CTR7>, <10>) + +define(<CTR0X>, <34>) +define(<CTR0SX>, <35>) +define(<CTR1X>, <36>) +define(<CTR2X>, <37>) +define(<CTR3X>, <38>) +define(<CTR4X>, <39>) +define(<CTR5X>, <40>) +define(<CTR6X>, <41>) +define(<CTR7X>, <42>) + +define(<I1>, <11>) +define(<I2>, <12>) +define(<I3>, <13>) +define(<I4>, <14>) +define(<I5>, <15>) +define(<I6>, <16>) +define(<I7>, <17>) +define(<I8>, <18>) + + .file "gcm-hash8.asm" + + # void gcm_init_key (union gcm_block *table) + + .text +.align 5 +PROLOGUE(_nettle_gcm_init_key8) + ld 7,.polynomial@got(TOCP) + lvx poly,0,7 + ld 7,.swap_mask@got(TOCP) + lvx swap_mask,0,7 + ld 7,.hidw_mask@got(TOCP) + lvx hidw_mask,0,7 + ld 7,.lodw_mask@got(TOCP) + lvx lodw_mask,0,7 + + li 10,0x800 + lxvd2x HX,10,TABLE # load H + vperm H,H,H,swap_mask + + # --- calculate H = H shift left 1 modulo polynomial --- + + vupkhsw msb,H # most significant bit word-extend + vspltisb sl1,1 # splat 1 for shift left + vspltw msb,msb,0 # most significant bit extend + vsl H,H,sl1 # H shift left 1 + vand msb,msb,poly + vxor zero,zero,zero + vxor H_t,H,msb + + vsldoi H,H_t,H_t,8 # doubleword swap + vsldoi Hh,H,zero,8 + vsldoi Hl,zero,H,8 + + # --- calculate H^2 = H*H --- + + # reduction pre-processing + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + # polynomial multiplication "classical" + vpmsumd H_h,H_t,Hh # H^1h*H^1h + vpmsumd H_l,H_t,Hl # H^1l*H^1l + vpmsumd H_m,H_t,H # H^1h*H^1l⊕H^1l*H^1h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] + vsldoi Ml,H_m,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor H_h,H_h,Mh # [2] + vxor H_l,H_l,Ml # [2] + vxor H_l,H_l,RP # [1] + + # reduction second phase + vpmsumd RP,H_l,poly_l + vxor H_h,H_l,H_h + vxor H2_t,H_h,RP + + vsldoi H2,H2_t,H2_t,8 + vsldoi H2h,H2,zero,8 + vsldoi H2l,zero,H2,8 + + # --- calculate [H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store H,[H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + li 8,0x00 + li 9,0x100 + li 10,0x200 + stxvd2x HlX,8,TABLE + stxvd2x HX,9,TABLE + stxvd2x HhX,10,TABLE + + li 8,0x300 + li 9,0x400 + li 10,0x500 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^3,H^4 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^2l + vpmsumd H_m,H_t,H2 # H^1h*H^2l⊕H^1l*H^2h + vpmsumd H_h,H_t,H2h # H^1h*H^2h + vpmsumd H2_l,H2_t,H2l # H^2l*H^2l + vpmsumd H2_m,H2_t,H2 # H^2h*H^2l⊕H^2l*H^2h + vpmsumd H2_h,H2_t,H2h # H^2h*H^2h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^3 + vpmsumd RP2,H2_l,poly_h # [1] H^4 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^3 + vsldoi M2h,zero,H2_m,8 # [2] H^4 + vsldoi Ml,H_m,zero,8 # [2] H^3 + vsldoi M2l,H2_m,zero,8 # [2] H^4 + vsldoi RP,RP,RP,8 # [1] H^3 + vsldoi RP2,RP2,RP2,8 # [1] H^4 + vxor H_h,H_h,Mh # [2] H^3 + vxor H2_h,H2_h,M2h # [2] H^4 + vxor H_l,H_l,Ml # [2] H^3 + vxor H2_l,H2_l,M2l # [2] H^4 + vxor H_l,H_l,RP # [1] H^3 + vxor H2_l,H2_l,RP2 # [1] H^4 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^3 + vpmsumd RP2,H2_l,poly_l # H^4 + vxor H_h,H_l,H_h # H^3 + vxor H2_h,H2_l,H2_h # H^4 + vxor H_h,H_h,RP # H^3 + vxor H2_h,H2_h,RP2 # H^4 + + vsldoi H2,H2_h,H2_h,8 # H^4 + vsldoi H,H_h,H_h,8 # H^3 + vsldoi H2l,zero,H2,8 # H^4 + vsldoi H2h,H2,zero,8 # H^4 + + # --- calculate [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + li 8,0x600 + li 9,0x700 + li 10,0x800 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^5,H^6 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^4l + vpmsumd H_m,H_t,H2 # H^1h*H^4l⊕H^1l*H^4h + vpmsumd H_h,H_t,H2h # H^1h*H^4h + vpmsumd H2_l,H2_t,H2l # H^2l*H^4l + vpmsumd H2_m,H2_t,H2 # H^2h*H^4l⊕H^2l*H^4h + vpmsumd H2_h,H2_t,H2h # H^2h*H^4h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^5 + vpmsumd RP2,H2_l,poly_h # [1] H^6 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^5 + vsldoi M2h,zero,H2_m,8 # [2] H^6 + vsldoi Ml,H_m,zero,8 # [2] H^5 + vsldoi M2l,H2_m,zero,8 # [2] H^6 + vsldoi RP,RP,RP,8 # [1] H^5 + vsldoi RP2,RP2,RP2,8 # [1] H^6 + vxor H_h,H_h,Mh # [2] H^5 + vxor H2_h,H2_h,M2h # [2] H^6 + vxor H_l,H_l,Ml # [2] H^5 + vxor H2_l,H2_l,M2l # [2] H^6 + vxor H_l,H_l,RP # [1] H^5 + vxor H2_l,H2_l,RP2 # [1] H^6 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^5 + vpmsumd RP2,H2_l,poly_l # H^6 + vxor H_h,H_l,H_h # H^5 + vxor H2_h,H2_l,H2_h # H^6 + vxor H_h,H_h,RP # H^5 + vxor H2_h,H2_h,RP2 # H^6 + + vsldoi H2,H2_h,H2_h,8 # H^6 + vsldoi H,H_h,H_h,8 # H^5 + vsldoi H2l,zero,H2,8 # H^6 + vsldoi H2h,H2,zero,8 # H^6 + + # --- calculate [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + li 8,0x900 + li 9,0xA00 + li 10,0xB00 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^7,H^8 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^6l + vpmsumd H_m,H_t,H2 # H^1h*H^6l⊕H^1l*H^6h + vpmsumd H_h,H_t,H2h # H^1h*H^6h + vpmsumd H2_l,H2_t,H2l # H^2l*H^6l + vpmsumd H2_m,H2_t,H2 # H^2h*H^6l⊕H^2l*H^6h + vpmsumd H2_h,H2_t,H2h # H^2h*H^6h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^7 + vpmsumd RP2,H2_l,poly_h # [1] H^8 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^7 + vsldoi M2h,zero,H2_m,8 # [2] H^8 + vsldoi Ml,H_m,zero,8 # [2] H^7 + vsldoi M2l,H2_m,zero,8 # [2] H^8 + vsldoi RP,RP,RP,8 # [1] H^7 + vsldoi RP2,RP2,RP2,8 # [1] H^8 + vxor H_h,H_h,Mh # [2] H^7 + vxor H2_h,H2_h,M2h # [2] H^8 + vxor H_l,H_l,Ml # [2] H^7 + vxor H2_l,H2_l,M2l # [2] H^8 + vxor H_l,H_l,RP # [1] H^7 + vxor H2_l,H2_l,RP2 # [1] H^8 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^7 + vpmsumd RP2,H2_l,poly_l # H^8 + vxor H_h,H_l,H_h # H^7 + vxor H2_h,H2_l,H2_h # H^8 + vxor H_h,H_h,RP # H^7 + vxor H2_h,H2_h,RP2 # H^8 + + vsldoi H,H_h,H_h,8 # H^7 + vsldoi H2,H2_h,H2_h,8 # H^8 + + # --- calculate [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + li 8,0xC00 + li 9,0xD00 + li 10,0xE00 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + blr +EPILOGUE(_nettle_gcm_init_key8) + + # void gcm_hash (const struct gcm_key *key, union gcm_block *x, + # size_t length, const uint8_t *data) + +.align 5 +PROLOGUE(_nettle_gcm_hash8) + vxor zero,zero,zero + + ld 7,.polynomial@got(TOCP) + lvx poly,0,7 + ld 7,.swap_mask@got(TOCP) + lvx swap_mask,0,7 + ld 7,.hidw_mask@got(TOCP) + lvx hidw_mask,0,7 + ld 7,.lodw_mask@got(TOCP) + lvx lodw_mask,0,7 + + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + lxvd2x CX,0,X # load X + vperm C,C,C,swap_mask + + srdi 7,LENGTH,7 # 8x loop count + cmpldi 7,0 + beq L2x + + # backup registers + stdu SP,-224(SP) + std 28,216(SP) + std 29,208(SP) + std 30,200(SP) + std 31,192(SP) + li 8,176 + stvx 20,8,SP + subi 8,8,16 + stvx 21,8,SP + subi 8,8,16 + stvx 22,8,SP + subi 8,8,16 + stvx 23,8,SP + subi 8,8,16 + stvx 24,8,SP + subi 8,8,16 + stvx 25,8,SP + subi 8,8,16 + stvx 26,8,SP + subi 8,8,16 + stvx 27,8,SP + subi 8,8,16 + stvx 28,8,SP + subi 8,8,16 + stvx 29,8,SP + subi 8,8,16 + stvx 30,8,SP + subi 8,8,16 + stvx 31,8,SP + + # table loading + li 8,0x300 + li 9,0x400 + li 10,0x500 + lxvd2x H21hX,8,TABLE + lxvd2x H21X,9,TABLE + lxvd2x H21lX,10,TABLE + li 8,0x600 + li 9,0x700 + li 10,0x800 + lxvd2x H43hX,8,TABLE + lxvd2x H43X,9,TABLE + lxvd2x H43lX,10,TABLE + li 8,0x900 + li 9,0xA00 + li 10,0xB00 + lxvd2x H65hX,8,TABLE + lxvd2x H65X,9,TABLE + lxvd2x H65lX,10,TABLE + li 8,0xC00 + li 9,0xD00 + li 10,0xE00 + lxvd2x H87hX,8,TABLE + lxvd2x H87X,9,TABLE + lxvd2x H87lX,10,TABLE + + li 8,0x10 + li 9,0x20 + li 10,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + + mtctr 7 +.align 5 +L8x_loop: + # input loading + lxvd2x C0X,0,DATA # load C0 + lxvd2x C1X,8,DATA # load C1 + lxvd2x C2X,9,DATA # load C2 + lxvd2x C3X,10,DATA # load C3 + + # swap permuting + vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask + vperm C2,C2,C2,swap_mask + vperm C3,C3,C3,swap_mask + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C23h,C2,C3,hidw_mask + vperm C23l,C2,C3,lodw_mask + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + + # input loading + lxvd2x C4X,28,DATA # load C4 + lxvd2x C5X,29,DATA # load C5 + lxvd2x C6X,30,DATA # load C6 + lxvd2x C7X,31,DATA # load C7 + + # swap permuting + vperm C4,C4,C4,swap_mask + vperm C5,C5,C5,swap_mask + vperm C6,C6,C6,swap_mask + vperm C7,C7,C7,swap_mask + + # polynomial multiplication "karatsuba" pre-processing + vperm C45h,C4,C5,hidw_mask + vperm C45l,C4,C5,lodw_mask + vperm C67h,C6,C7,hidw_mask + vperm C67l,C6,C7,lodw_mask + vxor C23,C23h,C23l + vxor C01,C01h,C01l + vxor C45,C45h,C45l + vxor C67,C67h,C67l + + # polynomial multiplication "karatsuba" + vpmsumd C23h,C23h,H65h # H23 = H^6h*C2h⊕H^5h*C3h + vpmsumd C23l,C23l,H65l # L23 = H^6l*C2l⊕H^5l*C3l + vpmsumd C01h,C01h,H87h # H01 = H^8h*C0h⊕H^7h*C1h + vpmsumd C01l,C01l,H87l # L01 = H^8l*C0l⊕H^7l*C1l + vpmsumd C67h,C67h,H21h # H67 = H^2h*C6h⊕H^1h*C7h + vpmsumd C67l,C67l,H21l # L67 = H^2l*C6l⊕H^1l*C7l + vpmsumd C45h,C45h,H43h # H45 = H^4h*C4h⊕H^3h*C5h + vpmsumd C45l,C45l,H43l # L45 = H^4l*C4l⊕H^3l*C5l + vpmsumd C23,C23,H65 # M23 = (H^6h⊕H^5h)*(C2h⊕C3h)⊕(H^6l⊕H^5l)*(C2l⊕C3l) + vpmsumd C01,C01,H87 # M01 = (H^8h⊕H^7h)*(C0h⊕C1h)⊕(H^8l⊕H^7l)*(C0l⊕C1l) + vpmsumd C45,C45,H43 # M45 = (H^4h⊕H^3h)*(C4h⊕C5h)⊕(H^4l⊕H^3l)*(C4l⊕C5l) + vpmsumd C67,C67,H21 # M67 = (H^2h⊕H^1h)*(C6h⊕C7h)⊕(H^2l⊕H^1l)*(C6l⊕C7l) + + # polynomial multiplication "karatsuba" post-processing + vxor C23,C23,C23h + vxor C01,C01,C01h + vxor C45,C45,C45h + vxor C67,C67,C67h + vxor C23,C23,C23l + vxor C01,C01,C01l + vxor C45,C45,C45l + vxor C67,C67,C67l + + # deferred recombination of partial products + vxor C01h,C01h,C23h # H0 = H01⊕H23 + vxor C45h,C45h,C67h # H1 = H45⊕H67 + vxor C01l,C01l,C23l # L0 = L01⊕L23 + vxor C45l,C45l,C67l # L1 = L45⊕L45 + vxor C01,C01,C23 # M0 = M01⊕M23 + vxor C45,C45,C67 # M1 = M45⊕M45 + vxor C01h,C01h,C45h # H = H0⊕H1 + vxor C01l,C01l,C45l # L = L0⊕L1 + vxor C01,C01,C45 # M = M0⊕M1 + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x80 + bdnz L8x_loop + + # restore registers + li 8,0 + lvx 31,8,SP + addi 8,8,16 + lvx 30,8,SP + addi 8,8,16 + lvx 29,8,SP + addi 8,8,16 + lvx 28,8,SP + addi 8,8,16 + lvx 27,8,SP + addi 8,8,16 + lvx 26,8,SP + addi 8,8,16 + lvx 25,8,SP + addi 8,8,16 + lvx 24,8,SP + addi 8,8,16 + lvx 23,8,SP + addi 8,8,16 + lvx 22,8,SP + addi 8,8,16 + lvx 21,8,SP + addi 8,8,16 + lvx 20,8,SP + ld 31,192(SP) + ld 30,200(SP) + ld 29,208(SP) + ld 28,216(SP) + addi SP,SP,224 + + clrldi LENGTH,LENGTH,57 +L2x: + srdi 7,LENGTH,5 + cmpldi 7,0 + beq L1x + + # table loading + li 8,0x300 + li 9,0x400 + li 10,0x500 + lxvd2x H21hX,8,TABLE + lxvd2x H21X,9,TABLE + lxvd2x H21lX,10,TABLE + + li 10,0x10 + + mtctr 7 +.align 5 +L2x_loop: + # input loading + lxvd2x C0X,0,DATA # load C0 + lxvd2x C1X,10,DATA # load C1 + + # swap permuting + vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + vxor C01,C01h,C01l + + # polynomial multiplication "karatsuba" + vpmsumd C01h,C01h,H21h # H01 = H^2h*C0h⊕H^1h*C1h + vpmsumd C01l,C01l,H21l # L01 = H^2l*C0l⊕H^1l*C1l + vpmsumd C01,C01,H21 # M01 = (H^2h⊕H^1h)*(C0h⊕C1h)⊕(H^2l⊕H^1l)*(C0l⊕C1l) + + # polynomial multiplication "karatsuba" post-processing + vxor C01,C01,C01h + vxor C01,C01,C01l + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x20 + bdnz L2x_loop + + clrldi LENGTH,LENGTH,59 +L1x: + srdi 7,LENGTH,4 + cmpldi 7,0 + beq Lrem + + # table loading + li 9,0x100 + li 10,0x200 + lxvd2x HlX,0,TABLE + lxvd2x HX, 9,TABLE + lxvd2x HhX,10,TABLE + + # input loading + lxvd2x C0X,0,DATA # load C0 + + # swap permuting + vperm C0,C0,C0,swap_mask + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + + addi DATA,DATA,0x10 + clrldi LENGTH,LENGTH,60 +Lrem: + cmpldi LENGTH,0 + beq Ldone + + # table loading + li 9,0x100 + li 10,0x200 + lxvd2x HlX,0,TABLE + lxvd2x HX, 9,TABLE + lxvd2x HhX,10,TABLE + + # input loading + stdu SP,-16(SP) + stvx zero,0,SP +Lst_loop: + subic. LENGTH,LENGTH,1 + lbzx 7,LENGTH,DATA + stbx 7,LENGTH,SP + bne Lst_loop + lxvd2x C0X,0,SP + addi SP,SP,16 + + # swap permuting + vperm C0,C0,C0,swap_mask + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + +Ldone: + vperm C,C,C,swap_mask + stxvd2x CX,0,X # store C + blr +EPILOGUE(_nettle_gcm_hash8) + + # gcm_fill (uint8_t *ctr, size_t blocks, union gcm_block *buffer) + +.align 5 +PROLOGUE(_nettle_gcm_fill) + ld 6,.swap_mask@got(TOCP) + lvx swap_mask,0,6 + + vxor zero,zero,zero + vspltisb I1,1 + vspltisb I2,2 + vspltisb I3,3 + vspltisb I4,4 + vspltisb I5,5 + vspltisb I6,6 + vspltisb I7,7 + vspltisb I8,8 + vsldoi I1,zero,I1,1 + vsldoi I2,zero,I2,1 + vsldoi I3,zero,I3,1 + vsldoi I4,zero,I4,1 + vsldoi I5,zero,I5,1 + vsldoi I6,zero,I6,1 + vsldoi I7,zero,I7,1 + vsldoi I8,zero,I8,1 + + lxvd2x CTR0X,0,CTR + vperm CTR0,CTR0,CTR0,swap_mask + + srdi 6,BLOCKS,3 # 8x loop count + cmpldi 6,0 + beq Lfill_4x + + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 25,0x10 + li 26,0x20 + li 27,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + + mtctr 6 +L8x_fill_loop: + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + vadduwm CTR4,CTR0,I4 + vadduwm CTR5,CTR0,I5 + vadduwm CTR6,CTR0,I6 + vadduwm CTR7,CTR0,I7 + + vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask + vperm CTR4,CTR4,CTR4,swap_mask + vperm CTR5,CTR5,CTR5,swap_mask + vperm CTR6,CTR6,CTR6,swap_mask + vperm CTR7,CTR7,CTR7,swap_mask + + stxvd2x CTR0SX,0,BUFFER + stxvd2x CTR1X,25,BUFFER + stxvd2x CTR2X,26,BUFFER + stxvd2x CTR3X,27,BUFFER + stxvd2x CTR4X,28,BUFFER + stxvd2x CTR5X,29,BUFFER + stxvd2x CTR6X,30,BUFFER + stxvd2x CTR7X,31,BUFFER + + vadduwm CTR0,CTR0,I8 + addi BUFFER,BUFFER,0x80 + bdnz L8x_fill_loop + + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi BLOCKS,BLOCKS,61 + +Lfill_4x: + srdi 6,BLOCKS,2 + cmpldi 6,0 + beq Lfill_2x + + li 8,0x10 + li 9,0x20 + li 10,0x30 + + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + + vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask + + stxvd2x CTR0SX,0,BUFFER + stxvd2x CTR1X,8,BUFFER + stxvd2x CTR2X,9,BUFFER + stxvd2x CTR3X,10,BUFFER + + vadduwm CTR0,CTR0,I4 + addi BUFFER,BUFFER,0x40 + + clrldi BLOCKS,BLOCKS,62 + +Lfill_2x: + srdi 6,BLOCKS,1 + cmpldi 6,0 + beq Lfill_1x + + li 10,0x10 + + vadduwm CTR1,CTR0,I1 + + vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + + stxvd2x CTR0SX,0,BUFFER + stxvd2x CTR1X,10,BUFFER + + vadduwm CTR0,CTR0,I2 + addi BUFFER,BUFFER,0x20 + + clrldi BLOCKS,BLOCKS,63 + +Lfill_1x: + cmpldi BLOCKS,0 + beq Lfill_done + + vperm CTR0S,CTR0,CTR0,swap_mask + + stxvd2x CTR0SX,0,BUFFER + + vadduwm CTR0,CTR0,I1 + +Lfill_done: + vperm CTR0,CTR0,CTR0,swap_mask + stxvd2x CTR0X,0,CTR + + blr +EPILOGUE(_nettle_gcm_fill) + + .data + .align 4 +.polynomial: + .byte 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0xc2 + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 + .align 4 +.hidw_mask: + .byte 23,22,21,20,19,18,17,16,7,6,5,4,3,2,1,0 + .align 4 +.lodw_mask: + .byte 31,30,29,28,27,26,25,24,15,14,13,12,11,10,9,8 diff --git a/powerpc64le/machine.m4 b/powerpc64le/machine.m4 new file mode 100644 index 00000000..e69de29b diff --git a/testsuite/gcm-test.c b/testsuite/gcm-test.c index c8174019..df1fc94a 100644 --- a/testsuite/gcm-test.c +++ b/testsuite/gcm-test.c @@ -170,6 +170,29 @@ test_main(void) "16aedbf5a0de6a57a637b39b"), SHEX("619cc5aefffe0bfa462af43c1699d050")); + /* Test 128 bytes */ + test_aead(&nettle_gcm_aes128, NULL, + SHEX("feffe9928665731c6d6a8f9467308308"), + SHEX(""), + SHEX("d9313225f88406e5a55909c5aff5269a" + "86a7a9531534f7da2e4c303d8a318a72" + "1c3c0c95956809532fcf0e2449a6b525" + "b16aedf5aa0de657ba637b391aafd255" + "5ae376bc5e9f6a1b08e34db7a6ee0736" + "9ba662ea12f6f197e6bc3ed69d2480f3" + "ea5691347f2ba69113eb37910ebc18c8" + "0f697234582016fa956ca8f63ae6b473"), + SHEX("42831ec2217774244b7221b784d0d49c" + "e3aa212f2c02a4e035c17e2329aca12e" + "21d514b25466931c7d8f6a5aac84aa05" + "1ba30b396a0aac973d58e091473f5985" + "874b1178906ddbeab04ab2fe6cce8c57" + "8d7e961bd13fd6a8c56b66ca5e576492" + "1a48cd8bda04e66343e73055118b69b9" + "ced486813846958a11e602c03cfc232b"), + SHEX("cafebabefacedbaddecaf888"), + SHEX("796836f1246c9d735c5e1be0a715ccc3")); + /* Test case 7 */ test_aead(&nettle_gcm_aes192, NULL, SHEX("00000000000000000000000000000000"

1 0

[PATCH] optimized AES and GHASH for PPC64LE
by Maamoun TK 18 Jun '20

18 Jun '20

diff --git a/configure.ac b/configure.ac index 90ea1ea8..1ea54ce8 100644 --- a/configure.ac +++ b/configure.ac @@ -435,6 +435,9 @@ if test "x$enable_assembler" = xyes ; then esac fi ;; + *powerpc64le*) + asm_path=powerpc64le + ;; *) enable_assembler=no ;; @@ -572,7 +575,9 @@ AH_VERBATIM([HAVE_NATIVE], #undef HAVE_NATIVE_ecc_secp384r1_redc #undef HAVE_NATIVE_ecc_secp521r1_modp #undef HAVE_NATIVE_ecc_secp521r1_redc +#undef HAVE_NATIVE_gcm_init_key8 #undef HAVE_NATIVE_gcm_hash8 +#undef HAVE_NATIVE_gcm_fill #undef HAVE_NATIVE_salsa20_core #undef HAVE_NATIVE_sha1_compress #undef HAVE_NATIVE_sha256_compress diff --git a/gcm.c b/gcm.c index cf615daf..809c03bc 100644 --- a/gcm.c +++ b/gcm.c @@ -140,6 +140,12 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) memcpy (x->b, Z.b, sizeof(Z)); } # elif GCM_TABLE_BITS == 8 +# if HAVE_NATIVE_gcm_init_key8 + +#define gcm_init_key _nettle_gcm_init_key8 +void +_nettle_gcm_init_key8 (union nettle_block16 *table); +# endif /* HAVE_NATIVE_gcm_init_key8 */ # if HAVE_NATIVE_gcm_hash8 #define gcm_hash _nettle_gcm_hash8 @@ -225,6 +231,13 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) #endif /* GCM_TABLE_BITS */ +#if HAVE_NATIVE_gcm_fill + +#define gcm_fill _nettle_gcm_fill +void +_nettle_gcm_fill (uint8_t *ctr, size_t blocks, union nettle_block16 *buffer); +#endif /* HAVE_NATIVE_gcm_fill */ + /* Increment the rightmost 32 bits. */ #define INC32(block) INCREMENT(4, (block.b) + GCM_BLOCK_SIZE - 4) @@ -245,7 +258,9 @@ gcm_set_key(struct gcm_key *key, memset(key->h[0].b, 0, GCM_BLOCK_SIZE); f (cipher, GCM_BLOCK_SIZE, key->h[i].b, key->h[0].b); -#if GCM_TABLE_BITS +#ifdef gcm_init_key + gcm_init_key(key->h); +#elif GCM_TABLE_BITS /* Algorithm 3 from the gcm paper. First do powers of two, then do the rest by adding. */ while (i /= 2) @@ -333,6 +348,7 @@ gcm_update(struct gcm_ctx *ctx, const struct gcm_key *key, ctx->auth_size += length; } +#ifndef gcm_fill static nettle_fill16_func gcm_fill; static void gcm_fill(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) @@ -349,6 +365,7 @@ gcm_fill(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) WRITE_UINT32(ctr + GCM_BLOCK_SIZE - 4, c); } +#endif /* !gcm_fill */ void gcm_encrypt (struct gcm_ctx *ctx, const struct gcm_key *key, diff --git a/powerpc64le/aes-decrypt-internal.asm b/powerpc64le/aes-decrypt-internal.asm index e69de29b..bde34779 100755 --- a/powerpc64le/aes-decrypt-internal.asm +++ b/powerpc64le/aes-decrypt-internal.asm @@ -0,0 +1,573 @@ +C powerpc64le/aes-decrypt-internal.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<ROUNDS>, <3>) +define(<KEYS>, <4>) +define(<LENGTH>, <6>) +define(<DST>, <7>) +define(<SRC>, <8>) + +define(<swap_mask>, <0>) + +define(<K>, <1>) +define(<S0>, <2>) +define(<S1>, <3>) +define(<S2>, <4>) +define(<S3>, <5>) +define(<S4>, <6>) +define(<S5>, <7>) +define(<S6>, <8>) +define(<S7>, <9>) +define(<S8>, <10>) +define(<S9>, <11>) +define(<S10>, <12>) +define(<S11>, <13>) +define(<S12>, <14>) +define(<S13>, <15>) +define(<S14>, <16>) +define(<S15>, <17>) + +define(<KX>, <33>) +define(<S0X>, <34>) +define(<S1X>, <35>) +define(<S2X>, <36>) +define(<S3X>, <37>) +define(<S4X>, <38>) +define(<S5X>, <39>) +define(<S6X>, <40>) +define(<S7X>, <41>) +define(<S8X>, <42>) +define(<S9X>, <43>) +define(<S10X>, <44>) +define(<S11X>, <45>) +define(<S12X>, <46>) +define(<S13X>, <47>) +define(<S14X>, <48>) +define(<S15X>, <49>) + +C ZERO vector register is used in place of RoundKey +C for vncipher instruction because the order of InvMixColumns +C and Xor processes are flipped in that instruction. +C The Xor process with RoundKey is executed afterward. +define(<ZERO>, <18>) + + .file "aes-decrypt-internal.asm" + + C _aes_decrypt(unsigned rounds, const uint32_t *keys, + C const struct aes_table *T, + C size_t length, uint8_t *dst, + C uint8_t *src) + + .text +.align 5 +PROLOGUE(_nettle_aes_decrypt) + vxor ZERO,ZERO,ZERO + + ld 5,.swap_mask@got(TOCP) + lvx swap_mask,0,5 + + subi ROUNDS,ROUNDS,1 + srdi LENGTH,LENGTH,4 + + srdi 5,LENGTH,4 # 16x loop count + cmpldi 5,0 + beq L8x + + std 17,-120(SP); + std 18,-112(SP); + std 19,-104(SP); + std 20,-96(SP); + std 21,-88(SP); + std 22,-80(SP); + std 23,-72(SP); + std 24,-64(SP); + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 17,0x10 + li 18,0x20 + li 19,0x30 + li 20,0x40 + li 21,0x50 + li 22,0x60 + li 23,0x70 + li 24,0x80 + li 25,0x90 + li 26,0xA0 + li 27,0xB0 + li 28,0xC0 + li 29,0xD0 + li 30,0xE0 + li 31,0xF0 + +.align 5 +Lx16_loop: + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + lxvd2x S1X,17,SRC + lxvd2x S2X,18,SRC + lxvd2x S3X,19,SRC + lxvd2x S4X,20,SRC + lxvd2x S5X,21,SRC + lxvd2x S6X,22,SRC + lxvd2x S7X,23,SRC + lxvd2x S8X,24,SRC + lxvd2x S9X,25,SRC + lxvd2x S10X,26,SRC + lxvd2x S11X,27,SRC + lxvd2x S12X,28,SRC + lxvd2x S13X,29,SRC + lxvd2x S14X,30,SRC + lxvd2x S15X,31,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L16x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vncipher S4,S4,ZERO + vncipher S5,S5,ZERO + vncipher S6,S6,ZERO + vncipher S7,S7,ZERO + vncipher S8,S8,ZERO + vncipher S9,S9,ZERO + vncipher S10,S10,ZERO + vncipher S11,S11,ZERO + vncipher S12,S12,ZERO + vncipher S13,S13,ZERO + vncipher S14,S14,ZERO + vncipher S15,S15,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + addi 10,10,0x10 + bdnz L16x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + vncipherlast S4,S4,K + vncipherlast S5,S5,K + vncipherlast S6,S6,K + vncipherlast S7,S7,K + vncipherlast S8,S8,K + vncipherlast S9,S9,K + vncipherlast S10,S10,K + vncipherlast S11,S11,K + vncipherlast S12,S12,K + vncipherlast S13,S13,K + vncipherlast S14,S14,K + vncipherlast S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + stxvd2x S0X,0,DST + stxvd2x S1X,17,DST + stxvd2x S2X,18,DST + stxvd2x S3X,19,DST + stxvd2x S4X,20,DST + stxvd2x S5X,21,DST + stxvd2x S6X,22,DST + stxvd2x S7X,23,DST + stxvd2x S8X,24,DST + stxvd2x S9X,25,DST + stxvd2x S10X,26,DST + stxvd2x S11X,27,DST + stxvd2x S12X,28,DST + stxvd2x S13X,29,DST + stxvd2x S14X,30,DST + stxvd2x S15X,31,DST + + addi SRC,SRC,0x100 + addi DST,DST,0x100 + subic. 5,5,1 + bne Lx16_loop + + ld 17,-120(SP); + ld 18,-112(SP); + ld 19,-104(SP); + ld 20,-96(SP); + ld 21,-88(SP); + ld 22,-80(SP); + ld 23,-72(SP); + ld 24,-64(SP); + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi LENGTH,LENGTH,60 + +L8x: + srdi 5,LENGTH,3 + cmpldi 5,0 + beq L4x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + addi 9,9,0x10 + lxvd2x S4X,9,SRC + addi 9,9,0x10 + lxvd2x S5X,9,SRC + addi 9,9,0x10 + lxvd2x S6X,9,SRC + addi 9,9,0x10 + lxvd2x S7X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L8x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vncipher S4,S4,ZERO + vncipher S5,S5,ZERO + vncipher S6,S6,ZERO + vncipher S7,S7,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + addi 10,10,0x10 + bdnz L8x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + vncipherlast S4,S4,K + vncipherlast S5,S5,K + vncipherlast S6,S6,K + vncipherlast S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + addi 9,9,0x10 + stxvd2x S4X,9,DST + addi 9,9,0x10 + stxvd2x S5X,9,DST + addi 9,9,0x10 + stxvd2x S6X,9,DST + addi 9,9,0x10 + stxvd2x S7X,9,DST + + addi SRC,SRC,0x80 + addi DST,DST,0x80 + + clrldi LENGTH,LENGTH,61 + +L4x: + srdi 5,LENGTH,2 + cmpldi 5,0 + beq L2x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L4x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + addi 10,10,0x10 + bdnz L4x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + + addi SRC,SRC,0x40 + addi DST,DST,0x40 + + clrldi LENGTH,LENGTH,62 + +L2x: + srdi 5,LENGTH,1 + cmpldi 5,0 + beq L1x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L2x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vxor S0,S0,K + vxor S1,S1,K + addi 10,10,0x10 + bdnz L2x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + + addi SRC,SRC,0x20 + addi DST,DST,0x20 + + clrldi LENGTH,LENGTH,63 + +L1x: + cmpldi LENGTH,0 + beq Ldone + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + + vxor S0,S0,K + + vperm S0,S0,S0,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L1x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vxor S0,S0,K + addi 10,10,0x10 + bdnz L1x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + + vperm S0,S0,S0,swap_mask + + stxvd2x S0X,0,DST + +Ldone: + blr +EPILOGUE(_nettle_aes_decrypt) + + .data + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 diff --git a/powerpc64le/aes-encrypt-internal.asm b/powerpc64le/aes-encrypt-internal.asm index e69de29b..1bbd86a8 100755 --- a/powerpc64le/aes-encrypt-internal.asm +++ b/powerpc64le/aes-encrypt-internal.asm @@ -0,0 +1,534 @@ +C powerpc64le/aes-encrypt-internal.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<ROUNDS>, <3>) +define(<KEYS>, <4>) +define(<LENGTH>, <6>) +define(<DST>, <7>) +define(<SRC>, <8>) + +define(<swap_mask>, <0>) + +define(<K>, <1>) +define(<S0>, <2>) +define(<S1>, <3>) +define(<S2>, <4>) +define(<S3>, <5>) +define(<S4>, <6>) +define(<S5>, <7>) +define(<S6>, <8>) +define(<S7>, <9>) +define(<S8>, <10>) +define(<S9>, <11>) +define(<S10>, <12>) +define(<S11>, <13>) +define(<S12>, <14>) +define(<S13>, <15>) +define(<S14>, <16>) +define(<S15>, <17>) + +define(<KX>, <33>) +define(<S0X>, <34>) +define(<S1X>, <35>) +define(<S2X>, <36>) +define(<S3X>, <37>) +define(<S4X>, <38>) +define(<S5X>, <39>) +define(<S6X>, <40>) +define(<S7X>, <41>) +define(<S8X>, <42>) +define(<S9X>, <43>) +define(<S10X>, <44>) +define(<S11X>, <45>) +define(<S12X>, <46>) +define(<S13X>, <47>) +define(<S14X>, <48>) +define(<S15X>, <49>) + + .file "aes-encrypt-internal.asm" + + C _aes_encrypt(unsigned rounds, const uint32_t *keys, + C const struct aes_table *T, + C size_t length, uint8_t *dst, + C uint8_t *src) + + .text +.align 5 +PROLOGUE(_nettle_aes_encrypt) + ld 5,.swap_mask@got(TOCP) + lvx swap_mask,0,5 + + subi ROUNDS,ROUNDS,1 + srdi LENGTH,LENGTH,4 + + srdi 5,LENGTH,4 # 16x loop count + cmpldi 5,0 + beq L8x + + std 17,-120(SP); + std 18,-112(SP); + std 19,-104(SP); + std 20,-96(SP); + std 21,-88(SP); + std 22,-80(SP); + std 23,-72(SP); + std 24,-64(SP); + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 17,0x10 + li 18,0x20 + li 19,0x30 + li 20,0x40 + li 21,0x50 + li 22,0x60 + li 23,0x70 + li 24,0x80 + li 25,0x90 + li 26,0xA0 + li 27,0xB0 + li 28,0xC0 + li 29,0xD0 + li 30,0xE0 + li 31,0xF0 + +.align 5 +Lx16_loop: + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + lxvd2x S1X,17,SRC + lxvd2x S2X,18,SRC + lxvd2x S3X,19,SRC + lxvd2x S4X,20,SRC + lxvd2x S5X,21,SRC + lxvd2x S6X,22,SRC + lxvd2x S7X,23,SRC + lxvd2x S8X,24,SRC + lxvd2x S9X,25,SRC + lxvd2x S10X,26,SRC + lxvd2x S11X,27,SRC + lxvd2x S12X,28,SRC + lxvd2x S13X,29,SRC + lxvd2x S14X,30,SRC + lxvd2x S15X,31,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L16x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + vcipher S4,S4,K + vcipher S5,S5,K + vcipher S6,S6,K + vcipher S7,S7,K + vcipher S8,S8,K + vcipher S9,S9,K + vcipher S10,S10,K + vcipher S11,S11,K + vcipher S12,S12,K + vcipher S13,S13,K + vcipher S14,S14,K + vcipher S15,S15,K + addi 10,10,0x10 + bdnz L16x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + vcipherlast S4,S4,K + vcipherlast S5,S5,K + vcipherlast S6,S6,K + vcipherlast S7,S7,K + vcipherlast S8,S8,K + vcipherlast S9,S9,K + vcipherlast S10,S10,K + vcipherlast S11,S11,K + vcipherlast S12,S12,K + vcipherlast S13,S13,K + vcipherlast S14,S14,K + vcipherlast S15,S15,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask + + stxvd2x S0X,0,DST + stxvd2x S1X,17,DST + stxvd2x S2X,18,DST + stxvd2x S3X,19,DST + stxvd2x S4X,20,DST + stxvd2x S5X,21,DST + stxvd2x S6X,22,DST + stxvd2x S7X,23,DST + stxvd2x S8X,24,DST + stxvd2x S9X,25,DST + stxvd2x S10X,26,DST + stxvd2x S11X,27,DST + stxvd2x S12X,28,DST + stxvd2x S13X,29,DST + stxvd2x S14X,30,DST + stxvd2x S15X,31,DST + + addi SRC,SRC,0x100 + addi DST,DST,0x100 + subic. 5,5,1 + bne Lx16_loop + + ld 17,-120(SP); + ld 18,-112(SP); + ld 19,-104(SP); + ld 20,-96(SP); + ld 21,-88(SP); + ld 22,-80(SP); + ld 23,-72(SP); + ld 24,-64(SP); + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi LENGTH,LENGTH,60 + +L8x: + srdi 5,LENGTH,3 + cmpldi 5,0 + beq L4x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + addi 9,9,0x10 + lxvd2x S4X,9,SRC + addi 9,9,0x10 + lxvd2x S5X,9,SRC + addi 9,9,0x10 + lxvd2x S6X,9,SRC + addi 9,9,0x10 + lxvd2x S7X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L8x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + vcipher S4,S4,K + vcipher S5,S5,K + vcipher S6,S6,K + vcipher S7,S7,K + addi 10,10,0x10 + bdnz L8x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + vcipherlast S4,S4,K + vcipherlast S5,S5,K + vcipherlast S6,S6,K + vcipherlast S7,S7,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + addi 9,9,0x10 + stxvd2x S4X,9,DST + addi 9,9,0x10 + stxvd2x S5X,9,DST + addi 9,9,0x10 + stxvd2x S6X,9,DST + addi 9,9,0x10 + stxvd2x S7X,9,DST + + addi SRC,SRC,0x80 + addi DST,DST,0x80 + + clrldi LENGTH,LENGTH,61 + +L4x: + srdi 5,LENGTH,2 + cmpldi 5,0 + beq L2x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L4x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + addi 10,10,0x10 + bdnz L4x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + + addi SRC,SRC,0x40 + addi DST,DST,0x40 + + clrldi LENGTH,LENGTH,62 + +L2x: + srdi 5,LENGTH,1 + cmpldi 5,0 + beq L1x + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + + vxor S0,S0,K + vxor S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L2x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + addi 10,10,0x10 + bdnz L2x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + + vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + + addi SRC,SRC,0x20 + addi DST,DST,0x20 + + clrldi LENGTH,LENGTH,63 + +L1x: + cmpldi LENGTH,0 + beq Ldone + + lxvd2x KX,0,KEYS + + lxvd2x S0X,0,SRC + + vxor S0,S0,K + + vperm S0,S0,S0,swap_mask + + mtctr ROUNDS + li 10,0x10 +.align 5 +L1x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + addi 10,10,0x10 + bdnz L1x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + + vperm S0,S0,S0,swap_mask + + stxvd2x S0X,0,DST + +Ldone: + blr +EPILOGUE(_nettle_aes_encrypt) + + .data + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 diff --git a/powerpc64le/gcm-hash8.asm b/powerpc64le/gcm-hash8.asm index e69de29b..a809f6ef 100755 --- a/powerpc64le/gcm-hash8.asm +++ b/powerpc64le/gcm-hash8.asm @@ -0,0 +1,992 @@ +C powerpc64le/gcm-hash8.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: +C VSX instructions is used to load and store data to memory "lxvd2x, stxvd2x" +C instead of VR instructions "lvx, stvx" as a workaround to access unaligned data +C VSX registers are defined with "X" suffix + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<TABLE>, <3>) +define(<X>, <4>) +define(<LENGTH>, <5>) +define(<DATA>, <6>) + +define(<zero>, <0>) +define(<swap_mask>, <1>) +define(<hidw_mask>, <2>) +define(<lodw_mask>, <3>) +define(<poly>, <4>) +define(<poly_h>, <4>) +define(<poly_l>, <5>) +define(<RP>, <6>) +define(<Mh>, <7>) +define(<Ml>, <8>) +define(<H>, <9>) +define(<Hh>, <10>) +define(<Hl>, <11>) +define(<RP2>, <9>) +define(<M2h>, <10>) +define(<M2l>, <11>) + +define(<HX>, <41>) +define(<HhX>, <42>) +define(<HlX>, <43>) +define(<H_HhX>, <44>) +define(<H_HX>, <45>) +define(<H_HlX>, <46>) + +define(<sl1>, <1>) +define(<msb>, <5>) +define(<H2>, <6>) +define(<H2h>, <7>) +define(<H2l>, <8>) +define(<H_h>, <12>) +define(<H_m>, <13>) +define(<H_l>, <14>) +define(<H_Hh>, <12>) +define(<H_H>, <13>) +define(<H_Hl>, <14>) +define(<H_t>, <15>) +define(<H2_h>, <16>) +define(<H2_m>, <17>) +define(<H2_l>, <18>) +define(<H2_t>, <19>) + +define(<C0X>, <38>) +define(<C1X>, <39>) +define(<C2X>, <40>) +define(<C3X>, <44>) +define(<C4X>, <38>) +define(<C5X>, <39>) +define(<C6X>, <40>) +define(<C7X>, <44>) + +define(<CX>, <45>) + +define(<C0>, <6>) +define(<C1>, <7>) +define(<C2>, <8>) +define(<C3>, <12>) +define(<C4>, <6>) +define(<C5>, <7>) +define(<C6>, <8>) +define(<C7>, <12>) + +define(<C>, <13>) + +define(<Ch>, <14>) +define(<Cl>, <15>) +define(<Cm>, <16>) + +define(<C01h>, <14>) +define(<C01l>, <15>) +define(<C01>, <16>) +define(<C23h>, <17>) +define(<C23l>, <18>) +define(<C23>, <19>) +define(<C45h>, <20>) +define(<C45l>, <21>) +define(<C45>, <22>) +define(<C67h>, <6>) +define(<C67l>, <7>) +define(<C67>, <8>) + +define(<H21>, <9>) +define(<H21h>, <10>) +define(<H21l>, <11>) +define(<H43>, <23>) +define(<H43h>, <24>) +define(<H43l>, <25>) +define(<H65>, <26>) +define(<H65h>, <27>) +define(<H65l>, <28>) +define(<H87>, <29>) +define(<H87h>, <30>) +define(<H87l>, <31>) + +define(<H21X>, <41>) +define(<H21hX>, <42>) +define(<H21lX>, <43>) +define(<H43X>, <55>) +define(<H43hX>, <56>) +define(<H43lX>, <57>) +define(<H65X>, <58>) +define(<H65hX>, <59>) +define(<H65lX>, <60>) +define(<H87X>, <61>) +define(<H87hX>, <62>) +define(<H87lX>, <63>) + +# gcm_fill registers: + +define(<CTR>, <3>) +define(<BLOCKS>, <4>) +define(<BUFFER>, <5>) + +define(<CTR0>, <2>) +define(<CTR0S>, <3>) +define(<CTR1>, <4>) +define(<CTR2>, <5>) +define(<CTR3>, <6>) +define(<CTR4>, <7>) +define(<CTR5>, <8>) +define(<CTR6>, <9>) +define(<CTR7>, <10>) + +define(<CTR0X>, <34>) +define(<CTR0SX>, <35>) +define(<CTR1X>, <36>) +define(<CTR2X>, <37>) +define(<CTR3X>, <38>) +define(<CTR4X>, <39>) +define(<CTR5X>, <40>) +define(<CTR6X>, <41>) +define(<CTR7X>, <42>) + +define(<I1>, <11>) +define(<I2>, <12>) +define(<I3>, <13>) +define(<I4>, <14>) +define(<I5>, <15>) +define(<I6>, <16>) +define(<I7>, <17>) +define(<I8>, <18>) + + .file "gcm-hash8.asm" + + # void gcm_init_key (union gcm_block *table) + + .text +.align 5 +PROLOGUE(_nettle_gcm_init_key8) + ld 7,.polynomial@got(TOCP) + lvx poly,0,7 + ld 7,.swap_mask@got(TOCP) + lvx swap_mask,0,7 + ld 7,.hidw_mask@got(TOCP) + lvx hidw_mask,0,7 + ld 7,.lodw_mask@got(TOCP) + lvx lodw_mask,0,7 + + li 10,0x800 + lxvd2x HX,10,TABLE # load H + vperm H,H,H,swap_mask + + # --- calculate H = H shift left 1 modulo polynomial --- + + vupkhsw msb,H # most significant bit word-extend + vspltisb sl1,1 # splat 1 for shift left + vspltw msb,msb,0 # most significant bit extend + vsl H,H,sl1 # H shift left 1 + vand msb,msb,poly + vxor zero,zero,zero + vxor H_t,H,msb + + vsldoi H,H_t,H_t,8 # doubleword swap + vsldoi Hh,H,zero,8 + vsldoi Hl,zero,H,8 + + # --- calculate H^2 = H*H --- + + # reduction pre-processing + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + # polynomial multiplication "classical" + vpmsumd H_h,H_t,Hh # H^1h*H^1h + vpmsumd H_l,H_t,Hl # H^1l*H^1l + vpmsumd H_m,H_t,H # H^1h*H^1l⊕H^1l*H^1h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] + vsldoi Ml,H_m,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor H_h,H_h,Mh # [2] + vxor H_l,H_l,Ml # [2] + vxor H_l,H_l,RP # [1] + + # reduction second phase + vpmsumd RP,H_l,poly_l + vxor H_h,H_l,H_h + vxor H2_t,H_h,RP + + vsldoi H2,H2_t,H2_t,8 + vsldoi H2h,H2,zero,8 + vsldoi H2l,zero,H2,8 + + # --- calculate [H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store H,[H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + li 8,0x00 + li 9,0x100 + li 10,0x200 + stxvd2x HlX,8,TABLE + stxvd2x HX,9,TABLE + stxvd2x HhX,10,TABLE + + li 8,0x300 + li 9,0x400 + li 10,0x500 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^3,H^4 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^2l + vpmsumd H_m,H_t,H2 # H^1h*H^2l⊕H^1l*H^2h + vpmsumd H_h,H_t,H2h # H^1h*H^2h + vpmsumd H2_l,H2_t,H2l # H^2l*H^2l + vpmsumd H2_m,H2_t,H2 # H^2h*H^2l⊕H^2l*H^2h + vpmsumd H2_h,H2_t,H2h # H^2h*H^2h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^3 + vpmsumd RP2,H2_l,poly_h # [1] H^4 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^3 + vsldoi M2h,zero,H2_m,8 # [2] H^4 + vsldoi Ml,H_m,zero,8 # [2] H^3 + vsldoi M2l,H2_m,zero,8 # [2] H^4 + vsldoi RP,RP,RP,8 # [1] H^3 + vsldoi RP2,RP2,RP2,8 # [1] H^4 + vxor H_h,H_h,Mh # [2] H^3 + vxor H2_h,H2_h,M2h # [2] H^4 + vxor H_l,H_l,Ml # [2] H^3 + vxor H2_l,H2_l,M2l # [2] H^4 + vxor H_l,H_l,RP # [1] H^3 + vxor H2_l,H2_l,RP2 # [1] H^4 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^3 + vpmsumd RP2,H2_l,poly_l # H^4 + vxor H_h,H_l,H_h # H^3 + vxor H2_h,H2_l,H2_h # H^4 + vxor H_h,H_h,RP # H^3 + vxor H2_h,H2_h,RP2 # H^4 + + vsldoi H2,H2_h,H2_h,8 # H^4 + vsldoi H,H_h,H_h,8 # H^3 + vsldoi H2l,zero,H2,8 # H^4 + vsldoi H2h,H2,zero,8 # H^4 + + # --- calculate [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + li 8,0x600 + li 9,0x700 + li 10,0x800 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^5,H^6 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^4l + vpmsumd H_m,H_t,H2 # H^1h*H^4l⊕H^1l*H^4h + vpmsumd H_h,H_t,H2h # H^1h*H^4h + vpmsumd H2_l,H2_t,H2l # H^2l*H^4l + vpmsumd H2_m,H2_t,H2 # H^2h*H^4l⊕H^2l*H^4h + vpmsumd H2_h,H2_t,H2h # H^2h*H^4h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^5 + vpmsumd RP2,H2_l,poly_h # [1] H^6 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^5 + vsldoi M2h,zero,H2_m,8 # [2] H^6 + vsldoi Ml,H_m,zero,8 # [2] H^5 + vsldoi M2l,H2_m,zero,8 # [2] H^6 + vsldoi RP,RP,RP,8 # [1] H^5 + vsldoi RP2,RP2,RP2,8 # [1] H^6 + vxor H_h,H_h,Mh # [2] H^5 + vxor H2_h,H2_h,M2h # [2] H^6 + vxor H_l,H_l,Ml # [2] H^5 + vxor H2_l,H2_l,M2l # [2] H^6 + vxor H_l,H_l,RP # [1] H^5 + vxor H2_l,H2_l,RP2 # [1] H^6 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^5 + vpmsumd RP2,H2_l,poly_l # H^6 + vxor H_h,H_l,H_h # H^5 + vxor H2_h,H2_l,H2_h # H^6 + vxor H_h,H_h,RP # H^5 + vxor H2_h,H2_h,RP2 # H^6 + + vsldoi H2,H2_h,H2_h,8 # H^6 + vsldoi H,H_h,H_h,8 # H^5 + vsldoi H2l,zero,H2,8 # H^6 + vsldoi H2h,H2,zero,8 # H^6 + + # --- calculate [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + li 8,0x900 + li 9,0xA00 + li 10,0xB00 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^7,H^8 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^6l + vpmsumd H_m,H_t,H2 # H^1h*H^6l⊕H^1l*H^6h + vpmsumd H_h,H_t,H2h # H^1h*H^6h + vpmsumd H2_l,H2_t,H2l # H^2l*H^6l + vpmsumd H2_m,H2_t,H2 # H^2h*H^6l⊕H^2l*H^6h + vpmsumd H2_h,H2_t,H2h # H^2h*H^6h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^7 + vpmsumd RP2,H2_l,poly_h # [1] H^8 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^7 + vsldoi M2h,zero,H2_m,8 # [2] H^8 + vsldoi Ml,H_m,zero,8 # [2] H^7 + vsldoi M2l,H2_m,zero,8 # [2] H^8 + vsldoi RP,RP,RP,8 # [1] H^7 + vsldoi RP2,RP2,RP2,8 # [1] H^8 + vxor H_h,H_h,Mh # [2] H^7 + vxor H2_h,H2_h,M2h # [2] H^8 + vxor H_l,H_l,Ml # [2] H^7 + vxor H2_l,H2_l,M2l # [2] H^8 + vxor H_l,H_l,RP # [1] H^7 + vxor H2_l,H2_l,RP2 # [1] H^8 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^7 + vpmsumd RP2,H2_l,poly_l # H^8 + vxor H_h,H_l,H_h # H^7 + vxor H2_h,H2_l,H2_h # H^8 + vxor H_h,H_h,RP # H^7 + vxor H2_h,H2_h,RP2 # H^8 + + vsldoi H,H_h,H_h,8 # H^7 + vsldoi H2,H2_h,H2_h,8 # H^8 + + # --- calculate [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + li 8,0xC00 + li 9,0xD00 + li 10,0xE00 + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + blr +EPILOGUE(_nettle_gcm_init_key8) + + # void gcm_hash (const struct gcm_key *key, union gcm_block *x, + # size_t length, const uint8_t *data) + +.align 5 +PROLOGUE(_nettle_gcm_hash8) + vxor zero,zero,zero + + ld 7,.polynomial@got(TOCP) + lvx poly,0,7 + ld 7,.swap_mask@got(TOCP) + lvx swap_mask,0,7 + ld 7,.hidw_mask@got(TOCP) + lvx hidw_mask,0,7 + ld 7,.lodw_mask@got(TOCP) + lvx lodw_mask,0,7 + + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + lxvd2x CX,0,X # load X + vperm C,C,C,swap_mask + + srdi 7,LENGTH,7 # 8x loop count + cmpldi 7,0 + beq L2x + + # backup registers + stdu SP,-224(SP) + std 28,216(SP) + std 29,208(SP) + std 30,200(SP) + std 31,192(SP) + li 8,176 + stvx 20,8,SP + subi 8,8,16 + stvx 21,8,SP + subi 8,8,16 + stvx 22,8,SP + subi 8,8,16 + stvx 23,8,SP + subi 8,8,16 + stvx 24,8,SP + subi 8,8,16 + stvx 25,8,SP + subi 8,8,16 + stvx 26,8,SP + subi 8,8,16 + stvx 27,8,SP + subi 8,8,16 + stvx 28,8,SP + subi 8,8,16 + stvx 29,8,SP + subi 8,8,16 + stvx 30,8,SP + subi 8,8,16 + stvx 31,8,SP + + # table loading + li 8,0x300 + li 9,0x400 + li 10,0x500 + lxvd2x H21hX,8,TABLE + lxvd2x H21X,9,TABLE + lxvd2x H21lX,10,TABLE + li 8,0x600 + li 9,0x700 + li 10,0x800 + lxvd2x H43hX,8,TABLE + lxvd2x H43X,9,TABLE + lxvd2x H43lX,10,TABLE + li 8,0x900 + li 9,0xA00 + li 10,0xB00 + lxvd2x H65hX,8,TABLE + lxvd2x H65X,9,TABLE + lxvd2x H65lX,10,TABLE + li 8,0xC00 + li 9,0xD00 + li 10,0xE00 + lxvd2x H87hX,8,TABLE + lxvd2x H87X,9,TABLE + lxvd2x H87lX,10,TABLE + + li 8,0x10 + li 9,0x20 + li 10,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + + mtctr 7 +.align 5 +L8x_loop: + # input loading + lxvd2x C0X,0,DATA # load C0 + lxvd2x C1X,8,DATA # load C1 + lxvd2x C2X,9,DATA # load C2 + lxvd2x C3X,10,DATA # load C3 + + # swap permuting + vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask + vperm C2,C2,C2,swap_mask + vperm C3,C3,C3,swap_mask + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C23h,C2,C3,hidw_mask + vperm C23l,C2,C3,lodw_mask + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + + # input loading + lxvd2x C4X,28,DATA # load C4 + lxvd2x C5X,29,DATA # load C5 + lxvd2x C6X,30,DATA # load C6 + lxvd2x C7X,31,DATA # load C7 + + # swap permuting + vperm C4,C4,C4,swap_mask + vperm C5,C5,C5,swap_mask + vperm C6,C6,C6,swap_mask + vperm C7,C7,C7,swap_mask + + # polynomial multiplication "karatsuba" pre-processing + vperm C45h,C4,C5,hidw_mask + vperm C45l,C4,C5,lodw_mask + vperm C67h,C6,C7,hidw_mask + vperm C67l,C6,C7,lodw_mask + vxor C23,C23h,C23l + vxor C01,C01h,C01l + vxor C45,C45h,C45l + vxor C67,C67h,C67l + + # polynomial multiplication "karatsuba" + vpmsumd C23h,C23h,H65h # H23 = H^6h*C2h⊕H^5h*C3h + vpmsumd C23l,C23l,H65l # L23 = H^6l*C2l⊕H^5l*C3l + vpmsumd C01h,C01h,H87h # H01 = H^8h*C0h⊕H^7h*C1h + vpmsumd C01l,C01l,H87l # L01 = H^8l*C0l⊕H^7l*C1l + vpmsumd C67h,C67h,H21h # H67 = H^2h*C6h⊕H^1h*C7h + vpmsumd C67l,C67l,H21l # L67 = H^2l*C6l⊕H^1l*C7l + vpmsumd C45h,C45h,H43h # H45 = H^4h*C4h⊕H^3h*C5h + vpmsumd C45l,C45l,H43l # L45 = H^4l*C4l⊕H^3l*C5l + vpmsumd C23,C23,H65 # M23 = (H^6h⊕H^5h)*(C2h⊕C3h)⊕(H^6l⊕H^5l)*(C2l⊕C3l) + vpmsumd C01,C01,H87 # M01 = (H^8h⊕H^7h)*(C0h⊕C1h)⊕(H^8l⊕H^7l)*(C0l⊕C1l) + vpmsumd C45,C45,H43 # M45 = (H^4h⊕H^3h)*(C4h⊕C5h)⊕(H^4l⊕H^3l)*(C4l⊕C5l) + vpmsumd C67,C67,H21 # M67 = (H^2h⊕H^1h)*(C6h⊕C7h)⊕(H^2l⊕H^1l)*(C6l⊕C7l) + + # polynomial multiplication "karatsuba" post-processing + vxor C23,C23,C23h + vxor C01,C01,C01h + vxor C45,C45,C45h + vxor C67,C67,C67h + vxor C23,C23,C23l + vxor C01,C01,C01l + vxor C45,C45,C45l + vxor C67,C67,C67l + + # deferred recombination of partial products + vxor C01h,C01h,C23h # H0 = H01⊕H23 + vxor C45h,C45h,C67h # H1 = H45⊕H67 + vxor C01l,C01l,C23l # L0 = L01⊕L23 + vxor C45l,C45l,C67l # L1 = L45⊕L45 + vxor C01,C01,C23 # M0 = M01⊕M23 + vxor C45,C45,C67 # M1 = M45⊕M45 + vxor C01h,C01h,C45h # H = H0⊕H1 + vxor C01l,C01l,C45l # L = L0⊕L1 + vxor C01,C01,C45 # M = M0⊕M1 + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x80 + bdnz L8x_loop + + # restore registers + li 8,0 + lvx 31,8,SP + addi 8,8,16 + lvx 30,8,SP + addi 8,8,16 + lvx 29,8,SP + addi 8,8,16 + lvx 28,8,SP + addi 8,8,16 + lvx 27,8,SP + addi 8,8,16 + lvx 26,8,SP + addi 8,8,16 + lvx 25,8,SP + addi 8,8,16 + lvx 24,8,SP + addi 8,8,16 + lvx 23,8,SP + addi 8,8,16 + lvx 22,8,SP + addi 8,8,16 + lvx 21,8,SP + addi 8,8,16 + lvx 20,8,SP + ld 31,192(SP) + ld 30,200(SP) + ld 29,208(SP) + ld 28,216(SP) + addi SP,SP,224 + + clrldi LENGTH,LENGTH,57 +L2x: + srdi 7,LENGTH,5 + cmpldi 7,0 + beq L1x + + # table loading + li 8,0x300 + li 9,0x400 + li 10,0x500 + lxvd2x H21hX,8,TABLE + lxvd2x H21X,9,TABLE + lxvd2x H21lX,10,TABLE + + li 10,0x10 + + mtctr 7 +.align 5 +L2x_loop: + # input loading + lxvd2x C0X,0,DATA # load C0 + lxvd2x C1X,10,DATA # load C1 + + # swap permuting + vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + vxor C01,C01h,C01l + + # polynomial multiplication "karatsuba" + vpmsumd C01h,C01h,H21h # H01 = H^2h*C0h⊕H^1h*C1h + vpmsumd C01l,C01l,H21l # L01 = H^2l*C0l⊕H^1l*C1l + vpmsumd C01,C01,H21 # M01 = (H^2h⊕H^1h)*(C0h⊕C1h)⊕(H^2l⊕H^1l)*(C0l⊕C1l) + + # polynomial multiplication "karatsuba" post-processing + vxor C01,C01,C01h + vxor C01,C01,C01l + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x20 + bdnz L2x_loop + + clrldi LENGTH,LENGTH,59 +L1x: + srdi 7,LENGTH,4 + cmpldi 7,0 + beq Lrem + + # table loading + li 9,0x100 + li 10,0x200 + lxvd2x HlX,0,TABLE + lxvd2x HX, 9,TABLE + lxvd2x HhX,10,TABLE + + # input loading + lxvd2x C0X,0,DATA # load C0 + + # swap permuting + vperm C0,C0,C0,swap_mask + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + + addi DATA,DATA,0x10 + clrldi LENGTH,LENGTH,60 +Lrem: + cmpldi LENGTH,0 + beq Ldone + + # table loading + li 9,0x100 + li 10,0x200 + lxvd2x HlX,0,TABLE + lxvd2x HX, 9,TABLE + lxvd2x HhX,10,TABLE + + # input loading + stdu SP,-16(SP) + stvx zero,0,SP +Lst_loop: + subic. LENGTH,LENGTH,1 + lbzx 7,LENGTH,DATA + stbx 7,LENGTH,SP + bne Lst_loop + lxvd2x C0X,0,SP + addi SP,SP,16 + + # swap permuting + vperm C0,C0,C0,swap_mask + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + +Ldone: + vperm C,C,C,swap_mask + stxvd2x CX,0,X # store C + blr +EPILOGUE(_nettle_gcm_hash8) + + # gcm_fill (uint8_t *ctr, size_t blocks, union gcm_block *buffer) + +.align 5 +PROLOGUE(_nettle_gcm_fill) + ld 6,.swap_mask@got(TOCP) + lvx swap_mask,0,6 + + vxor zero,zero,zero + vspltisb I1,1 + vspltisb I2,2 + vspltisb I3,3 + vspltisb I4,4 + vspltisb I5,5 + vspltisb I6,6 + vspltisb I7,7 + vspltisb I8,8 + vsldoi I1,zero,I1,1 + vsldoi I2,zero,I2,1 + vsldoi I3,zero,I3,1 + vsldoi I4,zero,I4,1 + vsldoi I5,zero,I5,1 + vsldoi I6,zero,I6,1 + vsldoi I7,zero,I7,1 + vsldoi I8,zero,I8,1 + + lxvd2x CTR0X,0,CTR + vperm CTR0,CTR0,CTR0,swap_mask + + srdi 6,BLOCKS,3 # 8x loop count + cmpldi 6,0 + beq Lfill_4x + + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 25,0x10 + li 26,0x20 + li 27,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + + mtctr 6 +L8x_fill_loop: + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + vadduwm CTR4,CTR0,I4 + vadduwm CTR5,CTR0,I5 + vadduwm CTR6,CTR0,I6 + vadduwm CTR7,CTR0,I7 + + vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask + vperm CTR4,CTR4,CTR4,swap_mask + vperm CTR5,CTR5,CTR5,swap_mask + vperm CTR6,CTR6,CTR6,swap_mask + vperm CTR7,CTR7,CTR7,swap_mask + + stxvd2x CTR0SX,0,BUFFER + stxvd2x CTR1X,25,BUFFER + stxvd2x CTR2X,26,BUFFER + stxvd2x CTR3X,27,BUFFER + stxvd2x CTR4X,28,BUFFER + stxvd2x CTR5X,29,BUFFER + stxvd2x CTR6X,30,BUFFER + stxvd2x CTR7X,31,BUFFER + + vadduwm CTR0,CTR0,I8 + addi BUFFER,BUFFER,0x80 + bdnz L8x_fill_loop + + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi BLOCKS,BLOCKS,61 + +Lfill_4x: + srdi 6,BLOCKS,2 + cmpldi 6,0 + beq Lfill_2x + + li 8,0x10 + li 9,0x20 + li 10,0x30 + + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + + vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask + + stxvd2x CTR0SX,0,BUFFER + stxvd2x CTR1X,8,BUFFER + stxvd2x CTR2X,9,BUFFER + stxvd2x CTR3X,10,BUFFER + + vadduwm CTR0,CTR0,I4 + addi BUFFER,BUFFER,0x40 + + clrldi BLOCKS,BLOCKS,62 + +Lfill_2x: + srdi 6,BLOCKS,1 + cmpldi 6,0 + beq Lfill_1x + + li 10,0x10 + + vadduwm CTR1,CTR0,I1 + + vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + + stxvd2x CTR0SX,0,BUFFER + stxvd2x CTR1X,10,BUFFER + + vadduwm CTR0,CTR0,I2 + addi BUFFER,BUFFER,0x20 + + clrldi BLOCKS,BLOCKS,63 + +Lfill_1x: + cmpldi BLOCKS,0 + beq Lfill_done + + vperm CTR0S,CTR0,CTR0,swap_mask + + stxvd2x CTR0SX,0,BUFFER + + vadduwm CTR0,CTR0,I1 + +Lfill_done: + vperm CTR0,CTR0,CTR0,swap_mask + stxvd2x CTR0X,0,CTR + + blr +EPILOGUE(_nettle_gcm_fill) + + .data + .align 4 +.polynomial: + .byte 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0xc2 + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 + .align 4 +.hidw_mask: + .byte 23,22,21,20,19,18,17,16,7,6,5,4,3,2,1,0 + .align 4 +.lodw_mask: + .byte 31,30,29,28,27,26,25,24,15,14,13,12,11,10,9,8 diff --git a/testsuite/gcm-test.c b/testsuite/gcm-test.c index c8174019..df1fc94a 100644 --- a/testsuite/gcm-test.c +++ b/testsuite/gcm-test.c @@ -170,6 +170,29 @@ test_main(void) "16aedbf5a0de6a57a637b39b"), SHEX("619cc5aefffe0bfa462af43c1699d050")); + /* Test 128 bytes */ + test_aead(&nettle_gcm_aes128, NULL, + SHEX("feffe9928665731c6d6a8f9467308308"), + SHEX(""), + SHEX("d9313225f88406e5a55909c5aff5269a" + "86a7a9531534f7da2e4c303d8a318a72" + "1c3c0c95956809532fcf0e2449a6b525" + "b16aedf5aa0de657ba637b391aafd255" + "5ae376bc5e9f6a1b08e34db7a6ee0736" + "9ba662ea12f6f197e6bc3ed69d2480f3" + "ea5691347f2ba69113eb37910ebc18c8" + "0f697234582016fa956ca8f63ae6b473"), + SHEX("42831ec2217774244b7221b784d0d49c" + "e3aa212f2c02a4e035c17e2329aca12e" + "21d514b25466931c7d8f6a5aac84aa05" + "1ba30b396a0aac973d58e091473f5985" + "874b1178906ddbeab04ab2fe6cce8c57" + "8d7e961bd13fd6a8c56b66ca5e576492" + "1a48cd8bda04e66343e73055118b69b9" + "ced486813846958a11e602c03cfc232b"), + SHEX("cafebabefacedbaddecaf888"), + SHEX("796836f1246c9d735c5e1be0a715ccc3")); + /* Test case 7 */ test_aead(&nettle_gcm_aes192, NULL, SHEX("00000000000000000000000000000000"

1 0

[PATCH v2 1/2] gost28147: move params to separate source file
by Dmitry Baryshkov 14 Jun '20

14 Jun '20

Move big tables to separate source file. Signed-off-by: Dmitry Baryshkov <dbaryshkov(a)gmail.com> --- Makefile.in | 2 +- gost28147-params.c | 571 +++++++++++++++++++++++++++++++++++++++++++++ gost28147.c | 535 ------------------------------------------ 3 files changed, 572 insertions(+), 536 deletions(-) create mode 100644 gost28147-params.c diff --git a/Makefile.in b/Makefile.in index e5ccfc76b901..41e0389f2979 100644 --- a/Makefile.in +++ b/Makefile.in @@ -104,7 +104,7 @@ nettle_SOURCES = aes-decrypt-internal.c aes-decrypt.c \ gcm-camellia256.c gcm-camellia256-meta.c \ cmac.c cmac64.c cmac-aes128.c cmac-aes256.c cmac-des3.c \ cmac-aes128-meta.c cmac-aes256-meta.c cmac-des3-meta.c \ - gost28147.c gosthash94.c gosthash94-meta.c \ + gost28147.c gost28147-params.c gosthash94.c gosthash94-meta.c \ hmac.c hmac-gosthash94.c hmac-md5.c hmac-ripemd160.c \ hmac-sha1.c hmac-sha224.c hmac-sha256.c hmac-sha384.c \ hmac-sha512.c \ diff --git a/gost28147-params.c b/gost28147-params.c new file mode 100644 index 000000000000..6addec31f170 --- /dev/null +++ b/gost28147-params.c @@ -0,0 +1,571 @@ +/* gost28147-params.c - GOST 28147-89 cipher implementation + * + * based on Russian standard GOST 28147-89 + * For English description, check RFC 5830. + * S-Boxes are expanded from the tables defined in RFC4357: + * https://tools.ietf.org/html/rfc4357 + * + * Copyright: 2019 Dmitry Eremin-Solenikov <dbaryshkov(a)gmail.com> + * Copyright: 2009-2012 Aleksey Kravchenko <rhash.admin(a)gmail.com> + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the + * "Software"), to deal in the Software without restriction, including + * without limitation the rights to use, copy, modify, merge, publish, + * distribute, sublicense, and/or sell copies of the Software, and to + * permit persons to whom the Software is furnished to do so, subject to + * the following conditions: + * + * The above copyright notice and this permission notice shall be included + * in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS + * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY + * CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, + * TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE + * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ + +#if HAVE_CONFIG_H +#include "config.h" +#endif + +#include "macros.h" +#include "gost28147-internal.h" + +/* pre-initialized GOST lookup tables based on rotated S-Box */ +const struct gost28147_param _gost28147_param_test_3411 = +{ + { + { /* 0 */ + 0x00072000, 0x00075000, 0x00074800, 0x00071000, + 0x00076800, 0x00074000, 0x00070000, 0x00077000, + 0x00073000, 0x00075800, 0x00070800, 0x00076000, + 0x00073800, 0x00077800, 0x00072800, 0x00071800, + 0x0005a000, 0x0005d000, 0x0005c800, 0x00059000, + 0x0005e800, 0x0005c000, 0x00058000, 0x0005f000, + 0x0005b000, 0x0005d800, 0x00058800, 0x0005e000, + 0x0005b800, 0x0005f800, 0x0005a800, 0x00059800, + 0x00022000, 0x00025000, 0x00024800, 0x00021000, + 0x00026800, 0x00024000, 0x00020000, 0x00027000, + 0x00023000, 0x00025800, 0x00020800, 0x00026000, + 0x00023800, 0x00027800, 0x00022800, 0x00021800, + 0x00062000, 0x00065000, 0x00064800, 0x00061000, + 0x00066800, 0x00064000, 0x00060000, 0x00067000, + 0x00063000, 0x00065800, 0x00060800, 0x00066000, + 0x00063800, 0x00067800, 0x00062800, 0x00061800, + 0x00032000, 0x00035000, 0x00034800, 0x00031000, + 0x00036800, 0x00034000, 0x00030000, 0x00037000, + 0x00033000, 0x00035800, 0x00030800, 0x00036000, + 0x00033800, 0x00037800, 0x00032800, 0x00031800, + 0x0006a000, 0x0006d000, 0x0006c800, 0x00069000, + 0x0006e800, 0x0006c000, 0x00068000, 0x0006f000, + 0x0006b000, 0x0006d800, 0x00068800, 0x0006e000, + 0x0006b800, 0x0006f800, 0x0006a800, 0x00069800, + 0x0007a000, 0x0007d000, 0x0007c800, 0x00079000, + 0x0007e800, 0x0007c000, 0x00078000, 0x0007f000, + 0x0007b000, 0x0007d800, 0x00078800, 0x0007e000, + 0x0007b800, 0x0007f800, 0x0007a800, 0x00079800, + 0x00052000, 0x00055000, 0x00054800, 0x00051000, + 0x00056800, 0x00054000, 0x00050000, 0x00057000, + 0x00053000, 0x00055800, 0x00050800, 0x00056000, + 0x00053800, 0x00057800, 0x00052800, 0x00051800, + 0x00012000, 0x00015000, 0x00014800, 0x00011000, + 0x00016800, 0x00014000, 0x00010000, 0x00017000, + 0x00013000, 0x00015800, 0x00010800, 0x00016000, + 0x00013800, 0x00017800, 0x00012800, 0x00011800, + 0x0001a000, 0x0001d000, 0x0001c800, 0x00019000, + 0x0001e800, 0x0001c000, 0x00018000, 0x0001f000, + 0x0001b000, 0x0001d800, 0x00018800, 0x0001e000, + 0x0001b800, 0x0001f800, 0x0001a800, 0x00019800, + 0x00042000, 0x00045000, 0x00044800, 0x00041000, + 0x00046800, 0x00044000, 0x00040000, 0x00047000, + 0x00043000, 0x00045800, 0x00040800, 0x00046000, + 0x00043800, 0x00047800, 0x00042800, 0x00041800, + 0x0000a000, 0x0000d000, 0x0000c800, 0x00009000, + 0x0000e800, 0x0000c000, 0x00008000, 0x0000f000, + 0x0000b000, 0x0000d800, 0x00008800, 0x0000e000, + 0x0000b800, 0x0000f800, 0x0000a800, 0x00009800, + 0x00002000, 0x00005000, 0x00004800, 0x00001000, + 0x00006800, 0x00004000, 0x00000000, 0x00007000, + 0x00003000, 0x00005800, 0x00000800, 0x00006000, + 0x00003800, 0x00007800, 0x00002800, 0x00001800, + 0x0003a000, 0x0003d000, 0x0003c800, 0x00039000, + 0x0003e800, 0x0003c000, 0x00038000, 0x0003f000, + 0x0003b000, 0x0003d800, 0x00038800, 0x0003e000, + 0x0003b800, 0x0003f800, 0x0003a800, 0x00039800, + 0x0002a000, 0x0002d000, 0x0002c800, 0x00029000, + 0x0002e800, 0x0002c000, 0x00028000, 0x0002f000, + 0x0002b000, 0x0002d800, 0x00028800, 0x0002e000, + 0x0002b800, 0x0002f800, 0x0002a800, 0x00029800, + 0x0004a000, 0x0004d000, 0x0004c800, 0x00049000, + 0x0004e800, 0x0004c000, 0x00048000, 0x0004f000, + 0x0004b000, 0x0004d800, 0x00048800, 0x0004e000, + 0x0004b800, 0x0004f800, 0x0004a800, 0x00049800, + }, { /* 1 */ + 0x03a80000, 0x03c00000, 0x03880000, 0x03e80000, + 0x03d00000, 0x03980000, 0x03a00000, 0x03900000, + 0x03f00000, 0x03f80000, 0x03e00000, 0x03b80000, + 0x03b00000, 0x03800000, 0x03c80000, 0x03d80000, + 0x06a80000, 0x06c00000, 0x06880000, 0x06e80000, + 0x06d00000, 0x06980000, 0x06a00000, 0x06900000, + 0x06f00000, 0x06f80000, 0x06e00000, 0x06b80000, + 0x06b00000, 0x06800000, 0x06c80000, 0x06d80000, + 0x05280000, 0x05400000, 0x05080000, 0x05680000, + 0x05500000, 0x05180000, 0x05200000, 0x05100000, + 0x05700000, 0x05780000, 0x05600000, 0x05380000, + 0x05300000, 0x05000000, 0x05480000, 0x05580000, + 0x00a80000, 0x00c00000, 0x00880000, 0x00e80000, + 0x00d00000, 0x00980000, 0x00a00000, 0x00900000, + 0x00f00000, 0x00f80000, 0x00e00000, 0x00b80000, + 0x00b00000, 0x00800000, 0x00c80000, 0x00d80000, + 0x00280000, 0x00400000, 0x00080000, 0x00680000, + 0x00500000, 0x00180000, 0x00200000, 0x00100000, + 0x00700000, 0x00780000, 0x00600000, 0x00380000, + 0x00300000, 0x00000000, 0x00480000, 0x00580000, + 0x04280000, 0x04400000, 0x04080000, 0x04680000, + 0x04500000, 0x04180000, 0x04200000, 0x04100000, + 0x04700000, 0x04780000, 0x04600000, 0x04380000, + 0x04300000, 0x04000000, 0x04480000, 0x04580000, + 0x04a80000, 0x04c00000, 0x04880000, 0x04e80000, + 0x04d00000, 0x04980000, 0x04a00000, 0x04900000, + 0x04f00000, 0x04f80000, 0x04e00000, 0x04b80000, + 0x04b00000, 0x04800000, 0x04c80000, 0x04d80000, + 0x07a80000, 0x07c00000, 0x07880000, 0x07e80000, + 0x07d00000, 0x07980000, 0x07a00000, 0x07900000, + 0x07f00000, 0x07f80000, 0x07e00000, 0x07b80000, + 0x07b00000, 0x07800000, 0x07c80000, 0x07d80000, + 0x07280000, 0x07400000, 0x07080000, 0x07680000, + 0x07500000, 0x07180000, 0x07200000, 0x07100000, + 0x07700000, 0x07780000, 0x07600000, 0x07380000, + 0x07300000, 0x07000000, 0x07480000, 0x07580000, + 0x02280000, 0x02400000, 0x02080000, 0x02680000, + 0x02500000, 0x02180000, 0x02200000, 0x02100000, + 0x02700000, 0x02780000, 0x02600000, 0x02380000, + 0x02300000, 0x02000000, 0x02480000, 0x02580000, + 0x03280000, 0x03400000, 0x03080000, 0x03680000, + 0x03500000, 0x03180000, 0x03200000, 0x03100000, + 0x03700000, 0x03780000, 0x03600000, 0x03380000, + 0x03300000, 0x03000000, 0x03480000, 0x03580000, + 0x06280000, 0x06400000, 0x06080000, 0x06680000, + 0x06500000, 0x06180000, 0x06200000, 0x06100000, + 0x06700000, 0x06780000, 0x06600000, 0x06380000, + 0x06300000, 0x06000000, 0x06480000, 0x06580000, + 0x05a80000, 0x05c00000, 0x05880000, 0x05e80000, + 0x05d00000, 0x05980000, 0x05a00000, 0x05900000, + 0x05f00000, 0x05f80000, 0x05e00000, 0x05b80000, + 0x05b00000, 0x05800000, 0x05c80000, 0x05d80000, + 0x01280000, 0x01400000, 0x01080000, 0x01680000, + 0x01500000, 0x01180000, 0x01200000, 0x01100000, + 0x01700000, 0x01780000, 0x01600000, 0x01380000, + 0x01300000, 0x01000000, 0x01480000, 0x01580000, + 0x02a80000, 0x02c00000, 0x02880000, 0x02e80000, + 0x02d00000, 0x02980000, 0x02a00000, 0x02900000, + 0x02f00000, 0x02f80000, 0x02e00000, 0x02b80000, + 0x02b00000, 0x02800000, 0x02c80000, 0x02d80000, + 0x01a80000, 0x01c00000, 0x01880000, 0x01e80000, + 0x01d00000, 0x01980000, 0x01a00000, 0x01900000, + 0x01f00000, 0x01f80000, 0x01e00000, 0x01b80000, + 0x01b00000, 0x01800000, 0x01c80000, 0x01d80000, + }, { /* 2 */ + 0x30000002, 0x60000002, 0x38000002, 0x08000002, + 0x28000002, 0x78000002, 0x68000002, 0x40000002, + 0x20000002, 0x50000002, 0x48000002, 0x70000002, + 0x00000002, 0x18000002, 0x58000002, 0x10000002, + 0xb0000005, 0xe0000005, 0xb8000005, 0x88000005, + 0xa8000005, 0xf8000005, 0xe8000005, 0xc0000005, + 0xa0000005, 0xd0000005, 0xc8000005, 0xf0000005, + 0x80000005, 0x98000005, 0xd8000005, 0x90000005, + 0x30000005, 0x60000005, 0x38000005, 0x08000005, + 0x28000005, 0x78000005, 0x68000005, 0x40000005, + 0x20000005, 0x50000005, 0x48000005, 0x70000005, + 0x00000005, 0x18000005, 0x58000005, 0x10000005, + 0x30000000, 0x60000000, 0x38000000, 0x08000000, + 0x28000000, 0x78000000, 0x68000000, 0x40000000, + 0x20000000, 0x50000000, 0x48000000, 0x70000000, + 0x00000000, 0x18000000, 0x58000000, 0x10000000, + 0xb0000003, 0xe0000003, 0xb8000003, 0x88000003, + 0xa8000003, 0xf8000003, 0xe8000003, 0xc0000003, + 0xa0000003, 0xd0000003, 0xc8000003, 0xf0000003, + 0x80000003, 0x98000003, 0xd8000003, 0x90000003, + 0x30000001, 0x60000001, 0x38000001, 0x08000001, + 0x28000001, 0x78000001, 0x68000001, 0x40000001, + 0x20000001, 0x50000001, 0x48000001, 0x70000001, + 0x00000001, 0x18000001, 0x58000001, 0x10000001, + 0xb0000000, 0xe0000000, 0xb8000000, 0x88000000, + 0xa8000000, 0xf8000000, 0xe8000000, 0xc0000000, + 0xa0000000, 0xd0000000, 0xc8000000, 0xf0000000, + 0x80000000, 0x98000000, 0xd8000000, 0x90000000, + 0xb0000006, 0xe0000006, 0xb8000006, 0x88000006, + 0xa8000006, 0xf8000006, 0xe8000006, 0xc0000006, + 0xa0000006, 0xd0000006, 0xc8000006, 0xf0000006, + 0x80000006, 0x98000006, 0xd8000006, 0x90000006, + 0xb0000001, 0xe0000001, 0xb8000001, 0x88000001, + 0xa8000001, 0xf8000001, 0xe8000001, 0xc0000001, + 0xa0000001, 0xd0000001, 0xc8000001, 0xf0000001, + 0x80000001, 0x98000001, 0xd8000001, 0x90000001, + 0x30000003, 0x60000003, 0x38000003, 0x08000003, + 0x28000003, 0x78000003, 0x68000003, 0x40000003, + 0x20000003, 0x50000003, 0x48000003, 0x70000003, + 0x00000003, 0x18000003, 0x58000003, 0x10000003, + 0x30000004, 0x60000004, 0x38000004, 0x08000004, + 0x28000004, 0x78000004, 0x68000004, 0x40000004, + 0x20000004, 0x50000004, 0x48000004, 0x70000004, + 0x00000004, 0x18000004, 0x58000004, 0x10000004, + 0xb0000002, 0xe0000002, 0xb8000002, 0x88000002, + 0xa8000002, 0xf8000002, 0xe8000002, 0xc0000002, + 0xa0000002, 0xd0000002, 0xc8000002, 0xf0000002, + 0x80000002, 0x98000002, 0xd8000002, 0x90000002, + 0xb0000004, 0xe0000004, 0xb8000004, 0x88000004, + 0xa8000004, 0xf8000004, 0xe8000004, 0xc0000004, + 0xa0000004, 0xd0000004, 0xc8000004, 0xf0000004, + 0x80000004, 0x98000004, 0xd8000004, 0x90000004, + 0x30000006, 0x60000006, 0x38000006, 0x08000006, + 0x28000006, 0x78000006, 0x68000006, 0x40000006, + 0x20000006, 0x50000006, 0x48000006, 0x70000006, + 0x00000006, 0x18000006, 0x58000006, 0x10000006, + 0xb0000007, 0xe0000007, 0xb8000007, 0x88000007, + 0xa8000007, 0xf8000007, 0xe8000007, 0xc0000007, + 0xa0000007, 0xd0000007, 0xc8000007, 0xf0000007, + 0x80000007, 0x98000007, 0xd8000007, 0x90000007, + 0x30000007, 0x60000007, 0x38000007, 0x08000007, + 0x28000007, 0x78000007, 0x68000007, 0x40000007, + 0x20000007, 0x50000007, 0x48000007, 0x70000007, + 0x00000007, 0x18000007, 0x58000007, 0x10000007, + }, { /* 3 */ + 0x000000e8, 0x000000d8, 0x000000a0, 0x00000088, + 0x00000098, 0x000000f8, 0x000000a8, 0x000000c8, + 0x00000080, 0x000000d0, 0x000000f0, 0x000000b8, + 0x000000b0, 0x000000c0, 0x00000090, 0x000000e0, + 0x000007e8, 0x000007d8, 0x000007a0, 0x00000788, + 0x00000798, 0x000007f8, 0x000007a8, 0x000007c8, + 0x00000780, 0x000007d0, 0x000007f0, 0x000007b8, + 0x000007b0, 0x000007c0, 0x00000790, 0x000007e0, + 0x000006e8, 0x000006d8, 0x000006a0, 0x00000688, + 0x00000698, 0x000006f8, 0x000006a8, 0x000006c8, + 0x00000680, 0x000006d0, 0x000006f0, 0x000006b8, + 0x000006b0, 0x000006c0, 0x00000690, 0x000006e0, + 0x00000068, 0x00000058, 0x00000020, 0x00000008, + 0x00000018, 0x00000078, 0x00000028, 0x00000048, + 0x00000000, 0x00000050, 0x00000070, 0x00000038, + 0x00000030, 0x00000040, 0x00000010, 0x00000060, + 0x000002e8, 0x000002d8, 0x000002a0, 0x00000288, + 0x00000298, 0x000002f8, 0x000002a8, 0x000002c8, + 0x00000280, 0x000002d0, 0x000002f0, 0x000002b8, + 0x000002b0, 0x000002c0, 0x00000290, 0x000002e0, + 0x000003e8, 0x000003d8, 0x000003a0, 0x00000388, + 0x00000398, 0x000003f8, 0x000003a8, 0x000003c8, + 0x00000380, 0x000003d0, 0x000003f0, 0x000003b8, + 0x000003b0, 0x000003c0, 0x00000390, 0x000003e0, + 0x00000568, 0x00000558, 0x00000520, 0x00000508, + 0x00000518, 0x00000578, 0x00000528, 0x00000548, + 0x00000500, 0x00000550, 0x00000570, 0x00000538, + 0x00000530, 0x00000540, 0x00000510, 0x00000560, + 0x00000268, 0x00000258, 0x00000220, 0x00000208, + 0x00000218, 0x00000278, 0x00000228, 0x00000248, + 0x00000200, 0x00000250, 0x00000270, 0x00000238, + 0x00000230, 0x00000240, 0x00000210, 0x00000260, + 0x000004e8, 0x000004d8, 0x000004a0, 0x00000488, + 0x00000498, 0x000004f8, 0x000004a8, 0x000004c8, + 0x00000480, 0x000004d0, 0x000004f0, 0x000004b8, + 0x000004b0, 0x000004c0, 0x00000490, 0x000004e0, + 0x00000168, 0x00000158, 0x00000120, 0x00000108, + 0x00000118, 0x00000178, 0x00000128, 0x00000148, + 0x00000100, 0x00000150, 0x00000170, 0x00000138, + 0x00000130, 0x00000140, 0x00000110, 0x00000160, + 0x000001e8, 0x000001d8, 0x000001a0, 0x00000188, + 0x00000198, 0x000001f8, 0x000001a8, 0x000001c8, + 0x00000180, 0x000001d0, 0x000001f0, 0x000001b8, + 0x000001b0, 0x000001c0, 0x00000190, 0x000001e0, + 0x00000768, 0x00000758, 0x00000720, 0x00000708, + 0x00000718, 0x00000778, 0x00000728, 0x00000748, + 0x00000700, 0x00000750, 0x00000770, 0x00000738, + 0x00000730, 0x00000740, 0x00000710, 0x00000760, + 0x00000368, 0x00000358, 0x00000320, 0x00000308, + 0x00000318, 0x00000378, 0x00000328, 0x00000348, + 0x00000300, 0x00000350, 0x00000370, 0x00000338, + 0x00000330, 0x00000340, 0x00000310, 0x00000360, + 0x000005e8, 0x000005d8, 0x000005a0, 0x00000588, + 0x00000598, 0x000005f8, 0x000005a8, 0x000005c8, + 0x00000580, 0x000005d0, 0x000005f0, 0x000005b8, + 0x000005b0, 0x000005c0, 0x00000590, 0x000005e0, + 0x00000468, 0x00000458, 0x00000420, 0x00000408, + 0x00000418, 0x00000478, 0x00000428, 0x00000448, + 0x00000400, 0x00000450, 0x00000470, 0x00000438, + 0x00000430, 0x00000440, 0x00000410, 0x00000460, + 0x00000668, 0x00000658, 0x00000620, 0x00000608, + 0x00000618, 0x00000678, 0x00000628, 0x00000648, + 0x00000600, 0x00000650, 0x00000670, 0x00000638, + 0x00000630, 0x00000640, 0x00000610, 0x00000660, + } + } +}; + +const struct gost28147_param _gost28147_param_CryptoPro_3411 = +{ + { + { /* 0 */ + 0x0002d000, 0x0002a000, 0x0002a800, 0x0002b000, + 0x0002c000, 0x00028800, 0x00029800, 0x0002b800, + 0x0002e800, 0x0002e000, 0x0002f000, 0x00028000, + 0x0002c800, 0x00029000, 0x0002d800, 0x0002f800, + 0x0007d000, 0x0007a000, 0x0007a800, 0x0007b000, + 0x0007c000, 0x00078800, 0x00079800, 0x0007b800, + 0x0007e800, 0x0007e000, 0x0007f000, 0x00078000, + 0x0007c800, 0x00079000, 0x0007d800, 0x0007f800, + 0x00025000, 0x00022000, 0x00022800, 0x00023000, + 0x00024000, 0x00020800, 0x00021800, 0x00023800, + 0x00026800, 0x00026000, 0x00027000, 0x00020000, + 0x00024800, 0x00021000, 0x00025800, 0x00027800, + 0x00005000, 0x00002000, 0x00002800, 0x00003000, + 0x00004000, 0x00000800, 0x00001800, 0x00003800, + 0x00006800, 0x00006000, 0x00007000, 0x00000000, + 0x00004800, 0x00001000, 0x00005800, 0x00007800, + 0x00015000, 0x00012000, 0x00012800, 0x00013000, + 0x00014000, 0x00010800, 0x00011800, 0x00013800, + 0x00016800, 0x00016000, 0x00017000, 0x00010000, + 0x00014800, 0x00011000, 0x00015800, 0x00017800, + 0x0006d000, 0x0006a000, 0x0006a800, 0x0006b000, + 0x0006c000, 0x00068800, 0x00069800, 0x0006b800, + 0x0006e800, 0x0006e000, 0x0006f000, 0x00068000, + 0x0006c800, 0x00069000, 0x0006d800, 0x0006f800, + 0x0005d000, 0x0005a000, 0x0005a800, 0x0005b000, + 0x0005c000, 0x00058800, 0x00059800, 0x0005b800, + 0x0005e800, 0x0005e000, 0x0005f000, 0x00058000, + 0x0005c800, 0x00059000, 0x0005d800, 0x0005f800, + 0x0004d000, 0x0004a000, 0x0004a800, 0x0004b000, + 0x0004c000, 0x00048800, 0x00049800, 0x0004b800, + 0x0004e800, 0x0004e000, 0x0004f000, 0x00048000, + 0x0004c800, 0x00049000, 0x0004d800, 0x0004f800, + 0x0000d000, 0x0000a000, 0x0000a800, 0x0000b000, + 0x0000c000, 0x00008800, 0x00009800, 0x0000b800, + 0x0000e800, 0x0000e000, 0x0000f000, 0x00008000, + 0x0000c800, 0x00009000, 0x0000d800, 0x0000f800, + 0x0003d000, 0x0003a000, 0x0003a800, 0x0003b000, + 0x0003c000, 0x00038800, 0x00039800, 0x0003b800, + 0x0003e800, 0x0003e000, 0x0003f000, 0x00038000, + 0x0003c800, 0x00039000, 0x0003d800, 0x0003f800, + 0x00035000, 0x00032000, 0x00032800, 0x00033000, + 0x00034000, 0x00030800, 0x00031800, 0x00033800, + 0x00036800, 0x00036000, 0x00037000, 0x00030000, + 0x00034800, 0x00031000, 0x00035800, 0x00037800, + 0x0001d000, 0x0001a000, 0x0001a800, 0x0001b000, + 0x0001c000, 0x00018800, 0x00019800, 0x0001b800, + 0x0001e800, 0x0001e000, 0x0001f000, 0x00018000, + 0x0001c800, 0x00019000, 0x0001d800, 0x0001f800, + 0x00065000, 0x00062000, 0x00062800, 0x00063000, + 0x00064000, 0x00060800, 0x00061800, 0x00063800, + 0x00066800, 0x00066000, 0x00067000, 0x00060000, + 0x00064800, 0x00061000, 0x00065800, 0x00067800, + 0x00075000, 0x00072000, 0x00072800, 0x00073000, + 0x00074000, 0x00070800, 0x00071800, 0x00073800, + 0x00076800, 0x00076000, 0x00077000, 0x00070000, + 0x00074800, 0x00071000, 0x00075800, 0x00077800, + 0x00055000, 0x00052000, 0x00052800, 0x00053000, + 0x00054000, 0x00050800, 0x00051800, 0x00053800, + 0x00056800, 0x00056000, 0x00057000, 0x00050000, + 0x00054800, 0x00051000, 0x00055800, 0x00057800, + 0x00045000, 0x00042000, 0x00042800, 0x00043000, + 0x00044000, 0x00040800, 0x00041800, 0x00043800, + 0x00046800, 0x00046000, 0x00047000, 0x00040000, + 0x00044800, 0x00041000, 0x00045800, 0x00047800, + }, { /* 1 */ + 0x02380000, 0x02780000, 0x02600000, 0x02700000, + 0x02480000, 0x02200000, 0x02080000, 0x02000000, + 0x02180000, 0x02580000, 0x02280000, 0x02100000, + 0x02300000, 0x02500000, 0x02400000, 0x02680000, + 0x05380000, 0x05780000, 0x05600000, 0x05700000, + 0x05480000, 0x05200000, 0x05080000, 0x05000000, + 0x05180000, 0x05580000, 0x05280000, 0x05100000, + 0x05300000, 0x05500000, 0x05400000, 0x05680000, + 0x03b80000, 0x03f80000, 0x03e00000, 0x03f00000, + 0x03c80000, 0x03a00000, 0x03880000, 0x03800000, + 0x03980000, 0x03d80000, 0x03a80000, 0x03900000, + 0x03b00000, 0x03d00000, 0x03c00000, 0x03e80000, + 0x06380000, 0x06780000, 0x06600000, 0x06700000, + 0x06480000, 0x06200000, 0x06080000, 0x06000000, + 0x06180000, 0x06580000, 0x06280000, 0x06100000, + 0x06300000, 0x06500000, 0x06400000, 0x06680000, + 0x00380000, 0x00780000, 0x00600000, 0x00700000, + 0x00480000, 0x00200000, 0x00080000, 0x00000000, + 0x00180000, 0x00580000, 0x00280000, 0x00100000, + 0x00300000, 0x00500000, 0x00400000, 0x00680000, + 0x07b80000, 0x07f80000, 0x07e00000, 0x07f00000, + 0x07c80000, 0x07a00000, 0x07880000, 0x07800000, + 0x07980000, 0x07d80000, 0x07a80000, 0x07900000, + 0x07b00000, 0x07d00000, 0x07c00000, 0x07e80000, + 0x01380000, 0x01780000, 0x01600000, 0x01700000, + 0x01480000, 0x01200000, 0x01080000, 0x01000000, + 0x01180000, 0x01580000, 0x01280000, 0x01100000, + 0x01300000, 0x01500000, 0x01400000, 0x01680000, + 0x04380000, 0x04780000, 0x04600000, 0x04700000, + 0x04480000, 0x04200000, 0x04080000, 0x04000000, + 0x04180000, 0x04580000, 0x04280000, 0x04100000, + 0x04300000, 0x04500000, 0x04400000, 0x04680000, + 0x07380000, 0x07780000, 0x07600000, 0x07700000, + 0x07480000, 0x07200000, 0x07080000, 0x07000000, + 0x07180000, 0x07580000, 0x07280000, 0x07100000, + 0x07300000, 0x07500000, 0x07400000, 0x07680000, + 0x00b80000, 0x00f80000, 0x00e00000, 0x00f00000, + 0x00c80000, 0x00a00000, 0x00880000, 0x00800000, + 0x00980000, 0x00d80000, 0x00a80000, 0x00900000, + 0x00b00000, 0x00d00000, 0x00c00000, 0x00e80000, + 0x03380000, 0x03780000, 0x03600000, 0x03700000, + 0x03480000, 0x03200000, 0x03080000, 0x03000000, + 0x03180000, 0x03580000, 0x03280000, 0x03100000, + 0x03300000, 0x03500000, 0x03400000, 0x03680000, + 0x02b80000, 0x02f80000, 0x02e00000, 0x02f00000, + 0x02c80000, 0x02a00000, 0x02880000, 0x02800000, + 0x02980000, 0x02d80000, 0x02a80000, 0x02900000, + 0x02b00000, 0x02d00000, 0x02c00000, 0x02e80000, + 0x06b80000, 0x06f80000, 0x06e00000, 0x06f00000, + 0x06c80000, 0x06a00000, 0x06880000, 0x06800000, + 0x06980000, 0x06d80000, 0x06a80000, 0x06900000, + 0x06b00000, 0x06d00000, 0x06c00000, 0x06e80000, + 0x05b80000, 0x05f80000, 0x05e00000, 0x05f00000, + 0x05c80000, 0x05a00000, 0x05880000, 0x05800000, + 0x05980000, 0x05d80000, 0x05a80000, 0x05900000, + 0x05b00000, 0x05d00000, 0x05c00000, 0x05e80000, + 0x04b80000, 0x04f80000, 0x04e00000, 0x04f00000, + 0x04c80000, 0x04a00000, 0x04880000, 0x04800000, + 0x04980000, 0x04d80000, 0x04a80000, 0x04900000, + 0x04b00000, 0x04d00000, 0x04c00000, 0x04e80000, + 0x01b80000, 0x01f80000, 0x01e00000, 0x01f00000, + 0x01c80000, 0x01a00000, 0x01880000, 0x01800000, + 0x01980000, 0x01d80000, 0x01a80000, 0x01900000, + 0x01b00000, 0x01d00000, 0x01c00000, 0x01e80000, + }, { /* 2 */ + 0xb8000003, 0xb0000003, 0xa0000003, 0xd8000003, + 0xc8000003, 0xe0000003, 0x90000003, 0xd0000003, + 0x88000003, 0xc0000003, 0x80000003, 0xf0000003, + 0xf8000003, 0xe8000003, 0x98000003, 0xa8000003, + 0x38000003, 0x30000003, 0x20000003, 0x58000003, + 0x48000003, 0x60000003, 0x10000003, 0x50000003, + 0x08000003, 0x40000003, 0x00000003, 0x70000003, + 0x78000003, 0x68000003, 0x18000003, 0x28000003, + 0x38000001, 0x30000001, 0x20000001, 0x58000001, + 0x48000001, 0x60000001, 0x10000001, 0x50000001, + 0x08000001, 0x40000001, 0x00000001, 0x70000001, + 0x78000001, 0x68000001, 0x18000001, 0x28000001, + 0x38000002, 0x30000002, 0x20000002, 0x58000002, + 0x48000002, 0x60000002, 0x10000002, 0x50000002, + 0x08000002, 0x40000002, 0x00000002, 0x70000002, + 0x78000002, 0x68000002, 0x18000002, 0x28000002, + 0xb8000006, 0xb0000006, 0xa0000006, 0xd8000006, + 0xc8000006, 0xe0000006, 0x90000006, 0xd0000006, + 0x88000006, 0xc0000006, 0x80000006, 0xf0000006, + 0xf8000006, 0xe8000006, 0x98000006, 0xa8000006, + 0xb8000004, 0xb0000004, 0xa0000004, 0xd8000004, + 0xc8000004, 0xe0000004, 0x90000004, 0xd0000004, + 0x88000004, 0xc0000004, 0x80000004, 0xf0000004, + 0xf8000004, 0xe8000004, 0x98000004, 0xa8000004, + 0xb8000007, 0xb0000007, 0xa0000007, 0xd8000007, + 0xc8000007, 0xe0000007, 0x90000007, 0xd0000007, + 0x88000007, 0xc0000007, 0x80000007, 0xf0000007, + 0xf8000007, 0xe8000007, 0x98000007, 0xa8000007, + 0x38000000, 0x30000000, 0x20000000, 0x58000000, + 0x48000000, 0x60000000, 0x10000000, 0x50000000, + 0x08000000, 0x40000000, 0x00000000, 0x70000000, + 0x78000000, 0x68000000, 0x18000000, 0x28000000, + 0x38000005, 0x30000005, 0x20000005, 0x58000005, + 0x48000005, 0x60000005, 0x10000005, 0x50000005, + 0x08000005, 0x40000005, 0x00000005, 0x70000005, + 0x78000005, 0x68000005, 0x18000005, 0x28000005, + 0xb8000000, 0xb0000000, 0xa0000000, 0xd8000000, + 0xc8000000, 0xe0000000, 0x90000000, 0xd0000000, + 0x88000000, 0xc0000000, 0x80000000, 0xf0000000, + 0xf8000000, 0xe8000000, 0x98000000, 0xa8000000, + 0xb8000002, 0xb0000002, 0xa0000002, 0xd8000002, + 0xc8000002, 0xe0000002, 0x90000002, 0xd0000002, + 0x88000002, 0xc0000002, 0x80000002, 0xf0000002, + 0xf8000002, 0xe8000002, 0x98000002, 0xa8000002, + 0xb8000005, 0xb0000005, 0xa0000005, 0xd8000005, + 0xc8000005, 0xe0000005, 0x90000005, 0xd0000005, + 0x88000005, 0xc0000005, 0x80000005, 0xf0000005, + 0xf8000005, 0xe8000005, 0x98000005, 0xa8000005, + 0x38000004, 0x30000004, 0x20000004, 0x58000004, + 0x48000004, 0x60000004, 0x10000004, 0x50000004, + 0x08000004, 0x40000004, 0x00000004, 0x70000004, + 0x78000004, 0x68000004, 0x18000004, 0x28000004, + 0x38000007, 0x30000007, 0x20000007, 0x58000007, + 0x48000007, 0x60000007, 0x10000007, 0x50000007, + 0x08000007, 0x40000007, 0x00000007, 0x70000007, + 0x78000007, 0x68000007, 0x18000007, 0x28000007, + 0x38000006, 0x30000006, 0x20000006, 0x58000006, + 0x48000006, 0x60000006, 0x10000006, 0x50000006, + 0x08000006, 0x40000006, 0x00000006, 0x70000006, + 0x78000006, 0x68000006, 0x18000006, 0x28000006, + 0xb8000001, 0xb0000001, 0xa0000001, 0xd8000001, + 0xc8000001, 0xe0000001, 0x90000001, 0xd0000001, + 0x88000001, 0xc0000001, 0x80000001, 0xf0000001, + 0xf8000001, 0xe8000001, 0x98000001, 0xa8000001, + }, { /* 3 */ + 0x000000e8, 0x000000f0, 0x000000a0, 0x00000088, + 0x000000b8, 0x00000080, 0x000000a8, 0x000000d0, + 0x00000098, 0x000000e0, 0x000000c0, 0x000000f8, + 0x000000b0, 0x00000090, 0x000000c8, 0x000000d8, + 0x000001e8, 0x000001f0, 0x000001a0, 0x00000188, + 0x000001b8, 0x00000180, 0x000001a8, 0x000001d0, + 0x00000198, 0x000001e0, 0x000001c0, 0x000001f8, + 0x000001b0, 0x00000190, 0x000001c8, 0x000001d8, + 0x00000568, 0x00000570, 0x00000520, 0x00000508, + 0x00000538, 0x00000500, 0x00000528, 0x00000550, + 0x00000518, 0x00000560, 0x00000540, 0x00000578, + 0x00000530, 0x00000510, 0x00000548, 0x00000558, + 0x000004e8, 0x000004f0, 0x000004a0, 0x00000488, + 0x000004b8, 0x00000480, 0x000004a8, 0x000004d0, + 0x00000498, 0x000004e0, 0x000004c0, 0x000004f8, + 0x000004b0, 0x00000490, 0x000004c8, 0x000004d8, + 0x000002e8, 0x000002f0, 0x000002a0, 0x00000288, + 0x000002b8, 0x00000280, 0x000002a8, 0x000002d0, + 0x00000298, 0x000002e0, 0x000002c0, 0x000002f8, + 0x000002b0, 0x00000290, 0x000002c8, 0x000002d8, + 0x000005e8, 0x000005f0, 0x000005a0, 0x00000588, + 0x000005b8, 0x00000580, 0x000005a8, 0x000005d0, + 0x00000598, 0x000005e0, 0x000005c0, 0x000005f8, + 0x000005b0, 0x00000590, 0x000005c8, 0x000005d8, + 0x00000268, 0x00000270, 0x00000220, 0x00000208, + 0x00000238, 0x00000200, 0x00000228, 0x00000250, + 0x00000218, 0x00000260, 0x00000240, 0x00000278, + 0x00000230, 0x00000210, 0x00000248, 0x00000258, + 0x000007e8, 0x000007f0, 0x000007a0, 0x00000788, + 0x000007b8, 0x00000780, 0x000007a8, 0x000007d0, + 0x00000798, 0x000007e0, 0x000007c0, 0x000007f8, + 0x000007b0, 0x00000790, 0x000007c8, 0x000007d8, + 0x00000468, 0x00000470, 0x00000420, 0x00000408, + 0x00000438, 0x00000400, 0x00000428, 0x00000450, + 0x00000418, 0x00000460, 0x00000440, 0x00000478, + 0x00000430, 0x00000410, 0x00000448, 0x00000458, + 0x00000368, 0x00000370, 0x00000320, 0x00000308, + 0x00000338, 0x00000300, 0x00000328, 0x00000350, + 0x00000318, 0x00000360, 0x00000340, 0x00000378, + 0x00000330, 0x00000310, 0x00000348, 0x00000358, + 0x000003e8, 0x000003f0, 0x000003a0, 0x00000388, + 0x000003b8, 0x00000380, 0x000003a8, 0x000003d0, + 0x00000398, 0x000003e0, 0x000003c0, 0x000003f8, + 0x000003b0, 0x00000390, 0x000003c8, 0x000003d8, + 0x00000768, 0x00000770, 0x00000720, 0x00000708, + 0x00000738, 0x00000700, 0x00000728, 0x00000750, + 0x00000718, 0x00000760, 0x00000740, 0x00000778, + 0x00000730, 0x00000710, 0x00000748, 0x00000758, + 0x000006e8, 0x000006f0, 0x000006a0, 0x00000688, + 0x000006b8, 0x00000680, 0x000006a8, 0x000006d0, + 0x00000698, 0x000006e0, 0x000006c0, 0x000006f8, + 0x000006b0, 0x00000690, 0x000006c8, 0x000006d8, + 0x00000068, 0x00000070, 0x00000020, 0x00000008, + 0x00000038, 0x00000000, 0x00000028, 0x00000050, + 0x00000018, 0x00000060, 0x00000040, 0x00000078, + 0x00000030, 0x00000010, 0x00000048, 0x00000058, + 0x00000168, 0x00000170, 0x00000120, 0x00000108, + 0x00000138, 0x00000100, 0x00000128, 0x00000150, + 0x00000118, 0x00000160, 0x00000140, 0x00000178, + 0x00000130, 0x00000110, 0x00000148, 0x00000158, + 0x00000668, 0x00000670, 0x00000620, 0x00000608, + 0x00000638, 0x00000600, 0x00000628, 0x00000650, + 0x00000618, 0x00000660, 0x00000640, 0x00000678, + 0x00000630, 0x00000610, 0x00000648, 0x00000658, + } + } +}; diff --git a/gost28147.c b/gost28147.c index b6db334b2a0b..669fc85d2083 100644 --- a/gost28147.c +++ b/gost28147.c @@ -35,541 +35,6 @@ #include "macros.h" #include "gost28147-internal.h" -/* pre-initialized GOST lookup tables based on rotated S-Box */ -const struct gost28147_param _gost28147_param_test_3411 = -{ - { - { /* 0 */ - 0x00072000, 0x00075000, 0x00074800, 0x00071000, - 0x00076800, 0x00074000, 0x00070000, 0x00077000, - 0x00073000, 0x00075800, 0x00070800, 0x00076000, - 0x00073800, 0x00077800, 0x00072800, 0x00071800, - 0x0005a000, 0x0005d000, 0x0005c800, 0x00059000, - 0x0005e800, 0x0005c000, 0x00058000, 0x0005f000, - 0x0005b000, 0x0005d800, 0x00058800, 0x0005e000, - 0x0005b800, 0x0005f800, 0x0005a800, 0x00059800, - 0x00022000, 0x00025000, 0x00024800, 0x00021000, - 0x00026800, 0x00024000, 0x00020000, 0x00027000, - 0x00023000, 0x00025800, 0x00020800, 0x00026000, - 0x00023800, 0x00027800, 0x00022800, 0x00021800, - 0x00062000, 0x00065000, 0x00064800, 0x00061000, - 0x00066800, 0x00064000, 0x00060000, 0x00067000, - 0x00063000, 0x00065800, 0x00060800, 0x00066000, - 0x00063800, 0x00067800, 0x00062800, 0x00061800, - 0x00032000, 0x00035000, 0x00034800, 0x00031000, - 0x00036800, 0x00034000, 0x00030000, 0x00037000, - 0x00033000, 0x00035800, 0x00030800, 0x00036000, - 0x00033800, 0x00037800, 0x00032800, 0x00031800, - 0x0006a000, 0x0006d000, 0x0006c800, 0x00069000, - 0x0006e800, 0x0006c000, 0x00068000, 0x0006f000, - 0x0006b000, 0x0006d800, 0x00068800, 0x0006e000, - 0x0006b800, 0x0006f800, 0x0006a800, 0x00069800, - 0x0007a000, 0x0007d000, 0x0007c800, 0x00079000, - 0x0007e800, 0x0007c000, 0x00078000, 0x0007f000, - 0x0007b000, 0x0007d800, 0x00078800, 0x0007e000, - 0x0007b800, 0x0007f800, 0x0007a800, 0x00079800, - 0x00052000, 0x00055000, 0x00054800, 0x00051000, - 0x00056800, 0x00054000, 0x00050000, 0x00057000, - 0x00053000, 0x00055800, 0x00050800, 0x00056000, - 0x00053800, 0x00057800, 0x00052800, 0x00051800, - 0x00012000, 0x00015000, 0x00014800, 0x00011000, - 0x00016800, 0x00014000, 0x00010000, 0x00017000, - 0x00013000, 0x00015800, 0x00010800, 0x00016000, - 0x00013800, 0x00017800, 0x00012800, 0x00011800, - 0x0001a000, 0x0001d000, 0x0001c800, 0x00019000, - 0x0001e800, 0x0001c000, 0x00018000, 0x0001f000, - 0x0001b000, 0x0001d800, 0x00018800, 0x0001e000, - 0x0001b800, 0x0001f800, 0x0001a800, 0x00019800, - 0x00042000, 0x00045000, 0x00044800, 0x00041000, - 0x00046800, 0x00044000, 0x00040000, 0x00047000, - 0x00043000, 0x00045800, 0x00040800, 0x00046000, - 0x00043800, 0x00047800, 0x00042800, 0x00041800, - 0x0000a000, 0x0000d000, 0x0000c800, 0x00009000, - 0x0000e800, 0x0000c000, 0x00008000, 0x0000f000, - 0x0000b000, 0x0000d800, 0x00008800, 0x0000e000, - 0x0000b800, 0x0000f800, 0x0000a800, 0x00009800, - 0x00002000, 0x00005000, 0x00004800, 0x00001000, - 0x00006800, 0x00004000, 0x00000000, 0x00007000, - 0x00003000, 0x00005800, 0x00000800, 0x00006000, - 0x00003800, 0x00007800, 0x00002800, 0x00001800, - 0x0003a000, 0x0003d000, 0x0003c800, 0x00039000, - 0x0003e800, 0x0003c000, 0x00038000, 0x0003f000, - 0x0003b000, 0x0003d800, 0x00038800, 0x0003e000, - 0x0003b800, 0x0003f800, 0x0003a800, 0x00039800, - 0x0002a000, 0x0002d000, 0x0002c800, 0x00029000, - 0x0002e800, 0x0002c000, 0x00028000, 0x0002f000, - 0x0002b000, 0x0002d800, 0x00028800, 0x0002e000, - 0x0002b800, 0x0002f800, 0x0002a800, 0x00029800, - 0x0004a000, 0x0004d000, 0x0004c800, 0x00049000, - 0x0004e800, 0x0004c000, 0x00048000, 0x0004f000, - 0x0004b000, 0x0004d800, 0x00048800, 0x0004e000, - 0x0004b800, 0x0004f800, 0x0004a800, 0x00049800, - }, { /* 1 */ - 0x03a80000, 0x03c00000, 0x03880000, 0x03e80000, - 0x03d00000, 0x03980000, 0x03a00000, 0x03900000, - 0x03f00000, 0x03f80000, 0x03e00000, 0x03b80000, - 0x03b00000, 0x03800000, 0x03c80000, 0x03d80000, - 0x06a80000, 0x06c00000, 0x06880000, 0x06e80000, - 0x06d00000, 0x06980000, 0x06a00000, 0x06900000, - 0x06f00000, 0x06f80000, 0x06e00000, 0x06b80000, - 0x06b00000, 0x06800000, 0x06c80000, 0x06d80000, - 0x05280000, 0x05400000, 0x05080000, 0x05680000, - 0x05500000, 0x05180000, 0x05200000, 0x05100000, - 0x05700000, 0x05780000, 0x05600000, 0x05380000, - 0x05300000, 0x05000000, 0x05480000, 0x05580000, - 0x00a80000, 0x00c00000, 0x00880000, 0x00e80000, - 0x00d00000, 0x00980000, 0x00a00000, 0x00900000, - 0x00f00000, 0x00f80000, 0x00e00000, 0x00b80000, - 0x00b00000, 0x00800000, 0x00c80000, 0x00d80000, - 0x00280000, 0x00400000, 0x00080000, 0x00680000, - 0x00500000, 0x00180000, 0x00200000, 0x00100000, - 0x00700000, 0x00780000, 0x00600000, 0x00380000, - 0x00300000, 0x00000000, 0x00480000, 0x00580000, - 0x04280000, 0x04400000, 0x04080000, 0x04680000, - 0x04500000, 0x04180000, 0x04200000, 0x04100000, - 0x04700000, 0x04780000, 0x04600000, 0x04380000, - 0x04300000, 0x04000000, 0x04480000, 0x04580000, - 0x04a80000, 0x04c00000, 0x04880000, 0x04e80000, - 0x04d00000, 0x04980000, 0x04a00000, 0x04900000, - 0x04f00000, 0x04f80000, 0x04e00000, 0x04b80000, - 0x04b00000, 0x04800000, 0x04c80000, 0x04d80000, - 0x07a80000, 0x07c00000, 0x07880000, 0x07e80000, - 0x07d00000, 0x07980000, 0x07a00000, 0x07900000, - 0x07f00000, 0x07f80000, 0x07e00000, 0x07b80000, - 0x07b00000, 0x07800000, 0x07c80000, 0x07d80000, - 0x07280000, 0x07400000, 0x07080000, 0x07680000, - 0x07500000, 0x07180000, 0x07200000, 0x07100000, - 0x07700000, 0x07780000, 0x07600000, 0x07380000, - 0x07300000, 0x07000000, 0x07480000, 0x07580000, - 0x02280000, 0x02400000, 0x02080000, 0x02680000, - 0x02500000, 0x02180000, 0x02200000, 0x02100000, - 0x02700000, 0x02780000, 0x02600000, 0x02380000, - 0x02300000, 0x02000000, 0x02480000, 0x02580000, - 0x03280000, 0x03400000, 0x03080000, 0x03680000, - 0x03500000, 0x03180000, 0x03200000, 0x03100000, - 0x03700000, 0x03780000, 0x03600000, 0x03380000, - 0x03300000, 0x03000000, 0x03480000, 0x03580000, - 0x06280000, 0x06400000, 0x06080000, 0x06680000, - 0x06500000, 0x06180000, 0x06200000, 0x06100000, - 0x06700000, 0x06780000, 0x06600000, 0x06380000, - 0x06300000, 0x06000000, 0x06480000, 0x06580000, - 0x05a80000, 0x05c00000, 0x05880000, 0x05e80000, - 0x05d00000, 0x05980000, 0x05a00000, 0x05900000, - 0x05f00000, 0x05f80000, 0x05e00000, 0x05b80000, - 0x05b00000, 0x05800000, 0x05c80000, 0x05d80000, - 0x01280000, 0x01400000, 0x01080000, 0x01680000, - 0x01500000, 0x01180000, 0x01200000, 0x01100000, - 0x01700000, 0x01780000, 0x01600000, 0x01380000, - 0x01300000, 0x01000000, 0x01480000, 0x01580000, - 0x02a80000, 0x02c00000, 0x02880000, 0x02e80000, - 0x02d00000, 0x02980000, 0x02a00000, 0x02900000, - 0x02f00000, 0x02f80000, 0x02e00000, 0x02b80000, - 0x02b00000, 0x02800000, 0x02c80000, 0x02d80000, - 0x01a80000, 0x01c00000, 0x01880000, 0x01e80000, - 0x01d00000, 0x01980000, 0x01a00000, 0x01900000, - 0x01f00000, 0x01f80000, 0x01e00000, 0x01b80000, - 0x01b00000, 0x01800000, 0x01c80000, 0x01d80000, - }, { /* 2 */ - 0x30000002, 0x60000002, 0x38000002, 0x08000002, - 0x28000002, 0x78000002, 0x68000002, 0x40000002, - 0x20000002, 0x50000002, 0x48000002, 0x70000002, - 0x00000002, 0x18000002, 0x58000002, 0x10000002, - 0xb0000005, 0xe0000005, 0xb8000005, 0x88000005, - 0xa8000005, 0xf8000005, 0xe8000005, 0xc0000005, - 0xa0000005, 0xd0000005, 0xc8000005, 0xf0000005, - 0x80000005, 0x98000005, 0xd8000005, 0x90000005, - 0x30000005, 0x60000005, 0x38000005, 0x08000005, - 0x28000005, 0x78000005, 0x68000005, 0x40000005, - 0x20000005, 0x50000005, 0x48000005, 0x70000005, - 0x00000005, 0x18000005, 0x58000005, 0x10000005, - 0x30000000, 0x60000000, 0x38000000, 0x08000000, - 0x28000000, 0x78000000, 0x68000000, 0x40000000, - 0x20000000, 0x50000000, 0x48000000, 0x70000000, - 0x00000000, 0x18000000, 0x58000000, 0x10000000, - 0xb0000003, 0xe0000003, 0xb8000003, 0x88000003, - 0xa8000003, 0xf8000003, 0xe8000003, 0xc0000003, - 0xa0000003, 0xd0000003, 0xc8000003, 0xf0000003, - 0x80000003, 0x98000003, 0xd8000003, 0x90000003, - 0x30000001, 0x60000001, 0x38000001, 0x08000001, - 0x28000001, 0x78000001, 0x68000001, 0x40000001, - 0x20000001, 0x50000001, 0x48000001, 0x70000001, - 0x00000001, 0x18000001, 0x58000001, 0x10000001, - 0xb0000000, 0xe0000000, 0xb8000000, 0x88000000, - 0xa8000000, 0xf8000000, 0xe8000000, 0xc0000000, - 0xa0000000, 0xd0000000, 0xc8000000, 0xf0000000, - 0x80000000, 0x98000000, 0xd8000000, 0x90000000, - 0xb0000006, 0xe0000006, 0xb8000006, 0x88000006, - 0xa8000006, 0xf8000006, 0xe8000006, 0xc0000006, - 0xa0000006, 0xd0000006, 0xc8000006, 0xf0000006, - 0x80000006, 0x98000006, 0xd8000006, 0x90000006, - 0xb0000001, 0xe0000001, 0xb8000001, 0x88000001, - 0xa8000001, 0xf8000001, 0xe8000001, 0xc0000001, - 0xa0000001, 0xd0000001, 0xc8000001, 0xf0000001, - 0x80000001, 0x98000001, 0xd8000001, 0x90000001, - 0x30000003, 0x60000003, 0x38000003, 0x08000003, - 0x28000003, 0x78000003, 0x68000003, 0x40000003, - 0x20000003, 0x50000003, 0x48000003, 0x70000003, - 0x00000003, 0x18000003, 0x58000003, 0x10000003, - 0x30000004, 0x60000004, 0x38000004, 0x08000004, - 0x28000004, 0x78000004, 0x68000004, 0x40000004, - 0x20000004, 0x50000004, 0x48000004, 0x70000004, - 0x00000004, 0x18000004, 0x58000004, 0x10000004, - 0xb0000002, 0xe0000002, 0xb8000002, 0x88000002, - 0xa8000002, 0xf8000002, 0xe8000002, 0xc0000002, - 0xa0000002, 0xd0000002, 0xc8000002, 0xf0000002, - 0x80000002, 0x98000002, 0xd8000002, 0x90000002, - 0xb0000004, 0xe0000004, 0xb8000004, 0x88000004, - 0xa8000004, 0xf8000004, 0xe8000004, 0xc0000004, - 0xa0000004, 0xd0000004, 0xc8000004, 0xf0000004, - 0x80000004, 0x98000004, 0xd8000004, 0x90000004, - 0x30000006, 0x60000006, 0x38000006, 0x08000006, - 0x28000006, 0x78000006, 0x68000006, 0x40000006, - 0x20000006, 0x50000006, 0x48000006, 0x70000006, - 0x00000006, 0x18000006, 0x58000006, 0x10000006, - 0xb0000007, 0xe0000007, 0xb8000007, 0x88000007, - 0xa8000007, 0xf8000007, 0xe8000007, 0xc0000007, - 0xa0000007, 0xd0000007, 0xc8000007, 0xf0000007, - 0x80000007, 0x98000007, 0xd8000007, 0x90000007, - 0x30000007, 0x60000007, 0x38000007, 0x08000007, - 0x28000007, 0x78000007, 0x68000007, 0x40000007, - 0x20000007, 0x50000007, 0x48000007, 0x70000007, - 0x00000007, 0x18000007, 0x58000007, 0x10000007, - }, { /* 3 */ - 0x000000e8, 0x000000d8, 0x000000a0, 0x00000088, - 0x00000098, 0x000000f8, 0x000000a8, 0x000000c8, - 0x00000080, 0x000000d0, 0x000000f0, 0x000000b8, - 0x000000b0, 0x000000c0, 0x00000090, 0x000000e0, - 0x000007e8, 0x000007d8, 0x000007a0, 0x00000788, - 0x00000798, 0x000007f8, 0x000007a8, 0x000007c8, - 0x00000780, 0x000007d0, 0x000007f0, 0x000007b8, - 0x000007b0, 0x000007c0, 0x00000790, 0x000007e0, - 0x000006e8, 0x000006d8, 0x000006a0, 0x00000688, - 0x00000698, 0x000006f8, 0x000006a8, 0x000006c8, - 0x00000680, 0x000006d0, 0x000006f0, 0x000006b8, - 0x000006b0, 0x000006c0, 0x00000690, 0x000006e0, - 0x00000068, 0x00000058, 0x00000020, 0x00000008, - 0x00000018, 0x00000078, 0x00000028, 0x00000048, - 0x00000000, 0x00000050, 0x00000070, 0x00000038, - 0x00000030, 0x00000040, 0x00000010, 0x00000060, - 0x000002e8, 0x000002d8, 0x000002a0, 0x00000288, - 0x00000298, 0x000002f8, 0x000002a8, 0x000002c8, - 0x00000280, 0x000002d0, 0x000002f0, 0x000002b8, - 0x000002b0, 0x000002c0, 0x00000290, 0x000002e0, - 0x000003e8, 0x000003d8, 0x000003a0, 0x00000388, - 0x00000398, 0x000003f8, 0x000003a8, 0x000003c8, - 0x00000380, 0x000003d0, 0x000003f0, 0x000003b8, - 0x000003b0, 0x000003c0, 0x00000390, 0x000003e0, - 0x00000568, 0x00000558, 0x00000520, 0x00000508, - 0x00000518, 0x00000578, 0x00000528, 0x00000548, - 0x00000500, 0x00000550, 0x00000570, 0x00000538, - 0x00000530, 0x00000540, 0x00000510, 0x00000560, - 0x00000268, 0x00000258, 0x00000220, 0x00000208, - 0x00000218, 0x00000278, 0x00000228, 0x00000248, - 0x00000200, 0x00000250, 0x00000270, 0x00000238, - 0x00000230, 0x00000240, 0x00000210, 0x00000260, - 0x000004e8, 0x000004d8, 0x000004a0, 0x00000488, - 0x00000498, 0x000004f8, 0x000004a8, 0x000004c8, - 0x00000480, 0x000004d0, 0x000004f0, 0x000004b8, - 0x000004b0, 0x000004c0, 0x00000490, 0x000004e0, - 0x00000168, 0x00000158, 0x00000120, 0x00000108, - 0x00000118, 0x00000178, 0x00000128, 0x00000148, - 0x00000100, 0x00000150, 0x00000170, 0x00000138, - 0x00000130, 0x00000140, 0x00000110, 0x00000160, - 0x000001e8, 0x000001d8, 0x000001a0, 0x00000188, - 0x00000198, 0x000001f8, 0x000001a8, 0x000001c8, - 0x00000180, 0x000001d0, 0x000001f0, 0x000001b8, - 0x000001b0, 0x000001c0, 0x00000190, 0x000001e0, - 0x00000768, 0x00000758, 0x00000720, 0x00000708, - 0x00000718, 0x00000778, 0x00000728, 0x00000748, - 0x00000700, 0x00000750, 0x00000770, 0x00000738, - 0x00000730, 0x00000740, 0x00000710, 0x00000760, - 0x00000368, 0x00000358, 0x00000320, 0x00000308, - 0x00000318, 0x00000378, 0x00000328, 0x00000348, - 0x00000300, 0x00000350, 0x00000370, 0x00000338, - 0x00000330, 0x00000340, 0x00000310, 0x00000360, - 0x000005e8, 0x000005d8, 0x000005a0, 0x00000588, - 0x00000598, 0x000005f8, 0x000005a8, 0x000005c8, - 0x00000580, 0x000005d0, 0x000005f0, 0x000005b8, - 0x000005b0, 0x000005c0, 0x00000590, 0x000005e0, - 0x00000468, 0x00000458, 0x00000420, 0x00000408, - 0x00000418, 0x00000478, 0x00000428, 0x00000448, - 0x00000400, 0x00000450, 0x00000470, 0x00000438, - 0x00000430, 0x00000440, 0x00000410, 0x00000460, - 0x00000668, 0x00000658, 0x00000620, 0x00000608, - 0x00000618, 0x00000678, 0x00000628, 0x00000648, - 0x00000600, 0x00000650, 0x00000670, 0x00000638, - 0x00000630, 0x00000640, 0x00000610, 0x00000660, - } - } -}; - -const struct gost28147_param _gost28147_param_CryptoPro_3411 = -{ - { - { /* 0 */ - 0x0002d000, 0x0002a000, 0x0002a800, 0x0002b000, - 0x0002c000, 0x00028800, 0x00029800, 0x0002b800, - 0x0002e800, 0x0002e000, 0x0002f000, 0x00028000, - 0x0002c800, 0x00029000, 0x0002d800, 0x0002f800, - 0x0007d000, 0x0007a000, 0x0007a800, 0x0007b000, - 0x0007c000, 0x00078800, 0x00079800, 0x0007b800, - 0x0007e800, 0x0007e000, 0x0007f000, 0x00078000, - 0x0007c800, 0x00079000, 0x0007d800, 0x0007f800, - 0x00025000, 0x00022000, 0x00022800, 0x00023000, - 0x00024000, 0x00020800, 0x00021800, 0x00023800, - 0x00026800, 0x00026000, 0x00027000, 0x00020000, - 0x00024800, 0x00021000, 0x00025800, 0x00027800, - 0x00005000, 0x00002000, 0x00002800, 0x00003000, - 0x00004000, 0x00000800, 0x00001800, 0x00003800, - 0x00006800, 0x00006000, 0x00007000, 0x00000000, - 0x00004800, 0x00001000, 0x00005800, 0x00007800, - 0x00015000, 0x00012000, 0x00012800, 0x00013000, - 0x00014000, 0x00010800, 0x00011800, 0x00013800, - 0x00016800, 0x00016000, 0x00017000, 0x00010000, - 0x00014800, 0x00011000, 0x00015800, 0x00017800, - 0x0006d000, 0x0006a000, 0x0006a800, 0x0006b000, - 0x0006c000, 0x00068800, 0x00069800, 0x0006b800, - 0x0006e800, 0x0006e000, 0x0006f000, 0x00068000, - 0x0006c800, 0x00069000, 0x0006d800, 0x0006f800, - 0x0005d000, 0x0005a000, 0x0005a800, 0x0005b000, - 0x0005c000, 0x00058800, 0x00059800, 0x0005b800, - 0x0005e800, 0x0005e000, 0x0005f000, 0x00058000, - 0x0005c800, 0x00059000, 0x0005d800, 0x0005f800, - 0x0004d000, 0x0004a000, 0x0004a800, 0x0004b000, - 0x0004c000, 0x00048800, 0x00049800, 0x0004b800, - 0x0004e800, 0x0004e000, 0x0004f000, 0x00048000, - 0x0004c800, 0x00049000, 0x0004d800, 0x0004f800, - 0x0000d000, 0x0000a000, 0x0000a800, 0x0000b000, - 0x0000c000, 0x00008800, 0x00009800, 0x0000b800, - 0x0000e800, 0x0000e000, 0x0000f000, 0x00008000, - 0x0000c800, 0x00009000, 0x0000d800, 0x0000f800, - 0x0003d000, 0x0003a000, 0x0003a800, 0x0003b000, - 0x0003c000, 0x00038800, 0x00039800, 0x0003b800, - 0x0003e800, 0x0003e000, 0x0003f000, 0x00038000, - 0x0003c800, 0x00039000, 0x0003d800, 0x0003f800, - 0x00035000, 0x00032000, 0x00032800, 0x00033000, - 0x00034000, 0x00030800, 0x00031800, 0x00033800, - 0x00036800, 0x00036000, 0x00037000, 0x00030000, - 0x00034800, 0x00031000, 0x00035800, 0x00037800, - 0x0001d000, 0x0001a000, 0x0001a800, 0x0001b000, - 0x0001c000, 0x00018800, 0x00019800, 0x0001b800, - 0x0001e800, 0x0001e000, 0x0001f000, 0x00018000, - 0x0001c800, 0x00019000, 0x0001d800, 0x0001f800, - 0x00065000, 0x00062000, 0x00062800, 0x00063000, - 0x00064000, 0x00060800, 0x00061800, 0x00063800, - 0x00066800, 0x00066000, 0x00067000, 0x00060000, - 0x00064800, 0x00061000, 0x00065800, 0x00067800, - 0x00075000, 0x00072000, 0x00072800, 0x00073000, - 0x00074000, 0x00070800, 0x00071800, 0x00073800, - 0x00076800, 0x00076000, 0x00077000, 0x00070000, - 0x00074800, 0x00071000, 0x00075800, 0x00077800, - 0x00055000, 0x00052000, 0x00052800, 0x00053000, - 0x00054000, 0x00050800, 0x00051800, 0x00053800, - 0x00056800, 0x00056000, 0x00057000, 0x00050000, - 0x00054800, 0x00051000, 0x00055800, 0x00057800, - 0x00045000, 0x00042000, 0x00042800, 0x00043000, - 0x00044000, 0x00040800, 0x00041800, 0x00043800, - 0x00046800, 0x00046000, 0x00047000, 0x00040000, - 0x00044800, 0x00041000, 0x00045800, 0x00047800, - }, { /* 1 */ - 0x02380000, 0x02780000, 0x02600000, 0x02700000, - 0x02480000, 0x02200000, 0x02080000, 0x02000000, - 0x02180000, 0x02580000, 0x02280000, 0x02100000, - 0x02300000, 0x02500000, 0x02400000, 0x02680000, - 0x05380000, 0x05780000, 0x05600000, 0x05700000, - 0x05480000, 0x05200000, 0x05080000, 0x05000000, - 0x05180000, 0x05580000, 0x05280000, 0x05100000, - 0x05300000, 0x05500000, 0x05400000, 0x05680000, - 0x03b80000, 0x03f80000, 0x03e00000, 0x03f00000, - 0x03c80000, 0x03a00000, 0x03880000, 0x03800000, - 0x03980000, 0x03d80000, 0x03a80000, 0x03900000, - 0x03b00000, 0x03d00000, 0x03c00000, 0x03e80000, - 0x06380000, 0x06780000, 0x06600000, 0x06700000, - 0x06480000, 0x06200000, 0x06080000, 0x06000000, - 0x06180000, 0x06580000, 0x06280000, 0x06100000, - 0x06300000, 0x06500000, 0x06400000, 0x06680000, - 0x00380000, 0x00780000, 0x00600000, 0x00700000, - 0x00480000, 0x00200000, 0x00080000, 0x00000000, - 0x00180000, 0x00580000, 0x00280000, 0x00100000, - 0x00300000, 0x00500000, 0x00400000, 0x00680000, - 0x07b80000, 0x07f80000, 0x07e00000, 0x07f00000, - 0x07c80000, 0x07a00000, 0x07880000, 0x07800000, - 0x07980000, 0x07d80000, 0x07a80000, 0x07900000, - 0x07b00000, 0x07d00000, 0x07c00000, 0x07e80000, - 0x01380000, 0x01780000, 0x01600000, 0x01700000, - 0x01480000, 0x01200000, 0x01080000, 0x01000000, - 0x01180000, 0x01580000, 0x01280000, 0x01100000, - 0x01300000, 0x01500000, 0x01400000, 0x01680000, - 0x04380000, 0x04780000, 0x04600000, 0x04700000, - 0x04480000, 0x04200000, 0x04080000, 0x04000000, - 0x04180000, 0x04580000, 0x04280000, 0x04100000, - 0x04300000, 0x04500000, 0x04400000, 0x04680000, - 0x07380000, 0x07780000, 0x07600000, 0x07700000, - 0x07480000, 0x07200000, 0x07080000, 0x07000000, - 0x07180000, 0x07580000, 0x07280000, 0x07100000, - 0x07300000, 0x07500000, 0x07400000, 0x07680000, - 0x00b80000, 0x00f80000, 0x00e00000, 0x00f00000, - 0x00c80000, 0x00a00000, 0x00880000, 0x00800000, - 0x00980000, 0x00d80000, 0x00a80000, 0x00900000, - 0x00b00000, 0x00d00000, 0x00c00000, 0x00e80000, - 0x03380000, 0x03780000, 0x03600000, 0x03700000, - 0x03480000, 0x03200000, 0x03080000, 0x03000000, - 0x03180000, 0x03580000, 0x03280000, 0x03100000, - 0x03300000, 0x03500000, 0x03400000, 0x03680000, - 0x02b80000, 0x02f80000, 0x02e00000, 0x02f00000, - 0x02c80000, 0x02a00000, 0x02880000, 0x02800000, - 0x02980000, 0x02d80000, 0x02a80000, 0x02900000, - 0x02b00000, 0x02d00000, 0x02c00000, 0x02e80000, - 0x06b80000, 0x06f80000, 0x06e00000, 0x06f00000, - 0x06c80000, 0x06a00000, 0x06880000, 0x06800000, - 0x06980000, 0x06d80000, 0x06a80000, 0x06900000, - 0x06b00000, 0x06d00000, 0x06c00000, 0x06e80000, - 0x05b80000, 0x05f80000, 0x05e00000, 0x05f00000, - 0x05c80000, 0x05a00000, 0x05880000, 0x05800000, - 0x05980000, 0x05d80000, 0x05a80000, 0x05900000, - 0x05b00000, 0x05d00000, 0x05c00000, 0x05e80000, - 0x04b80000, 0x04f80000, 0x04e00000, 0x04f00000, - 0x04c80000, 0x04a00000, 0x04880000, 0x04800000, - 0x04980000, 0x04d80000, 0x04a80000, 0x04900000, - 0x04b00000, 0x04d00000, 0x04c00000, 0x04e80000, - 0x01b80000, 0x01f80000, 0x01e00000, 0x01f00000, - 0x01c80000, 0x01a00000, 0x01880000, 0x01800000, - 0x01980000, 0x01d80000, 0x01a80000, 0x01900000, - 0x01b00000, 0x01d00000, 0x01c00000, 0x01e80000, - }, { /* 2 */ - 0xb8000003, 0xb0000003, 0xa0000003, 0xd8000003, - 0xc8000003, 0xe0000003, 0x90000003, 0xd0000003, - 0x88000003, 0xc0000003, 0x80000003, 0xf0000003, - 0xf8000003, 0xe8000003, 0x98000003, 0xa8000003, - 0x38000003, 0x30000003, 0x20000003, 0x58000003, - 0x48000003, 0x60000003, 0x10000003, 0x50000003, - 0x08000003, 0x40000003, 0x00000003, 0x70000003, - 0x78000003, 0x68000003, 0x18000003, 0x28000003, - 0x38000001, 0x30000001, 0x20000001, 0x58000001, - 0x48000001, 0x60000001, 0x10000001, 0x50000001, - 0x08000001, 0x40000001, 0x00000001, 0x70000001, - 0x78000001, 0x68000001, 0x18000001, 0x28000001, - 0x38000002, 0x30000002, 0x20000002, 0x58000002, - 0x48000002, 0x60000002, 0x10000002, 0x50000002, - 0x08000002, 0x40000002, 0x00000002, 0x70000002, - 0x78000002, 0x68000002, 0x18000002, 0x28000002, - 0xb8000006, 0xb0000006, 0xa0000006, 0xd8000006, - 0xc8000006, 0xe0000006, 0x90000006, 0xd0000006, - 0x88000006, 0xc0000006, 0x80000006, 0xf0000006, - 0xf8000006, 0xe8000006, 0x98000006, 0xa8000006, - 0xb8000004, 0xb0000004, 0xa0000004, 0xd8000004, - 0xc8000004, 0xe0000004, 0x90000004, 0xd0000004, - 0x88000004, 0xc0000004, 0x80000004, 0xf0000004, - 0xf8000004, 0xe8000004, 0x98000004, 0xa8000004, - 0xb8000007, 0xb0000007, 0xa0000007, 0xd8000007, - 0xc8000007, 0xe0000007, 0x90000007, 0xd0000007, - 0x88000007, 0xc0000007, 0x80000007, 0xf0000007, - 0xf8000007, 0xe8000007, 0x98000007, 0xa8000007, - 0x38000000, 0x30000000, 0x20000000, 0x58000000, - 0x48000000, 0x60000000, 0x10000000, 0x50000000, - 0x08000000, 0x40000000, 0x00000000, 0x70000000, - 0x78000000, 0x68000000, 0x18000000, 0x28000000, - 0x38000005, 0x30000005, 0x20000005, 0x58000005, - 0x48000005, 0x60000005, 0x10000005, 0x50000005, - 0x08000005, 0x40000005, 0x00000005, 0x70000005, - 0x78000005, 0x68000005, 0x18000005, 0x28000005, - 0xb8000000, 0xb0000000, 0xa0000000, 0xd8000000, - 0xc8000000, 0xe0000000, 0x90000000, 0xd0000000, - 0x88000000, 0xc0000000, 0x80000000, 0xf0000000, - 0xf8000000, 0xe8000000, 0x98000000, 0xa8000000, - 0xb8000002, 0xb0000002, 0xa0000002, 0xd8000002, - 0xc8000002, 0xe0000002, 0x90000002, 0xd0000002, - 0x88000002, 0xc0000002, 0x80000002, 0xf0000002, - 0xf8000002, 0xe8000002, 0x98000002, 0xa8000002, - 0xb8000005, 0xb0000005, 0xa0000005, 0xd8000005, - 0xc8000005, 0xe0000005, 0x90000005, 0xd0000005, - 0x88000005, 0xc0000005, 0x80000005, 0xf0000005, - 0xf8000005, 0xe8000005, 0x98000005, 0xa8000005, - 0x38000004, 0x30000004, 0x20000004, 0x58000004, - 0x48000004, 0x60000004, 0x10000004, 0x50000004, - 0x08000004, 0x40000004, 0x00000004, 0x70000004, - 0x78000004, 0x68000004, 0x18000004, 0x28000004, - 0x38000007, 0x30000007, 0x20000007, 0x58000007, - 0x48000007, 0x60000007, 0x10000007, 0x50000007, - 0x08000007, 0x40000007, 0x00000007, 0x70000007, - 0x78000007, 0x68000007, 0x18000007, 0x28000007, - 0x38000006, 0x30000006, 0x20000006, 0x58000006, - 0x48000006, 0x60000006, 0x10000006, 0x50000006, - 0x08000006, 0x40000006, 0x00000006, 0x70000006, - 0x78000006, 0x68000006, 0x18000006, 0x28000006, - 0xb8000001, 0xb0000001, 0xa0000001, 0xd8000001, - 0xc8000001, 0xe0000001, 0x90000001, 0xd0000001, - 0x88000001, 0xc0000001, 0x80000001, 0xf0000001, - 0xf8000001, 0xe8000001, 0x98000001, 0xa8000001, - }, { /* 3 */ - 0x000000e8, 0x000000f0, 0x000000a0, 0x00000088, - 0x000000b8, 0x00000080, 0x000000a8, 0x000000d0, - 0x00000098, 0x000000e0, 0x000000c0, 0x000000f8, - 0x000000b0, 0x00000090, 0x000000c8, 0x000000d8, - 0x000001e8, 0x000001f0, 0x000001a0, 0x00000188, - 0x000001b8, 0x00000180, 0x000001a8, 0x000001d0, - 0x00000198, 0x000001e0, 0x000001c0, 0x000001f8, - 0x000001b0, 0x00000190, 0x000001c8, 0x000001d8, - 0x00000568, 0x00000570, 0x00000520, 0x00000508, - 0x00000538, 0x00000500, 0x00000528, 0x00000550, - 0x00000518, 0x00000560, 0x00000540, 0x00000578, - 0x00000530, 0x00000510, 0x00000548, 0x00000558, - 0x000004e8, 0x000004f0, 0x000004a0, 0x00000488, - 0x000004b8, 0x00000480, 0x000004a8, 0x000004d0, - 0x00000498, 0x000004e0, 0x000004c0, 0x000004f8, - 0x000004b0, 0x00000490, 0x000004c8, 0x000004d8, - 0x000002e8, 0x000002f0, 0x000002a0, 0x00000288, - 0x000002b8, 0x00000280, 0x000002a8, 0x000002d0, - 0x00000298, 0x000002e0, 0x000002c0, 0x000002f8, - 0x000002b0, 0x00000290, 0x000002c8, 0x000002d8, - 0x000005e8, 0x000005f0, 0x000005a0, 0x00000588, - 0x000005b8, 0x00000580, 0x000005a8, 0x000005d0, - 0x00000598, 0x000005e0, 0x000005c0, 0x000005f8, - 0x000005b0, 0x00000590, 0x000005c8, 0x000005d8, - 0x00000268, 0x00000270, 0x00000220, 0x00000208, - 0x00000238, 0x00000200, 0x00000228, 0x00000250, - 0x00000218, 0x00000260, 0x00000240, 0x00000278, - 0x00000230, 0x00000210, 0x00000248, 0x00000258, - 0x000007e8, 0x000007f0, 0x000007a0, 0x00000788, - 0x000007b8, 0x00000780, 0x000007a8, 0x000007d0, - 0x00000798, 0x000007e0, 0x000007c0, 0x000007f8, - 0x000007b0, 0x00000790, 0x000007c8, 0x000007d8, - 0x00000468, 0x00000470, 0x00000420, 0x00000408, - 0x00000438, 0x00000400, 0x00000428, 0x00000450, - 0x00000418, 0x00000460, 0x00000440, 0x00000478, - 0x00000430, 0x00000410, 0x00000448, 0x00000458, - 0x00000368, 0x00000370, 0x00000320, 0x00000308, - 0x00000338, 0x00000300, 0x00000328, 0x00000350, - 0x00000318, 0x00000360, 0x00000340, 0x00000378, - 0x00000330, 0x00000310, 0x00000348, 0x00000358, - 0x000003e8, 0x000003f0, 0x000003a0, 0x00000388, - 0x000003b8, 0x00000380, 0x000003a8, 0x000003d0, - 0x00000398, 0x000003e0, 0x000003c0, 0x000003f8, - 0x000003b0, 0x00000390, 0x000003c8, 0x000003d8, - 0x00000768, 0x00000770, 0x00000720, 0x00000708, - 0x00000738, 0x00000700, 0x00000728, 0x00000750, - 0x00000718, 0x00000760, 0x00000740, 0x00000778, - 0x00000730, 0x00000710, 0x00000748, 0x00000758, - 0x000006e8, 0x000006f0, 0x000006a0, 0x00000688, - 0x000006b8, 0x00000680, 0x000006a8, 0x000006d0, - 0x00000698, 0x000006e0, 0x000006c0, 0x000006f8, - 0x000006b0, 0x00000690, 0x000006c8, 0x000006d8, - 0x00000068, 0x00000070, 0x00000020, 0x00000008, - 0x00000038, 0x00000000, 0x00000028, 0x00000050, - 0x00000018, 0x00000060, 0x00000040, 0x00000078, - 0x00000030, 0x00000010, 0x00000048, 0x00000058, - 0x00000168, 0x00000170, 0x00000120, 0x00000108, - 0x00000138, 0x00000100, 0x00000128, 0x00000150, - 0x00000118, 0x00000160, 0x00000140, 0x00000178, - 0x00000130, 0x00000110, 0x00000148, 0x00000158, - 0x00000668, 0x00000670, 0x00000620, 0x00000608, - 0x00000638, 0x00000600, 0x00000628, 0x00000650, - 0x00000618, 0x00000660, 0x00000640, 0x00000678, - 0x00000630, 0x00000610, 0x00000648, 0x00000658, - } - } -}; - /* * A macro that performs a full encryption round of GOST 28147-89. */ -- 2.27.0

1 1

[PATCH 2/3] v4.0 Blowfish: Add bcrypt support.
by Stephen R. van den Berg 12 Jun '20

12 Jun '20

--- Makefile.in | 2 +- blowfish-bcrypt.c | 519 ++++++++++++++++++++++++++++++++++++++++++++++ blowfish.h | 15 ++ nettle.texinfo | 67 ++++++ 4 files changed, 602 insertions(+), 1 deletion(-) create mode 100644 blowfish-bcrypt.c diff --git a/Makefile.in b/Makefile.in index 64ff1001..77efb5c9 100644 --- a/Makefile.in +++ b/Makefile.in @@ -73,7 +73,7 @@ nettle_SOURCES = aes-decrypt-internal.c aes-decrypt.c \ aes256-set-encrypt-key.c aes256-set-decrypt-key.c \ aes256-meta.c \ arcfour.c arcfour-crypt.c \ - arctwo.c arctwo-meta.c blowfish.c \ + arctwo.c arctwo-meta.c blowfish.c blowfish-bcrypt.c \ base16-encode.c base16-decode.c base16-meta.c \ base64-encode.c base64-decode.c base64-meta.c \ base64url-encode.c base64url-decode.c base64url-meta.c \ diff --git a/blowfish-bcrypt.c b/blowfish-bcrypt.c new file mode 100644 index 00000000..c06f9e90 --- /dev/null +++ b/blowfish-bcrypt.c @@ -0,0 +1,519 @@ +/* blowfish-bcrypt.c + + The blowfish bcrypt implementation. + + Copyright (c) 2020 Stephen R. van den Berg + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +*/ + +#if HAVE_CONFIG_H +#include "config.h" +#endif + +#include <assert.h> +#include <string.h> +#include <stdlib.h> + +#include "blowfish.h" +#include "blowfish-internal.h" +#include "base64.h" + +#include "macros.h" + +#define CRYPTPLEN 7 +#define SALTLEN ((BLOWFISH_BCRYPT_BINSALT_SIZE*8+5) / 6) + +#define HASHOFFSET (CRYPTPLEN + SALTLEN) + +static const signed char radix64_decode_table[0x100] = { + /* White space is HT, VT, FF, CR, LF and SPC */ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -2, -2, -2, -2, -2, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -2, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0, 1, + 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, -1, -1, -1, -3, -1, -1, + -1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, + 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, -1, -1, -1, -1, -1, + -1, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, + 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, +}; + +static const char radix64_encode_table[64] = + "./ABCDEFGHIJKLMNOPQRSTUVWXYZ" + "abcdefghijklmnopqrstuvwxyz" + "0123456789"; + +int +blowfish_bcrypt_verify(const char *key, + const char *hashed) +{ + char newhash[BLOWFISH_BCRYPT_HASH_SIZE]; + + return blowfish_bcrypt_hash(sizeof newhash, + newhash, key, hashed, -1, (void*)0) + && !strcmp(newhash, hashed); +} + +static char *encode_radix64(char *dst, size_t len, const uint8_t *src) +{ + struct base64_encode_ctx ctx; + base64_encode_init(&ctx); + ctx.alphabet = radix64_encode_table; + dst += base64_encode_update(&ctx, dst, len, src); + dst += base64_encode_final(&ctx, dst); + *--dst = '\0'; /* Strip the trailing = */ + return dst; +} + +/* + * Large parts of the code below are based on public domain sources. + * The comments and copyright notices have been preserved. + * Any code added or modified by me is licensed under the + * licenses listed above. -- Stephen R. van den Berg + */ + +/* + * This code comes from John the Ripper password cracker, with reentrant + * and crypt(3) interfaces added, but optimizations specific to password + * cracking removed. + * + * Written by Solar Designer <solar at openwall.com> in 1998-2015. + * No copyright is claimed, and the software is hereby placed in the public + * domain. In case this attempt to disclaim copyright and place the software + * in the public domain is deemed null and void, then the software is + * Copyright (c) 1998-2015 Solar Designer and it is hereby released to the + * general public under the following terms: + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted. + * + * There's ABSOLUTELY NO WARRANTY, express or implied. + * + * It is my intent that you should be able to use this on your system, + * as part of a software package, or anywhere else to improve security, + * ensure compatibility, or for any other purpose. I would appreciate + * it if you give credit where it is due and keep your modifications in + * the public domain as well, but I don't require that in order to let + * you place this code and any modifications you make under a license + * of your choice. + * + * This implementation is fully compatible with OpenBSD's bcrypt.c for prefix + * "$2b$", originally by Niels Provos <provos at citi.umich.edu>, and it uses + * some of his ideas. The password hashing algorithm was designed by David + * Mazieres <dm at lcs.mit.edu>. For information on the level of + * compatibility for bcrypt hash prefixes other than "$2b$", please refer to + * the comments in set_key() below and to the included crypt(3) man page. + */ + +typedef uint32_t bf_key[_BLOWFISH_ROUNDS + 2]; + +/* + * Magic IV for 64 Blowfish encryptions that we do at the end. + * The string is "OrpheanBeholderScryDoubt" on big-endian. + */ +static uint32_t magic_w[6] = { + 0x4F727068, 0x65616E42, 0x65686F6C, + 0x64657253, 0x63727944, 0x6F756274 +}; + +static void swap32(uint32_t *x, int count) +{ +#if !WORDS_BIGENDIAN + do { + uint32_t tmp = *x; + tmp = (tmp << 16) | (tmp >> 16); + *x++ = ((tmp & 0x00FF00FF) << 8) | ((tmp >> 8) & 0x00FF00FF); + } while (--count); +#endif +} + +static void set_xkey(const char *key, bf_key expanded, bf_key initial, + unsigned bug, uint32_t safety) +{ + const char *ptr = key; + unsigned i, j; + uint32_t sign, diff, tmp[2]; + +/* + * There was a sign extension bug in older revisions of this function. While + * we would have liked to simply fix the bug and move on, we have to provide + * a backwards compatibility feature (essentially the bug) for some systems and + * a safety measure for some others. The latter is needed because for certain + * multiple inputs to the buggy algorithm there exist easily found inputs to + * the correct algorithm that produce the same hash. Thus, we optionally + * deviate from the correct algorithm just enough to avoid such collisions. + * While the bug itself affected the majority of passwords containing + * characters with the 8th bit set (although only a percentage of those in a + * collision-producing way), the anti-collision safety measure affects + * only a subset of passwords containing the '\xff' character (not even all of + * those passwords, just some of them). This character is not found in valid + * UTF-8 sequences and is rarely used in popular 8-bit character encodings. + * Thus, the safety measure is unlikely to cause much annoyance, and is a + * reasonable tradeoff to use when authenticating against existing hashes that + * are not reliably known to have been computed with the correct algorithm. + * + * We use an approach that tries to minimize side-channel leaks of password + * information - that is, we mostly use fixed-cost bitwise operations instead + * of branches or table lookups. (One conditional branch based on password + * length remains. It is not part of the bug aftermath, though, and is + * difficult and possibly unreasonable to avoid given the use of C strings by + * the caller, which results in similar timing leaks anyway.) + * + * For actual implementation, we set an array index in the variable "bug" + * (0 means no bug, 1 means sign extension bug emulation) and a flag in the + * variable "safety" (bit 16 is set when the safety measure is requested). + * Valid combinations of settings are: + * + * Prefix "$2a$": bug = 0, safety = 0x10000 + * Prefix "$2b$": bug = 0, safety = 0 + * Prefix "$2x$": bug = 1, safety = 0 + * Prefix "$2y$": bug = 0, safety = 0 + */ + + sign = diff = 0; + + for (i = 0; i < _BLOWFISH_ROUNDS + 2; i++) { + tmp[0] = tmp[1] = 0; + for (j = 0; j < 4; j++) { + tmp[0] <<= 8; + tmp[0] |= (unsigned char)*ptr; /* correct */ + tmp[1] <<= 8; + tmp[1] |= (signed char)*ptr; /* bug */ +/* + * Sign extension in the first char has no effect - nothing to overwrite yet, + * and those extra 24 bits will be fully shifted out of the 32-bit word. For + * chars 2, 3, 4 in each four-char block, we set bit 7 of "sign" if sign + * extension in tmp[1] occurs. Once this flag is set, it remains set. + */ + if (j) + sign |= tmp[1] & 0x80; + if (!*ptr) + ptr = key; + else + ptr++; + } + diff |= tmp[0] ^ tmp[1]; /* Non-zero on any differences */ + + expanded[i] = tmp[bug]; + initial[i] = _nettle_blowfish_initial_ctx.p[i] ^ tmp[bug]; + } + +/* + * At this point, "diff" is zero if the correct and buggy algorithms produced + * exactly the same result. If so and if "sign" is non-zero, which indicates + * that there was a non-benign sign extension, this means that we have a + * collision between the correctly computed hash for this password and a set of + * passwords that could be supplied to the buggy algorithm. Our safety measure + * is meant to protect from such many-buggy to one-correct collisions, by + * deviating from the correct algorithm in such cases. Let's check for this. + */ + diff |= diff >> 16; /* still zero if exact match */ + diff &= 0xffff; /* ditto */ + diff += 0xffff; /* bit 16 set if "diff" was non-zero (on non-match) */ + sign <<= 9; /* move the non-benign sign extension flag to bit 16 */ + sign &= ~diff & safety; /* action needed? */ + +/* + * If we have determined that we need to deviate from the correct algorithm, + * flip bit 16 in initial expanded key. (The choice of 16 is arbitrary, but + * let's stick to it now. It came out of the approach we used above, and it's + * not any worse than any other choice we could make.) + * + * It is crucial that we don't do the same to the expanded key used in the main + * Eksblowfish loop. By doing it to only one of these two, we deviate from a + * state that could be directly specified by a password to the buggy algorithm + * (and to the fully correct one as well, but that's a side-effect). + */ + initial[0] ^= sign; +} + +static int ibcrypt(size_t length, char *dst, + const char *key, const char *scheme, + int minlog2rounds, + int log2rounds, const uint8_t *salt) +{ + struct { + struct blowfish_ctx ctx; + bf_key expanded_key; + union { + uint32_t salt[4]; + uint32_t output[6]; + } binary; + } data; + uint8_t psalt[BLOWFISH_BCRYPT_BINSALT_SIZE]; + uint32_t L, R; + uint32_t *ptr; + uint32_t count; + int i; + size_t lenscheme = strlen(scheme); + unsigned cscheme; + unsigned bug = 0; + uint32_t safety = 0; + if (length < BLOWFISH_BCRYPT_HASH_SIZE || + lenscheme < 2) + return 0; + + if (lenscheme >= 3 && *scheme++ != '$') + return 0; + if (*scheme++ != '2') + return 0; + + switch (cscheme = *scheme++) { + default: + return 0; + case 'a': safety = 0x10000; + break; + case 'x': bug = 1; + break; + case 'b': case 'y': + break; + } + + if (lenscheme >= 4) { + if (*scheme++ != '$') + return 0; + if (lenscheme >= 6) { + if (log2rounds < 0) + log2rounds = atoi(scheme); + scheme += 2; + if (lenscheme >= CRYPTPLEN && *scheme++ != '$') + return 0; + if (lenscheme >= HASHOFFSET && !salt) { + struct base64_decode_ctx ctx; + size_t saltlen = BLOWFISH_BCRYPT_BINSALT_SIZE; + + base64_decode_init(&ctx); + ctx.table = radix64_decode_table; + + if (!base64_decode_update(&ctx, &saltlen, (uint8_t *) data.binary.salt, + SALTLEN, scheme) + || saltlen != BLOWFISH_BCRYPT_BINSALT_SIZE) + return 0; + } + } + } + + if (salt) + memcpy(data.binary.salt, salt, BLOWFISH_BCRYPT_BINSALT_SIZE); + else if (lenscheme < HASHOFFSET) + return 0; + memcpy(psalt, data.binary.salt, BLOWFISH_BCRYPT_BINSALT_SIZE); + swap32(data.binary.salt, 4); + + if (log2rounds < minlog2rounds || log2rounds > 31) + return 0; + count = (uint32_t)1 << log2rounds; + + set_xkey(key, data.expanded_key, data.ctx.p, bug, safety); + memcpy(data.ctx.s, _nettle_blowfish_initial_ctx.s, sizeof(data.ctx.s)); + + L = R = 0; + for (i = 0; i < _BLOWFISH_ROUNDS + 2; i += 2) { + L ^= data.binary.salt[i & 2]; + R ^= data.binary.salt[(i & 2) + 1]; + _nettle_blowfish_encround(&data.ctx, &L, &R); + data.ctx.p[i] = L; + data.ctx.p[i + 1] = R; + } + + ptr = data.ctx.s[0]; + do { + ptr += 4; + L ^= data.binary.salt[(_BLOWFISH_ROUNDS + 2) & 3]; + R ^= data.binary.salt[(_BLOWFISH_ROUNDS + 3) & 3]; + _nettle_blowfish_encround(&data.ctx, &L, &R); + *(ptr - 4) = L; + *(ptr - 3) = R; + + L ^= data.binary.salt[(_BLOWFISH_ROUNDS + 4) & 3]; + R ^= data.binary.salt[(_BLOWFISH_ROUNDS + 5) & 3]; + _nettle_blowfish_encround(&data.ctx, &L, &R); + *(ptr - 2) = L; + *(ptr - 1) = R; + } while (ptr < &data.ctx.s[3][0xFF]); + + do { + int done; + + for (i = 0; i < _BLOWFISH_ROUNDS + 2; i += 2) { + data.ctx.p[i] ^= data.expanded_key[i]; + data.ctx.p[i + 1] ^= data.expanded_key[i + 1]; + } + + done = 0; + do { + uint32_t tmp1, tmp2, tmp3, tmp4; + + L = R = 0; + ptr = data.ctx.p; + do { + ptr += 2; + _nettle_blowfish_encround(&data.ctx, &L, &R); + *(ptr - 2) = L; + *(ptr - 1) = R; + } while (ptr < &data.ctx.p[_BLOWFISH_ROUNDS + 2]); + + ptr = data.ctx.s[0]; + do { + ptr += 2; + _nettle_blowfish_encround(&data.ctx, &L, &R); + *(ptr - 2) = L; + *(ptr - 1) = R; + } while (ptr < &data.ctx.s[3][0xFF]); + + if (done) + break; + done = 1; + + tmp1 = data.binary.salt[0]; + tmp2 = data.binary.salt[1]; + tmp3 = data.binary.salt[2]; + tmp4 = data.binary.salt[3]; + for (i = 0; i < _BLOWFISH_ROUNDS; i += 4) { + data.ctx.p[i] ^= tmp1; + data.ctx.p[i + 1] ^= tmp2; + data.ctx.p[i + 2] ^= tmp3; + data.ctx.p[i + 3] ^= tmp4; + } + data.ctx.p[16] ^= tmp1; + data.ctx.p[17] ^= tmp2; + } while (1); + } while (--count); + + for (i = 0; i < 6; i += 2) { + L = magic_w[i]; + R = magic_w[i + 1]; + + count = 64; + do + _nettle_blowfish_encround(&data.ctx, &L, &R); + while (--count); + + data.binary.output[i] = L; + data.binary.output[i + 1] = R; + } + + *dst++ = '$'; + *dst++ = '2'; + *dst++ = cscheme; + *dst++ = '$'; + *dst++ = '0' + log2rounds / 10; + *dst++ = '0' + log2rounds % 10; + *dst++ = '$'; + dst = encode_radix64(dst, BLOWFISH_BCRYPT_BINSALT_SIZE, psalt) - 1; + + swap32(data.binary.output, 6); +/* This has to be bug-compatible with the original implementation, so + only encode 23 of the 24 bytes. */ + encode_radix64(dst, 23, (uint8_t *) data.binary.output); + return cscheme; +} + +/* + * Please preserve the runtime self-test. It serves two purposes at once: + * + * 1. We really can't afford the risk of producing incompatible hashes e.g. + * when there's something like gcc bug 26587 again, whereas an application or + * library integrating this code might not also integrate our external tests or + * it might not run them after every build. Even if it does, the miscompile + * might only occur on the production build, but not on a testing build (such + * as because of different optimization settings). It is painful to recover + * from incorrectly-computed hashes - merely fixing whatever broke is not + * enough. Thus, a proactive measure like this self-test is needed. + * + * 2. We don't want to leave sensitive data from our actual password hash + * computation on the stack or in registers. Previous revisions of the code + * would do explicit cleanups, but simply running the self-test after hash + * computation is more reliable. + * + * The performance cost of this quick self-test is around 0.6% at the "$2a$08" + * setting. + */ +int blowfish_bcrypt_hash(size_t length, char *dst, + const char *key, const char *scheme, + int log2rounds, const uint8_t *salt) +{ + const char *test_pw = "8b \xd0\xc1\xd2\xcf\xcc\xd8"; + const char *test_scheme = "$2a$00$abcdefghijklmnopqrstuu"; + static const char * const test_hashes[2] = + {"i1D709vfamulimlGcq0qq3UvuUasvEa\0\x55", /* 'a', 'b', 'y' */ + "VUrPmXD6q/nVSSp7pNDhCR9071IfIRe\0\x55"}; /* 'x' */ + const char *test_hash = test_hashes[0]; + int cscheme; + int ok; + struct { + char s[HASHOFFSET + 1]; + char o[HASHOFFSET + 31 + 1 + 1 + 1]; + } buf; + + if (length) + *dst = '\0'; +/* Hash the supplied password */ + cscheme = ibcrypt(length, dst, key, scheme, 4, log2rounds, salt); + +/* + * Do a quick self-test. It is important that we make both calls to ibcrypt() + * from the same scope such that they likely use the same stack locations, + * which makes the second call overwrite the first call's sensitive data on the + * stack and makes it more likely that any alignment related issues would be + * detected by the self-test. + */ + memcpy(buf.s, test_scheme, sizeof(buf.s)); + + if (cscheme) + test_hash = test_hashes[(buf.s[2] = cscheme) == 'x']; + + memset(buf.o, 0x55, sizeof(buf.o)); + buf.o[sizeof(buf.o) - 1] = 0; + ok = ibcrypt(sizeof(buf.o) - (1 + 1), buf.o, test_pw, + buf.s, 0, -1, (void*)0); + + ok = (ok && + !memcmp(buf.o, buf.s, HASHOFFSET) && + !memcmp(buf.o + HASHOFFSET, test_hash, 31 + 1 + 1 + 1)); + + { + const char *k = "\xff\xa3" "34" "\xff\xff\xff\xa3" "345"; + bf_key ae, ai, ye, yi; + set_xkey(k, ae, ai, 0, 0x10000); /* $2a$ */ + set_xkey(k, ye, yi, 0, 0); /* $2y$ */ + ai[0] ^= 0x10000; /* undo the safety (for comparison) */ + ok = ok && ai[0] == 0xdb9c59bc && ye[17] == 0x33343500 && + !memcmp(ae, ye, sizeof(ae)) && + !memcmp(ai, yi, sizeof(ai)); + } + + return ok && !!cscheme; +} diff --git a/blowfish.h b/blowfish.h index bcdc7cb6..af48e20f 100644 --- a/blowfish.h +++ b/blowfish.h @@ -46,6 +46,8 @@ extern "C" { #define blowfish128_set_key nettle_blowfish128_set_key #define blowfish_encrypt nettle_blowfish_encrypt #define blowfish_decrypt nettle_blowfish_decrypt +#define blowfish_bcrypt_hash nettle_blowfish_bcrypt_hash +#define blowfish_bcrypt_verify nettle_blowfish_bcrypt_verify #define BLOWFISH_BLOCK_SIZE 8 @@ -60,6 +62,9 @@ extern "C" { #define _BLOWFISH_ROUNDS 16 +#define BLOWFISH_BCRYPT_HASH_SIZE (60 + 1) /* Including null-terminator */ +#define BLOWFISH_BCRYPT_BINSALT_SIZE 16 /* Binary string size */ + struct blowfish_ctx { uint32_t s[4][256]; @@ -81,6 +86,16 @@ void blowfish_decrypt(const struct blowfish_ctx *ctx, size_t length, uint8_t *dst, const uint8_t *src); +int +blowfish_bcrypt_hash(size_t length, + char *dst, + const char *key, + const char *scheme, + int log2rounds, + const uint8_t *salt); +int +blowfish_bcrypt_verify(const char *key, + const char *hashed); #ifdef __cplusplus } diff --git a/nettle.texinfo b/nettle.texinfo index 995d5de8..75e18b58 100644 --- a/nettle.texinfo +++ b/nettle.texinfo @@ -1513,6 +1513,73 @@ in any other way. Analogous to @code{blowfish_encrypt} @end deftypefun +@deftypefun int blowfish_bcrypt_hash (size_t @var{length}, char *@var{dst}, const char *@var{key}, const char *@var{scheme}, int @var{log2rounds}, const uint8_t *@var{salt}) +Compute the bcrypt password hash. +The function will return @code{0} if the hash cannot be computed +due to invalid input. +The function will return @code{1} and store the computed hash +in the array pointed to by @var{dst}. The hash is computed based +on the chosen @var{scheme}, number of rounds @var{log2rounds} and +specified @var{salt}. + +@var{length} must be at least @code{BLOWFISH_BCRYPT_HASH_SIZE}. + +@var{dst} must point to a character array of the specified @var{length}. + +@var{key} contains the zero terminated plaintext password string. + +@var{scheme} contains either just the chosen scheme (valid schemes +are: @code{2a}, @code{2b}, @code{2x} or @code{2y}), or +(the prefix of) an existing hashed password (typically @code{$2b$10$...}). + +@var{log2rounds} contains the log2 of the number of encryption rounds +that must be used to compute the hash. If it is @code{-1} the value +will be extracted from @var{scheme}. + +@var{salt} should point to an array of @code{BLOWFISH_BCRYPT_BINSALT_SIZE} +random bytes to be used to perturb the hash computation. If it is @code{NULL} +the salt will be extracted from @var{scheme}. + +Sample code to generate a bcrypt hash: +@example +char cleartxtpassword[] = "ExamplePassword"; +uint8_t salt[BLOWFISH_BCRYPT_BINSALT_SIZE]; +@dots{} +/* Make sure that salt is filled with random bytes */ +@dots{} +char hashedresult[BLOWFISH_BCRYPT_HASH_SIZE]; +int result = blowfish_bcrypt(sizeof(hashedresult), hashedresult, + cleartxtpassword, "2b", 10, salt); +if (result) + printf("%s\n", hashedresult); +@end example +@end deftypefun + +@deftypefun int blowfish_bcrypt_verify (const char *@var{key}, const char *@var{hashed}) +Verifies the bcrypt password hash against the supplied plaintext password. +The function will return @code{0} if the password does not match. +The function will return @code{1} if the password matches. + +@var{key} contains the zero terminated plaintext password string. + +@var{hashed} contains the zero terminated hashed string to compare with. + +Sample code to verify a bcrypt hash: +@example +char cleartxtpassword[] = "ExamplePassword"; +char existinghashed[] = + "$2y$" /* Hash algorithm version */ + "10" /* 2^10 hash rounds (strength) */ + "$" /* separator */ + "1b2lPgo4XumibnJGN3r3sO" /* base64 encoded 16-byte salt */ + "u7wE7xNfYDKlAxZffJDCJdVfFTAyevu"; /* Hashedpart */ +if (blowfish_bcrypt_verify(cleartxtpassword, existinghashed)) + printf("Password is correct."); +else + printf("Password is incorrect."); +@end example +@end deftypefun + @subsection Camellia @cindex Camellia -- 2.20.1

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

nettle-bugs