Basically a first implementation of the chacha20 Poly1305 algorithm

[mikerabat]

Hope this works as intendent....

Basically I implemented the base chacha20 algorithm and a poly1305 mode following the GCM mode implementation.
I need to say I did quite some rewriting on the base class (don't be mad...) to have a more leaner interface in the
cipher classes.

In addtion I rewrote the GCM part such that it works on streams too - multiple encode/decode calls are now allowed.
This was a rewrite of bug #87 based on the new base authenticator class.

I did not rewrite teh CCM stuff - simply I didn't knew how...

There are also SSE, AVX version of the chacha algorithm so I hope I managed to put it correctly in there - unfortunately I was not
successfull with a SSE/AVX version of the Poly1305 algorithm, this one is way more complicated that chacha...

Let me know what you think of the code...

[mikerabat]

Added code to implement XChaCha20 as well

[MHumm]

I need to find the time to look at this over the next few weeks. Please be a bit patient! I value your contribution. Thanks for the warning about the changes in the base class. Yes, they need to be evaluated carefully.

Does your code contain unit tests?
If so did you specify the source of the test data? Best would be an official source e.g. from the original developers or from a standards body?

[mikerabat]

Yes.. at least the two vectors from the original RFC. I found a bunch of test vectors here:
https://github.com/C2SP/wycheproof/blob/main/testvectors/chacha20_poly1305_test.json
do you think it is a good idea to implement them here? I would try to convert them into your own format (if possible).

This commit would also tackle the problems with the encodings - I changed the base class so it could be used in such a way. The chacha and GCM modules are already converted. Let me know what you think of that.

[MHumm]

We should try to have unit tests for as many areas of the library as possible.

[MHumm]

Refering to this one: "do you think it is a good idea to implement them here?"
Would it be possible to check if their collection also contains those or at least some of those test vectors you found from the RFC? That way it should be safe to assume that their vectors really test the same thing etc. and are trustworthy.
Everybody can implement some algorithm and claim it is "algorithm X" and make up test vectors...
But with your tests we would be as safe as we can currently check.

[mikerabat]

They actually include the RFC test in their test suit. It is even the first test....
The RFC test vector is actually already in my unit test....
I'm wondering ... how do you crreate your test vectors - what sources do you use?

[MHumm]

I try to find original test vectors. Ok, when I took the library over there was some "crude" automatic test program which worked on a text file which contained the test vectors. And there were about a dozend unit tests for some newer date helper functions (from DECUtil). I started to implement unit tests for those test vectors in the text file, without questioning where they came from.

Later on somebody reported a few flaws and fixes for them and that made me start to look for original test vectors for all the hash algorithms. After all those years I couldn't find them for all the hash algorithms but for those where I found them I added them.

For KDF it was like this: the original author had implemented KDF2. At least that's what he had claimed. I found there were no tests yet and found the KDF specification and the test vectors for KDF2 didn't match the results produced by DEC. So I started to investigate and found out, that the original author had implemented KDF1 and KDF2 and 3 are only small deviations from that. The only difference is the order of 2-3 lines of code in their main method.

So I implemented these two as well and added tests from that specification.

=> I try to find the original specification of the algorithm, be it some PDF, an RFC whatever and check if that contains test vectors. Also NIST is sometimes a source for test vectors or other standard bodies.

Does this answer your question?

[mikerabat]

I see... thanks for the info.
I checked the sources and the test vectors are mainly from:
https://github.com/C2SP/wycheproof
which seems to be a reliable source I guess.

[mikerabat]

These two last commits include the found test vector - about 300 tests.
I added these in these commits. The tests include some edge cases and also some tests that actually "need" to fail...

[mikerabat]

Just a few updates here:

• the commit changed the way the chacha20 implementation uses the ifdefs
• I needed to update the .inc file - it did not work for 64 bit windows. Basically it always fell back to NO_ASM
• there was a problem in the 64bit byteswap routine

And... There are now new routines that utilizes the Intel AES instructions + a class variable that allows to fall back to the purepascal implementation (the original).

[mikerabat]

Any thoughts about the update?

[mikerabat]

Just checking ;) any news if that implementation is useful? There is now also some AVX assembler code for the Poly1305 implementation which does some decent speed increase I guess...

[MHumm]

Thanks for reaching out and sorry that I still haven't found the time to look at it!
I still think it will be a good thing, once integrated. I need to check your architectural changes. To which branch did you contribute this? I'd like to create a special development branch for this for you. I guess it would make my handling of this a bit easier.

I can also tell you that I only should have time for this starting in summer, as I'm still in the middle of my preparations for a lecture at the local university (my first one) which will start in March and that's not the only time bound thing at the moment. I do hope that after the international Pascal Congress, where I'm a speaker (yes, I still need to prepare that one as well), see here: https://pascalcongress.com/talks.html things should get a bit calmer and I do hope to find then a bit of time for DEC again.

Can we agree on moving this development to its own branch? Is this ok for you?

[mikerabat]

I actually was working on the development branch (a fork of it) - I created a pull request which seems to commit to the master branch though - I hope that is ok...

Congrats by the way to the IPC talk and your

course in the university :) I hope these will rock 👍

I don't know if it is actually necessary to have a separate branch to work on - I can still use my fork and create pull requests right? (I'm not so fluent with git and try my best ;)

[MHumm]

I'm not so fluent with git either and I'd really like you to not work on the master branch. For this I ccreated a new branch for you from the current state of the development branch:
https://github.com/MHumm/DelphiEncryptionCompendium/tree/development_chacha20poly1305
It would be really nice if we could get your pull requests pointed to that one. Maybe we have to seek for assistance in the community for doing this.

[mikerabat]

Oh great :) thanks for that - I was able to change the branch target

[MHumm]

Great that you managed this!

[mikerabat]

Is the PR dead or unwanted?
No comments just sliced in small chunks with some AI comments on the commit?

[MHumm]

No, the PR is not really dead. I'm sorry for not finding the time up to now to integrate this.
I guess you have looked at the commit history of lately and found out that I have, with big help from somebody fixed a few outstanding bugs and there's one further PR to review and integrate before we're looking at yours. I hope to get the other PR out of the way over the weekend.

About yours: the algorithm provided was deemed really useful, it just might be that we need to change some structure things as some internal stuff has been changed lately in order to be able to fix those bugs and make handling more uniform. For instance some stream handling stuff (if I remember correctly) now also requires to call Done at the end.

[omonien]

Hi Mike,

definitely not dead, and sorry for the long silence on this. Your PR turned out to be really valuable. Because it touches both the AEAD architecture and adds a lot of new algorithm and SIMD code, we split it into smaller packages so each part can be reviewed and tested on its own. The architecture part (new authenticator base, multi-call GCM, CCM via Done) has already landed in development through #102, #105, #106 and #107, building directly on your work.

The ChaCha20/Poly1305 part and your AES-NI routines are prepared on our side and currently being brought up to date and tested. Given the complexity, especially of the SSE/AVX code, this will still take some time. The AI comments you saw came from automated review tools we used during the split, sorry if that felt impersonal.

Thanks a lot for your contribution, it was by no means in vain.

Best regards,
Olaf

[MHumm]

What I'd need to check is, whether for the AVX routines also pure pascal versions exist, as not all CPUs have AVX, especially not on other platforms. But we'll see this soon I guess.

[omonien]

Hi Markus,

good point, and we agree: every assembler routine needs a pure Pascal version anyway. In Delphi 13.x only the two classic x86 compilers accept assembler at all, and only in x86/x64 syntax. DCC32 (Win32) accepts both asm blocks inside Pascal routines and pure asm routines. DCC64 (Win64) accepts only pure asm routines; an asm block inside a Pascal routine fails with E1025 Unsupported language feature: 'ASM'. Every other Delphi compiler, i.e. the LLVM based ones for Windows on Arm (DCCARM64EC), Android 32/64, iOS device and simulator, macOS x64/ARM64 and Linux64, rejects every asm form with E1025, and no Delphi compiler has ever accepted ARM/AArch64 assembler. For Arm64EC Embarcadero explicitly says to "provide a Pascal code equivalent" (ARM64EC docs). I reproduced this with Delphi 13.1 for Win32, Win64 and Arm64EC.

Hand written ARM/NEON code could only come in as an externally built object file or static library per target (external 'lib.a' name ...; {$L} fails with E1030 on Arm64EC), and Delphi has no NEON intrinsics. That is not a reasonable path for a pure Pascal library like DEC.

So our rule for the ChaCha20/Poly1305 and AES-NI packages is: every SSE/AVX/AES-NI routine has an equivalent pure Pascal implementation with identical results, the asm is only compiled for CPUX86/CPUX64 and only used after a runtime CPU feature check, and the tests compare both paths on the same vectors. Two pitfalls we noticed: ASSEMBLER is also defined on Win64, so an asm block mixed into Pascal code must be guarded with {$IFDEF CPUX86}, not {$IFDEF ASSEMBLER}. And on Arm64EC CPUARM is defined together with CPUARM64, so {$IFDEF CPUARM} must not be used to mean 32 bit ARM.

Best regards,
Olaf

[mikerabat]

for every method I implemented here there actually is a pue pascal implementation - this was actually the starting point ;)
Just FYI: there is also a little class that checks for the cpu features (see DECCPUSupport.pas) . That ensures that in case the
cpu does not feature the selected api the automatic fallback to the pure pascal implementation is used.

The code also uses quite a few ifdefs to check if the platform is correct. For ARM the fallback should be automatically the pure pascal implementation - also the cpu flags class flags all the features to "false".

[MHumm]

Sorry, but I'm not too well with Git yet. I had created a branch for getting this new algorithm development stabilized (means: adapted to the changes we meanwhile made on our end) and due to what I made the pull request was closed. But the work is far from finished. I tried to resolve all merge conflicts but seem to have intorduced some bugs in that branch now. It will take some more time to fix all these bugs I introduced. Please be a bit more patient while I'm trying to understand what I broke and how to fix it.

[MHumm]

You might notice that it is officially in the development branch now. We still might do some further formatting, but it's in the next release for sure. Thanks for contributing this and with the CPUID class we should have a basis for further optimized ASM implementations of algorithms.