Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor and add support classes for KDBX 4 #1179

Merged
merged 17 commits into from
Nov 26, 2017
Merged

Refactor and add support classes for KDBX 4 #1179

merged 17 commits into from
Nov 26, 2017

Conversation

angelsl
Copy link
Contributor

@angelsl angelsl commented Nov 12, 2017

The remaining changes are here. I will open a PR once this gets merged.

Description

Refer to commit log

Motivation and context

#148

How has this been tested?

Screenshots (if appropriate):

Types of changes

  • ✅ New feature (non-breaking change which adds functionality)

Checklist:

  • ✅ I have read the CONTRIBUTING document. [REQUIRED]
  • ✅ My code follows the code style of this project. [REQUIRED]
  • ✅ All new and existing tests passed. [REQUIRED]
  • ✅ I have compiled and verified my code with -DWITH_ASAN=ON. [REQUIRED]
  • ✅ My change requires a change to the documentation and I have updated it accordingly.
  • ✅ I have added tests to cover my changes.

@angelsl
Copy link
Contributor Author

angelsl commented Nov 12, 2017

Travis is failing due to too old libgcrypt.

@Antidote1911
Copy link

nice work !

@droidmonkey
Copy link
Member

droidmonkey commented Nov 12, 2017

Since you renamed the old KeePass2 format read/write to Kdbx3... would it make sense to make the new one Kdbx4....

Really we should be instantiating a single common read/write shell class and passing the specific reader/writer as a constructor parameter. This would ensure adding a new format would be painless and independent.

We can implement that more advanced usage later, but it might be smart to rename KeePass2.cpp to Kdbx4Reader/Writer.cpp

@TheZ3ro
Copy link
Contributor

TheZ3ro commented Nov 12, 2017

@droidmonkey the Kdbx4 classes are in the next PR this is only for the initial renaming

@droidmonkey
Copy link
Member

sweet! Please disregard my post 😄

}
}
if (updateTransformSalt) {
m_data.kdf->randomizeSalt();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@angelsl We could name this function updateTranformSalt, or rename the option randomizeSalt, just to be consistent.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

randomizeTransformSalt()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

if (error) {
qWarning("Gcrypt error (ctor): %s", gcry_strerror(error));
qWarning("Gcrypt error (ctor): %s", gcry_strsource(error));
}
Q_ASSERT(error == 0); // TODO: error handling
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@angelsl not sure how we should be handling those errors, but I guess we should at some point. Right now it'll just fail silently unless the application was built in debug mode.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but I think this PR is starting to be quite large..

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally the calling function would be notified that there was an error so it could display a notification to the user. Perhaps we could use a generic "cryptoError" signal that includes the error string and can be listened to and acted upon.

: d_ptr(new CryptoHashPrivate())
{
Q_D(CryptoHash);

Q_ASSERT(Crypto::initalized());

int algoGcrypt;
int algoGcrypt = -1;
unsigned int flagsGcrypt = 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@angelsl What about using the GCRY_MD_FLAG_SECURE flag by default? From the doc:

Allocate all buffers and the resulting digest in "secure memory". Use this is the hashed data is highly confidential.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added that.

return m_cipher.init(CryptoHash::hash(key, CryptoHash::Sha256),
KeePass2::INNER_STREAM_SALSA20_IV);
switch (m_cipher.algorithm()) {
case SymmetricCipher::Salsa20:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@angelsl case and switch on same indentation levels

KeePass2::ProtectedStreamAlgo KeePass2::idToProtectedStreamAlgo(quint32 id)
{
switch (id) {
case static_cast<quint32>(KeePass2::ArcFourVariant):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@angelsl case and switch on same indentation level

Copy link
Member

@droidmonkey droidmonkey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this is tremendous work. Needs to be simplified (kdf fields!) and heavily tested.

@@ -374,6 +323,7 @@ void Database::setEmitModified(bool value)
void Database::copyAttributesFrom(const Database* other)
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is only used in one instance (ToDbExporter) which is not used in the code at all. I recommend in PR2 that we remove this function and the ToDbExporter code.

@@ -479,3 +429,38 @@ QString Database::saveToFile(QString filePath)
return saveFile.errorString();
}
}

Kdf* Database::kdf() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommend rename to getKdf() and also return const Kdf*.

@@ -139,7 +139,7 @@ QByteArray int16ToBytes(qint16 num, QSysInfo::Endian byteOrder)
qToLittleEndian<qint16>(num, reinterpret_cast<uchar*>(ba.data()));
}
else {
qToBigEndian<qint64>(num, reinterpret_cast<uchar*>(ba.data()));
qToBigEndian<qint16>(num, reinterpret_cast<uchar*>(ba.data()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow great catch!

QByteArray sha512Test = CryptoHash::hash("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq",
CryptoHash::Sha512);

if (sha512Test != QByteArray::fromHex("204a8fc6dda82f0a0ced7beb8e08a41657c16ef468b228a8279be331a703c33596fd15c13b1b07f9aa1d3bea57789ca031ad85c7a71dd70354ec631238ca3445")) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please post the reference that confirms this is the correct sequence. Other crypto tests pulled known test strings from RFC standards docs. https://tools.ietf.org/html/rfc4231#section-4.1

@@ -285,3 +308,30 @@ bool Crypto::testSalsa20()

return true;
}

bool Crypto::testChaCha20() {
QByteArray chacha20Key = QByteArray::fromHex("0000000000000000000000000000000000000000000000000000000000000000");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, please add comment that points to the reference that confirms the test/result strings. https://tools.ietf.org/html/rfc7539#appendix-A.2

quint32 version() const;
BaseKeePass2Reader* reader() const;
private:
QScopedPointer<BaseKeePass2Reader> m_reader;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am confused why this class both inherits from BaseKeePass2Reader and implements a worker of type BaseKeePass2Reader. Either use the worker or use yourself, pick one.

virtual QString errorString() override;

private:
QScopedPointer<BaseKeePass2Writer> m_writer;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment, don't inherit and implement the same base class.

}
}

void DatabaseSettingsWidget::displayKdf(const Kdf& kdf)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function drives me bonkers. Needs to be implemented as a form and when fields get refactored... simplified immensely.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, please, please make this a from. Don't convolute the CPP files with GUI boilerplate code.


#include "TestKeePass2XmlReader.h"

QTEST_GUILESS_MAIN(TestKdbx3XmlReader)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where did the rest of these tests go?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look at CMakeLists.

Copy link
Member

@droidmonkey droidmonkey Nov 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK... so there are two cpp files, one is basically blank. Why are the TestKdbx3XmlReader member functions defined in TestKeePass2XmlReader.cpp instead of it's own cpp file??

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess you didn't take a look at the 4 commits that were left out of this PR.

TestKeePass2XmlReader has tests that instantiate KeePass2XmlReader directly like this one. In order to avoid having to duplicate the test functions between Kdbx3XmlReader and Kdbx4XmlReader, I pulled out the XML read and writes into virtual functions that use the correct XmlReader class. I decided to keep them in the same source file so that if these functions are ever changed, it is less likely that one half is forgotten, and so that it is easier to verify that they are identical aside from the different XmlReader class used.

The better solution, arguably, would have been to use templates, but Qt moc does not support templates.

Another solution would have been making the two XmlReader classes a subclass of a base XmlReader class like the other Reader classes, but since the XmlReaders are only ever used by the Reader classes themselves, I didn't think that was appropriate: the XmlReader classes are implementation details of the Reader classes and it does not make sense for them to be sibling classes. The only way in which they are similar is that they both take XML data (that happens to be similarly structured) and spit out a database. Arguably the XmlReader classes should be nested classes of the Reader classes, but I digress.

Finally, in case you are wondering why I didn't stick both QTEST_GUILESS_MAIN in the TestKeePass2XmlReader file, QTEST_GUILESS_MAIN is a macro that expands to a definition of int main.

@@ -512,6 +552,10 @@ void TestKeePass2XmlReader::testRepairUuidHistoryItem()
QVERIFY(!entry->uuid().isNull());
QVERIFY(!historyItem->uuid().isNull());
QCOMPARE(historyItem->uuid(), entry->uuid());

if (db) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can just delete db even if its nullptr (its a noop)

@angelsl
Copy link
Contributor Author

angelsl commented Nov 19, 2017

The point of the KDF fields thing is to avoid having these details hardcoded in the GUI.

If you feel that is OK, then yes, we can drop the whole list-of-fields thing, but I will not personally author that.

@angelsl
Copy link
Contributor Author

angelsl commented Nov 19, 2017

I will work on the rest of the comments in a few days.

It would do you good to look at the rest of the PR, @droidmonkey, so you can understand why some things were done the way they were done, so that I don't have to explain myself repeatedly. We went through some of this in March already, in case you don't remember.

Some of the inconsistencies are there because the original file had them. I'm trying to minimise code churn in this PR. Perhaps you'd like to fix the formatting and whatnot first in a separate PR, and then I, or someone else, can rebase this again.

I'm not very motivated to get this merged, especially if I keep getting comments that result from not fully understanding the codebase. Just laying it out there.

@phoerious
Copy link
Member

It would do you good to look at the rest of the PR, @droidmonkey, so you can understand why some things were done the way they were done, so that I don't have to explain myself repeatedly. We went through some of this in March already, in case you don't remember.

TBH, things that don't become clear from the code itself, but only from looking at previous discussion about that code need to be documented. Although I know that you will refuse to do that, I would generally prefer doc blocks above every function. I also know that this is a requirement the rest of our code doesn't meet (and therefore I won't refuse to merge because of this), but we want to change that.

@droidmonkey
Copy link
Member

droidmonkey commented Nov 19, 2017

I spent a solid 2.5 days diving into your contribution. If a comment is invalid or I'm being a jerk address it in that specific comment. I'm not unreasonable, this contrib is basically open heart surgery on this app. If this does not work flawlessly, the app is worthless and potentially thousands of peoples databases can be corrupted or have flawed security.

@keepassxreboot keepassxreboot deleted a comment from angelsl Nov 19, 2017
@droidmonkey
Copy link
Member

droidmonkey commented Nov 19, 2017

FYI - I deleted the atEnd() comments since there was no action required

@hifi
Copy link
Member

hifi commented Nov 19, 2017

I suggest splitting this even more by leaving all KDBX format related changes out for now (including renames) so the things that everyone agree on can be merged in sooner making future work easier to review and comment on.

I have added a custom KDF in C style from OpenSSH in my PR and it would be really nice to refactor it into a class API that could be defined by this PR. I also used QCryptographicHash for SHA-512 and again this PR could lay the work to bring SHA-512 into the main crypto classes and I could switch to use that instead.

In my eyes this is great work towards the bigger picture and getting even half of the blood and sweat that has gone into it merged will help advance that. Keep up the good work!

Copy link
Member

@phoerious phoerious left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is already a lot closer to a mergeable state. Thanks for your continued work on this.

My main complaint is also the fields concept, which just needs to be dropped altogether. Also the Kdf/AesKdf class structure needs a bit of restructuring. Apart from these major complaints and a small handful of medium problems, most requested changes are rather minor style issues.

}

if (m_data.kdf != nullptr) {
delete m_data.kdf;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use a QSharedPointer instead?

QByteArray transformedMasterKey;
Kdf* kdf;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above.

@@ -88,25 +90,18 @@ class Database : public QObject

Uuid cipher() const;
Database::CompressionAlgorithm compressionAlgo() const;
QByteArray transformSeed() const;
quint64 transformRounds() const;
Kdf* kdf();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Method should be const (see @droidmonkey's comment for this method's implementation).

@@ -120,6 +115,7 @@ class Database : public QObject
* Returns a unique id that is only valid as long as the Database exists.
*/
Uuid uuid();
bool changeKdf(Kdf* kdf);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The difference between setKdf and changeKdf isn't easily understandable for the user of this API without looking at the implementation. Please document. I would also move this up a few lines to group it with the setter method.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These functions probably shouldn't exist at all. The database class should create and maintain the kdf object with external entities merely telling the database which kdf to use.

@@ -42,6 +42,9 @@ namespace Endian {
QByteArray int16ToBytes(qint16 num, QSysInfo::Endian byteOrder);
QByteArray int32ToBytes(qint32 num, QSysInfo::Endian byteOrder);
QByteArray int64ToBytes(qint64 num, QSysInfo::Endian byteOrder);
QByteArray uint16ToBytes(quint16 num, QSysInfo::Endian byteOrder);
QByteArray uint32ToBytes(quint32 num, QSysInfo::Endian byteOrder);
QByteArray uint64ToBytes(quint64 num, QSysInfo::Endian byteOrder);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. The implementation of all these methods is basically identical.
ba.resize(2|4|8); can be rewritten as ba.resize(sizeof(T));

}
}

void DatabaseSettingsWidget::displayKdf(const Kdf& kdf)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, please, please make this a from. Don't convolute the CPP files with GUI boilerplate code.


void DatabaseSettingsWidget::clearKdfWidgets()
{
m_benchmarkField = nullptr;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drop this function together with the one above.

if (m_error) {
return -1;
}
else if (m_eof) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remember to not insert newlines before else.


m_buffer.clear();
}
m_blockIndex++;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++m_blockIndex;

QByteArray seed() const;

bool setRounds(quint64 rounds);
bool setSeed(const QByteArray& seed);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These four methods should be move to the parent class as pure virtual methods. They are generic enough to be applicable to any kind of KDF.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would remove the need for fields.

@phoerious phoerious changed the base branch from develop to feature/kdbx4 November 26, 2017 21:03
@phoerious phoerious merged commit 5020666 into keepassxreboot:feature/kdbx4 Nov 26, 2017
@phoerious
Copy link
Member

I merged your work into a working branch in our repository. You are very welcome to continue working on this with us, but overall, we'll take it from here. Thank you very much for your contribution!

@phoerious phoerious mentioned this pull request Nov 26, 2017
@angelsl
Copy link
Contributor Author

angelsl commented Nov 29, 2017

I haven't had time to look at this until now. Sorry about that.

@phoerious

TBH, things that don't become clear from the code itself, but only from looking at previous discussion about that code need to be documented.

My comment about some things having been discussed back in March was about certain choices I made, which (I think) would have been clear had this PR not been chopped in half.

Although I know that you will refuse to do that, I would generally prefer doc blocks above every function.

Indeed, I do not generally think there is a need to document every single internal function, but I would've agreed to add that.

@droidmonkey

I spent a solid 2.5 days diving into your contribution. If a comment is invalid or I'm being a jerk address it in that specific comment.

I did.

I'm not unreasonable,

I got pretty frustrated with these comments

This function drives me bonkers. Needs to be implemented as a form and when fields get refactored... simplified immensely.

What is the purpose of the fields if you just store it in a member variable anyway!, I am so confused. The whole kdf code needs to be completely redone and SIMPLIFIED.

because you appear to completely dismiss the fields abstraction without appearing to consider at all why I did it that way.

My original intention was to support arbitrary KDFs, if more are ever added, without having to modify the GUI code.

It is clear both you and @phoerious don't think that is necessary. So be it.

this contrib is basically open heart surgery on this app. If this does not work flawlessly, the app is worthless and potentially thousands of peoples databases can be corrupted or have flawed security.

I made an effort to make sure (almost?) every single commit in this PR compiles and passes the full test suite. So there's that.

In any case, I honestly don't feel like these changes are terribly major. A lot of it was copy and paste, or adding glue code to the crypto classes to support ciphers already supported by gcrypt, etc. A visual review plus manual testing plus the unit test suite should uncover any bugs — or perhaps I'm being too optimistic.

Perhaps it is time you reformat the entire codebase once through, so that everything is consistent and you can spend less time commenting on stylistic issues. The reason I did weird things like newline-before-else was because that was the style used by the source file, and I did not want to go in and reformat everything (too much churn), nor did I want to create a source file with a mix of different styles.

I guess it is for the better that you guys take over the PR. Be sure to pick up the rest of the KDBX 4 changes as well.

Feel free to contact me if you need any clarifications or help.

phoerious added a commit that referenced this pull request Feb 27, 2018
- Add support for KDBX 4.0, Argon2 and ChaCha20 [#148, #1179, #1230, #1494]
- Add SSH Agent feature [#1098, #1450, #1463]
- Add preview panel with details of the selected entry [#879, #1338]
- Add more and configurable columns to entry table and allow copying of values by double click [#1305]
- Add KeePassXC-Browser API as a replacement for KeePassHTTP [#608]
- Deprecate KeePassHTTP [#1392]
- Add support for Steam one-time passwords [#1206]
- Add support for multiple Auto-Type sequences for a single entry [#1390]
- Adjust YubiKey HMAC-SHA1 challenge-response key generation for KDBX 4.0 [#1060]
- Replace qHttp with cURL for website icon downloads [#1460]
- Remove lock file [#1231]
- Add option to create backup file before saving [#1385]
- Ask to save a generated password before closing the entry password generator [#1499]
- Resolve placeholders recursively [#1078]
- Add Auto-Type button to the toolbar [#1056]
- Improve window focus handling for Auto-Type dialogs [#1204, #1490]
- Auto-Type dialog and password generator can now be exited with ESC [#1252, #1412]
- Add optional dark tray icon [#1154]
- Add new "Unsafe saving" option to work around saving problems with file sync services [#1385]
- Add IBus support to AppImage and additional image formats to Windows builds [#1534, #1537]
- Add diceware password generator to CLI [#1406]
- Add --key-file option to CLI [#816, #824]
- Add DBus interface for opening and closing KeePassXC databases [#283]
- Add KDBX compression options to database settings [#1419]
- Discourage use of old fixed-length key files in favor of arbitrary files [#1326, #1327]
- Correct reference resolution in entry fields [#1486]
- Fix window state and recent databases not being remembered on exit [#1453]
- Correct history item generation when configuring TOTP for an entry [#1446]
- Correct multiple TOTP bugs [#1414]
- Automatic saving after every change is now a default [#279]
- Allow creation of new entries during search [#1398]
- Correct menu issues on macOS [#1335]
- Allow compilation on OpenBSD [#1328]
- Improve entry attachments view [#1139, #1298]
- Fix auto lock for Gnome and Xfce [#910, #1249]
- Don't remember key files in file dialogs when the setting is disabled [#1188]
- Improve database merging and conflict resolution [#807, #1165]
- Fix macOS pasteboard issues [#1202]
- Improve startup times on some platforms [#1205]
- Hide the notes field by default [#1124]
- Toggle main window by clicking tray icon with the middle mouse button [#992]
- Fix custom icons not copied over when databases are merged [#1008]
- Allow use of DEL key to delete entries [#914]
- Correct intermittent crash due to stale history items [#1527]
- Sanitize newline characters in title, username and URL fields [#1502]
- Reopen previously opened databases in correct order [#774]
- Use system's zxcvbn library if available [#701]
- Implement various i18n improvements [#690, #875, #1436]
@sebast889
Copy link

I guess it is for the better that you guys take over the PR. Be sure to pick up the rest of the KDBX 4 changes as well.

@phoerious were these changes ever picked up? If not, are they are still needed?

@droidmonkey
Copy link
Member

This PR is merged.....

@sebast889
Copy link

I realize that. I just thought those changes in the linked repo might have been forgotten about. If they weren't then disregard my comment.

@droidmonkey
Copy link
Member

Yes we completely overhauled this implementation. It's been in since 2.3.0

@phoerious phoerious added pr: new feature Pull request that adds a new feature and removed new feature labels Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
high priority 🚨 pr: new feature Pull request that adds a new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants