Automate C code formatting #391

ilammy · 2019-02-25T18:39:18Z

Let's introduce an automatic code style formatting and checks for Themis. We use clang-format as this tool is generally available and gives reasonable results.

Introduce two new targets for Makefile:

make fmt
make fmt_check

fmt is intended for humans to reformat the code before submitting changes for review. Currently it formats only C code but it may be extended to other languages as well.

fmt_check is intended for CI to do format checks to ensure that the code is formatted accordingly. Unfortunately, clang-format does not have a convenient check mode so we hack one with git diff.

Then actually reformat the whole code base in accordance with the style. Enforce the new code style during CI builds. We use a separate job so that style errors are reported separately from failed tests.

This pull includes the formatting changes as well to ensure that the build is green. There is no point in reviewing the restyling commit itself, focus on the other ones.

Do not merge this pull request via GitHub interface, I'll do it manually to preserve the authorship information as I don't want to be credited for authoring half of this repository.

Closes #270

ilammy · 2019-02-25T18:49:35Z

Please review the code style and provide feedback. You'd better just check out my branch and inspect it yourself in your development environment:

git remote add ilammy [email protected]:ilammy/themis.git
git fetch ilammy
git checkout formatting

Meanwhile I'll be negotiating behavior differences between clang-format 6.0 and 8.0...

Lagovas

the code is much better now. just one remark that will be better to avoid oneline if statements without braces because statistic tells that it a good place for human mistakes

docs/examples/c++/secure_message_server.cpp

src/soter/boringssl/soter_sign_ecdsa.c

ilammy · 2019-02-26T10:40:28Z

@Lagovas I fully agree with you on that one-line ifs also need braces. Unfortunately, clang-format does not seem to be able to insert them. I looked for a couple of other formatters like astyle, but they were worse then clang-format in other places. Another option that I have in mind is trying clang-tidy, maybe it will work better.

Makefile

Let's introduce an automatic code style formatting and checks for Themis. We use clang-format and clang-tidy tools as they are generally available and gives reasonable results. Introduce two new targets for Makefile: - make fmt - make fmt_check "fmt" is intended for humans to reformat the code before submitting changes for review. Currently it formats only C code but it may be extended to other languages as well. "fmt_check" is intended for CI to do format checks to ensure that the code is formatted accordingly. Unfortunately, clang-format does not have a convenient check mode so we hack one with git diff. clang-tidy requires the compilation flags so we integrate the format checking into the build pipeline. It is based on a standalone tool which can be integrated into developer's IDE if necessary. We start by formatting only Soter and Themis source code.

Drop commented out code that cause problems with formatting. clang-tidy and clang-format conflict in line-wrapping this part.

Actually reformat the whole code base in accordance with the style.

Enforce the new code style during CI builds. We use a separate job so that style errors are reported separately from failed tests. We also check both flavors as they use different Soter sources.

ilammy · 2019-02-27T17:42:17Z

@Lagovas @secumod I have updated the PR to use the new approach to formatting. It is now uses Makefile targets to guide formatting and static checks. This enables nice progress and error reports.

The main targets are the same:

make fmt to fixup the code in working directory (for developers)
make fmt_check to check the formatting without changing the files (for CI)

Plus there is a new one:

make fmt_clean to cleanup marker files added during reformatting (to force reformatting again)

The formatting logic is placed into a new script scripts/fmt.sh which may be used as a standlone tool (e.g., from the IDE/editor).

We now also use clang-tidy tool to apply source-level fixups (e.g., inserting braces around one-line conditional statements). The ruleset is fairly minimal now but can be extended later.

I have also limited the changes in this PR to main source code of Soter and Themis. We'll update other places later when we settle on the code style.

This PR currently includes commits from #392 which are necessary for the tooling to work. They will be removed when #392 is merged into master.

ilammy · 2019-02-27T17:44:45Z

.clang-tidy

@@ -0,0 +1,4 @@
+---
+Checks: 'readability-*'


There are more check available, but I'd like to enable them later in separate pull requests as they can change the code semantics.

ilammy · 2019-02-27T17:45:31Z

src/soter/boringssl/soter.mk

@@ -15,6 +15,10 @@
 #

 SOTER_SRC += $(wildcard $(CRYPTO_ENGINE)/*.c)
+SOTER_AUD_SRC += $(wildcard $(CRYPTO_ENGINE)/*.c)
+SOTER_AUD_SRC += $(wildcard $(CRYPTO_ENGINE)/*.h)


It turned out that BoringSSL build for audit did not include BoringSSL code for Soter. I've added it here.

Initialize submodules to pull BoringSSL source code. For some reason CircleCI does not do that for us.

ignatk · 2019-02-27T17:48:02Z

src/soter/boringssl/soter.mk

+SOTER_AUD_SRC += $(wildcard $(CRYPTO_ENGINE)/*.c)
+SOTER_AUD_SRC += $(wildcard $(CRYPTO_ENGINE)/*.h)
+SOTER_FMT_SRC += $(wildcard $(CRYPTO_ENGINE)/*.c)
+SOTER_FMT_SRC += $(wildcard $(CRYPTO_ENGINE)/*.h)


why are we doing all these variable copies for the same data?

Consistency. Should I just replace this with the following?

CRYPTO_ENGINE_SOURCES = $(wildcard $(CRYPTO_ENGINE)/*.c) CRYPTO_ENGINE_HEADERS = $(wildcard $(CRYPTO_ENGINE)/*.h) SOTER_SRC += $(CRYPTO_ENGINE_SOURCES) SOTER_AUD_SRC += $(CRYPTO_ENGINE_SOURCES) $(CRYPTO_ENGINE_HEADERS) SOTER_FMT_SRC += $(CRYPTO_ENGINE_SOURCES) $(CRYPTO_ENGINE_HEADERS)

we already have some variables, why create more aggregate variables? Our format target can just depend on $(SOTER_SRC) $(CRYPTO_ENGINE_SOURCES) ..

ignatk · 2019-02-27T17:51:25Z

Makefile

+FMT_FILES += $(THEMIS_FMT)
+FMT_FILES += $(SOTER_FMT)
+
+$(OBJ_PATH)/%.fmt: CMD = CFLAGS='$(CFLAGS)' ./scripts/fmt.sh $< $@


I personally don't favour hiding the code in a separate script (unless it is too complicated), because it just creates another level of indirection

let's keep the language suffix, so we know which files are being formatted (.c for formatting C code). Later we should reuse the framework to format other languages (.go and .rs at least).

I put it into a script because we'll probably have to write almost the same Makefile targets in what seems like 5+ places (JNI wrapper, Soter and Themis tests, ThemisPP code and tests, JSThemis code).

While the C compiler is easy to use via $(CC), this formatting abomination is not. And it has two operation modes with different command lines. So I used a script to avoid duplicating the code.

The same logic applies to suffixes. We'll have to format at least *.c, *.h, *.hpp, *.cpp files. I would not like to write the same targets four times. The script could guess the language by the file extension and run an appropriate formatter.

you can write them in one place and just include whatever.mk where needed

you can have a single command for multiple extensions, if they are processed the same:

%.foo %.bar: <some common processing for both *.foo and *.bar> %.baz: <some other command for baz>

Remove custom installation of clang-format and clang-tidy since they are a part of the Docker image from now on.

Avoid copy-pasting wildcard substitutions all around, put them into variable and make the file lists more compact that way.

Just inline invocations of clang tools into the Makefile, do not use an intermediate shell script as that complicates inspection. Also, limit the effect to *.c and *.h files in $(SRC_PATH). We can extend the formatting later to other file types. Now that we don't have a configurable script we have to duplicate the fixups and checks. However, that allows us to be more precise with checks and avoid interference between fixups and checks. With that we no longer need the "fmt_clean" target, just a regular "clean" will do.

ilammy · 2019-02-28T11:57:11Z

@secumod I've updated the PR, please give it a look.

Avoided repeating $(wildcard ...) all the time, now the files are mentioned only once.
Replaced the shell script with inline invocations of clang-format and clang-tidy.
Improved dependency tracking for fixups and checks which now can be performed independently.
Limited formatting invocations to only *.c and *.h files of Soter and Themis main code

I still don't quite like how we have to duplicate these lines:

$(OBJ_PATH)/%.fmt_fixup: $(SRC_PATH)/%
	@mkdir -p $(@D)
	@echo -n "fixup $< "
	@$(BUILD_CMD_)

$(OBJ_PATH)/%.fmt_check: $(SRC_PATH)/%
	@mkdir -p $(@D)
	@echo -n "check $< "
	@$(BUILD_CMD_)

However, I don't see how it can be avoided without doing some crazy macro stuff with $(call ...) and $(eval ...).

ignatk

Looks good from first glance. I suggest the following: break this PR into two (or three):

one that introduces the changes to the build system and format targets
the other one, which actually is the result of make fmt
enforcing the format in CI (maybe merged with the previous one - don't have a strong opinion here)

ignatk · 2019-02-28T12:04:15Z

I still don't quite like how we have to duplicate these lines

You think you duplicate, because you hide the difference by redefining the BUILD_CMD_ (the approach I don't really support TBH - feels like the shell script as well). Also it is a bit confusing, because this is not actually a build command, rather check/enforce/etc command

ilammy · 2019-02-28T12:18:58Z

Well, even without BUILD_CMD this would be two separate targets:

$(OBJ_PATH)/%.fmt_fixup: $(SRC_PATH)/%
	@$(CLANG_TIDY) -fix $< -- $(CFLAGS) 2>/dev/null
	@$(CLANG_FORMAT) -i $<
	@touch $@

$(OBJ_PATH)/%.fmt_check: $(SRC_PATH)/%
	@$(CLANG_FORMAT) $< | diff -u $< -
	@$(CLANG_TIDY) $< -- $(CFLAGS) 2>/dev/null
	@touch $@

Though the reason for duplication will be more apparent in this case, as they really do different actions.

Lagovas · 2019-02-28T12:22:02Z

oh, @$(CLANG_FORMAT) $< | diff -u $< - looks like some dialect of brainfuck language )) I hope we always will have some guru of bash and makefiles that will understand all these code after several months ; )

ilammy · 2019-02-28T12:25:43Z

Oh, even more targets as we should apply clang tools only to C files:

$(OBJ_PATH)/%.c.fmt_fixup: $(SRC_PATH)/%.c
	# fixup

$(OBJ_PATH)/%.h.fmt_fixup: $(SRC_PATH)/%.h
	# fixup

$(OBJ_PATH)/%.c.fmt_check: $(SRC_PATH)/%.c
	# check

$(OBJ_PATH)/%.h.fmt_check: $(SRC_PATH)/%.h
	# check

with fixup and check actions being exactly the same.

Plus we'll have to write the same lines for every other combination of source file and object-file... Though, I guess we can avoid this particular pain point by not reusing the object file hierarchy for checks and just mirroring the source file locations, using the rules like:

$(CHECK_PATH)/%.h.fmt_check: %.h
	# check C header

$(CHECK_PATH)/%.c.fmt_check: %.c
	# check C source

and generating targets like this:

-THEMIS_FMT_FIXUP = $(patsubst $(SRC_PATH)/%,$(OBJ_PATH)/%.fmt_fixup,$(THEMIS_FMT_SRC))
+THEMIS_FMT_FIXUP = $(patsubst %,$(CHECK_PATH)/%.fmt_fixup,$(THEMIS_FMT_SRC))

ilammy · 2019-02-28T12:42:31Z

Anyway... This PR ended up being too complex to review. I have split it into three parts (as it should be done from the starts). They include all changes discussed here rebased on top of the current master branch.

ilammy requested review from Lagovas, ignatk, shadinua, storojs72 and vixentael as code owners February 25, 2019 18:39

Lagovas reviewed Feb 26, 2019

View reviewed changes

docs/examples/c++/secure_message_server.cpp Outdated Show resolved Hide resolved

src/soter/boringssl/soter_sign_ecdsa.c Show resolved Hide resolved

ignatk requested changes Feb 26, 2019

View reviewed changes

Makefile Show resolved Hide resolved

ilammy and others added 9 commits February 26, 2019 18:22

Fix and optimize includes: themis

bffee99

Fix and optimize includes: soter

9b27142

Fix detailed include in JNI code

50a65fe

Consistent use of quotes vs. angled brackets

bb960d2

Merge branch 'optimize-includes' into formatting

da56d21

Automate C code formatting [2/4]

5651d6f

Drop commented out code that cause problems with formatting. clang-tidy and clang-format conflict in line-wrapping this part.

Automate C code formatting [3/4]

bc686c1

Actually reformat the whole code base in accordance with the style.

Automate C code formatting [4/4]

6ed0064

Enforce the new code style during CI builds. We use a separate job so that style errors are reported separately from failed tests. We also check both flavors as they use different Soter sources.

ilammy force-pushed the formatting branch from dfb949a to 6ed0064 Compare February 27, 2019 17:34

ilammy requested a review from ignatk February 27, 2019 17:35

ilammy commented Feb 27, 2019

View reviewed changes

fixup! Automate C code formatting [4/4]

3033c7d

Initialize submodules to pull BoringSSL source code. For some reason CircleCI does not do that for us.

ignatk reviewed Feb 27, 2019

View reviewed changes

ilammy added 3 commits February 27, 2019 20:43

fixup! Automate C code formatting [4/4]

0381f6c

Remove custom installation of clang-format and clang-tidy since they are a part of the Docker image from now on.

Simplify files lists in makefiles

5f4098e

Avoid copy-pasting wildcard substitutions all around, put them into variable and make the file lists more compact that way.

ilammy requested a review from ignatk February 28, 2019 11:52

ignatk reviewed Feb 28, 2019

View reviewed changes

This was referenced Feb 28, 2019

Automate C code formatting [1/3] #395

Merged

Automate C code formatting [2/3] #396

Merged

Automate C code formatting [3/3] #397

Merged

ilammy closed this Feb 28, 2019

ilammy deleted the formatting branch March 28, 2019 18:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automate C code formatting #391

Automate C code formatting #391

ilammy commented Feb 25, 2019 •

edited

Loading

ilammy commented Feb 25, 2019

Lagovas left a comment

ilammy commented Feb 26, 2019

ilammy commented Feb 27, 2019

ilammy Feb 27, 2019

ilammy Feb 27, 2019

ignatk Feb 27, 2019

ilammy Feb 27, 2019

ignatk Feb 27, 2019

ignatk Feb 27, 2019

ilammy Feb 27, 2019

ignatk Feb 27, 2019

ilammy commented Feb 28, 2019

ignatk left a comment

ignatk commented Feb 28, 2019

ilammy commented Feb 28, 2019

Lagovas commented Feb 28, 2019

ilammy commented Feb 28, 2019

ilammy commented Feb 28, 2019

Automate C code formatting #391

Automate C code formatting #391

Conversation

ilammy commented Feb 25, 2019 • edited Loading

ilammy commented Feb 25, 2019

Lagovas left a comment

Choose a reason for hiding this comment

ilammy commented Feb 26, 2019

ilammy commented Feb 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilammy commented Feb 28, 2019

ignatk left a comment

Choose a reason for hiding this comment

ignatk commented Feb 28, 2019

ilammy commented Feb 28, 2019

Lagovas commented Feb 28, 2019

ilammy commented Feb 28, 2019

ilammy commented Feb 28, 2019

ilammy commented Feb 25, 2019 •

edited

Loading