DevDiary: Fjoni

[fjoniyz]

This is the dev diary of @fjoniyz

[fjoniyz]

Week of 01.05-07.05

• Read about enrichment techniques when it comes to messages in Kafka. This includes various internet articles regarding implementation of a broker as a proxy which modifies the messages before they are received by the consumer. This idea did not have that many resources online(at least until 03.05 I couldn't find anything useful), thus I shifted my focus on finding another approach which may work and then I read about Kafka Streams and a bit more in detail in the book Kafka: The definitive guide.
• Updated README and some small things in the repo (protecting the main branch, added linter as a job for checks and also creating the issue for the dev diary)
• Implemented a simple message enrichment method using the small tutorial provided by Apache Kafka (https://kafka.apache.org/34/documentation/streams/tutorial)

[fjoniyz]

Week of 08.05-14.05

• Added properties file for the setup of Kafka
• Containerized the application. Now one only needs to run the Dockerfile
• Made some changes regarding pom.xml file and linter jobs
• Had a look at possibilities of binding Python files in Java. The only alternative is to use Jython which is unfortunately outdated since it only works with Python 2.7. We thought about this idea because some projects on Github already implement what we want to do(our anonymization methods) thus we thought of binding the files with our project.

[fjoniyz]

Week of 15.05-21.05

• First implementation of Delta Doca in Java and some bug fixes in it

[fjoniyz]

Week of 22.05-28.05

• Implemented Kafka Streams approach within the Doca implementation

[fjoniyz]

Week of 29.05-04.06

• Checked whether the algorithm works fine, and it does. The problem was the small dataset.
• Implemented the state store for the records. Now records (before they are anonymized) are going to get saved in the state store first, and each time a new record comes, the whole entries in state store are going to be fed to the Doca algorithm and they are going to be anonymized and written to the output topic.(TODO figure out the best way to feed the data to the algorithm and implement it)

[fjoniyz]

Week of 05.06-11.06

• Redis persistence added for Kafka Streams
• Midterm presentation

[fjoniyz]

Week of 12.06-18.06

• Castleguard can now be called from Kafka Streams
• Config file for Delta Doca where the user can input the fields he/she wants to anonymize(we'll have a look whether this should be changed or not) and also the parameters for it

[fjoniyz]

Week of 19.06-25.06

• Added serde for ChargingStation example from Ampeers
• Refactoring and restructuring of the files

[fjoniyz]

Week of 26.06-02.07

• Worked on the interface for both anonymization algorithms
• Restructuring of Pipe and how we write messages in kafka topic

[fjoniyz]

Week of 03.07-09.07

• Implementation of prognosis
• Debugging of prognosis
• Helped a bit with the debugging of visualizer

[fjoniyz]

Week of 10.07-16.07

• Final presentation week
• Small fixes to bring the main branch to a stable version
• Changed how we retrieve information from the redis cache for delta doca

[fjoniyz]

Week of 17.07-23.07

• Final implementation of prognosis
• Starting the report

[fjoniyz]

Week of 24.07-30.07

• Refactoring of code and standardization of interfaces

[fjoniyz]

Week of 31.07-06.08

• Implemented the Pipe in such a way that we send json messages to topic instead of strings
• Cleaned up not needed classes because of the first point
• Solved merge conflicts with main

[fjoniyz]

August

• Prognosis
• Plotting for information loss and average difference of prognosis results
• Report writing
• Clean up of code